FP8 grouped-expert prefill with on-chip int4-to-e4m3 dequant

1. Contribution

Claimed. On an idle consumer 8 GB GPU running real int4-quantized shards, making both the dense q/k/v/o attention projections and the expert FFN FP8 (e4m3) — and replacing per-token int4 expert dispatch (one dispatch per routed token per layer) with a single grouped expert GEMM per layer — speeds the prefill phase by 2.13x at depth L=8 and 2.18x at L=12, with logit parity cosine >= 0.997 versus the int4 baseline on the same inputs. A secondary engineering claim: int4 weights can be dequantized to e4m3 on-chip without unpacking the resident packed-int4 weight bank, with a per-(token, group) activation scale folded into the accumulator path.

NOT claimed. This is a prefill-phase latency engineering result on two layer-depth configurations on a single GPU — not a novel algorithm and not a benchmark-scale evaluation. It does NOT claim: (a) decode-phase speedups (only prefill was measured); (b) any statement about generation quality — cosine-of-logits parity is a numerical proxy, not a quality measurement; (c) that the cosine holds at greater depth — it is observed to narrow with depth (about 0.9987 at L=8 to 0.9973 at L=12); (d) dense-FP8 benefit at small prompt length (dense FP8 requires a sufficiently large prompt); (e) full-model behavior — only L=8 and L=12 subsets were measured. The altitude is a correctness-gated latency optimization, not a research contribution.

2. Methods

3. Results

Both MoE configs PASS the parity gate (>= 0.997).

Measured depth curve (this work)

A fresh single-run sweep on the same consumer 8 GB GPU, with a short prompt and FP8 prefill enabled, varying layer depth L. Cosine is final-logits FP8-vs-int4.

Cosine narrows monotonically with depth (about 0.9992 -> 0.9989 -> 0.9979) but stays above the 0.997 gate through L=12, with exact argmax match at every measured depth. At L>=16, FP8 weight preparation could not fit within the 8 GB VRAM budget (a resource limit on this hardware, NOT a correctness result) — so the depth curve here is capped at L=12 and the deeper-stack trend is unmeasured on this hardware. Attention-projection speedup is depth-stable (~2.66-2.70x); end-to-end speedup rises slightly with depth (1.67x -> 1.76x) as more layers amortize the dense win. (Coherence text from a partial layer stack is meaningless and is not a generation claim.)

4. Threats to validity / Limitations

• Sample size. Two prefill configurations on a single GPU. This is a latency smoke test on subsets, not a benchmark-scale evaluation. Full-model behavior was not measured.
• Single run, no variance. Latencies and speedups are single-run; no variance, confidence interval, or repeat count is recorded in source notes.
• Parity != quality. The >= 0.997 gate is logit-cosine versus the engine's own int4 baseline. It bounds numerical agreement with int4, NOT generation quality, and the int4 baseline is itself a lossy reference (not an fp16/fp32 oracle).
• Precision ceiling / depth trend. e4m3 has 3 mantissa bits and rounds harder than fp16. Cosine narrows monotonically with depth across the measured points (about 0.9987 at L=8 -> 0.9973 at L=12). The trend is downward toward the gate; nothing here proves the gate holds at greater depth.
• Scope of the speedup. Prefill only — decode was not measured. Dense FP8 only engages for sufficiently large prompts; the grouped expert path engages for any prompt length. The 2.7x attention-projection figure is reported without its own cosine.
• Evidence level. Carried as reproduced; source notes describe single-run measurements on two configs. The reproduction strength is bounded by the single-run, two-config scope above.
• Baseline-comparison weakness. Speedup is int4-path vs FP8-path within the same engine; there is no external/library baseline comparison.

5. Reproducibility / Provenance

• Hardware: Consumer 8 GB GPU, idle.
• Stack: links against the vendor GPU runtime only (no third-party GEMM/DNN libraries).
• Build/run toolchain: standard vendor GPU compiler at the target architecture, optimized C++ build.
• Validate-baseline caveat: the parity gate compares FP8 against the engine's own int4 path, which is a lossy reference; a stronger validation would anchor against an fp16/fp32 oracle, which is not recorded here.