Independent GGUF Forward Pass and Bit-Identical Cross-Backend Consistency for a 26B Text-Diffusion Model

1. Contribution (Claimed / NOT claimed)

Claimed. An independent, self-contained implementation loads and correctly executes the full forward pass of a 26B-parameter, non-autoregressive text-diffusion model. Concretely, this report documents:

1. An independently implemented GGUF reader and dequantizer covering the common quantization formats (including the 4-bit and 6-bit K-quant families plus 8-bit, 5-bit, half- and single-precision floats), replacing prior loaders that were tied to a single architecture family. The 6-bit and 4-bit paths were parity-verified element-wise and through the full pipeline against a golden.
2. From-scratch forward parity versus a self-built llama.cpp eval-callback golden, on a short two-token input: per-row cosine above 0.9999997 with exact argmax match over the full 262144-dimensional logits. The reference path is pure single-precision against the golden's mixed half/brain-float path, leaving a small residual per logit, i.e. the implementation sits on the higher-precision side.
3. Nine load-bearing architecture facts that parity forced (Section 4).
4. Bit-for-bit cross-backend consistency proven by real runs on an Apple Silicon path (handwritten GPU matmul plus portable host C++), an AMD ROCm host, and an NVIDIA CUDA host: all paths produce an identical logits head and identical argmax, matching the golden.
5. Production integration as an in-house backend, with a dedicated diffusion handler registered ahead of the prior stub handler and a byte-exact tokenizer giving self-contained text-to-text.
6. GPU offload using a from-scratch dequant and operators, with no external BLAS, and an honest bottleneck finding: per-expert 4-bit dequant dominates, not dense GEMM.

NOT claimed. This is not a claim about generation quality, decode coverage, or end-to-end answer correctness. Forward parity establishes per-token logit fidelity, not that multi-step diffusion decode converges to the right answer. Self-conditioning is implemented to the exact graph, but its quality convergence is unverified, and the demo decode runs at small canvas only. This is also not a throughput benchmark — the performance numbers are single-condition speedups, not a competitive comparison.

2. Methods

3. Results (measured numbers + conditions)

Forward parity (Apple reference vs golden), short two-token input:

• Both rows: per-row cosine above 0.9999997, with exact argmax match.
• Full 262144-dim logits; intermediate checkpoints: 6-bit embedding dequant element-wise match; 4-bit full-pipeline max relative error on the order of 1e-5; layer-0 attention-out / MoE-FFN-out / layer-out aligned to golden (within ~1e-3).

Cross-backend consistency (same fixed input; golden argmax fixed per row):

All four paths produce a bit-for-bit identical logits head and identical argmax, matching the golden. The ROCm GPU variant additionally verified full argmax match at a 64-token canvas.

GPU offload (measured, single-condition):

• ROCm (dense linears to handwritten half-precision GEMM, host-streaming): flat at 2 tokens; at 64 tokens, roughly 1.6x wall-clock improvement. Parity preserved at both.
• CUDA (dense linears to handwritten tiled GEMM, host-streaming): roughly 1.06x at 2 tokens; logits bit-identical, argmax preserved.
• Host-streaming peak GPU memory was small (on the order of a few hundred MB or less), with the model never resident on the GPU, working around a model larger than VRAM.

Apple speedups (measured, single-condition, illustrative):

• Batching the output-head matmuls (many M=1 GEMMs into one M=W GEMM): per-step decode time roughly halved at small canvas.
• Expert weight caching plus chunk-parallel matmul-vector: forward time reduced by roughly a quarter.
• Expert-resident plus batched concurrent execution: decode time reduced by roughly 1.6x at a small canvas; decode ids bit-identical; forward argmax unchanged.
• Cumulative within one session: first self-conditioned decode dropped roughly 3x per step at small canvas, with forward argmax and decode output ids unchanged throughout.

Decode (end-to-end): an entropy-bound diffusion decode runs end-to-end inside the independent code and produces self-detokenized text. A reported demo at small canvas with a modest step count and no self-conditioning, on a templated "capital of France" prompt, produces coherent on-topic text reaching the start of the reasoning trace; reaching the final answer requires a larger canvas, which is gated by per-step expert dequant cost.

4. Ablations / negative controls

The nine load-bearing architecture facts (each pinned by per-tensor golden comparison; each is a trap that breaks parity if wrong):

1. Dual embedding scaling — prompt tokens and canvas tokens are scaled differently (one scaled by sqrt of the embedding dimension, the other passed through a weightless RMSNorm).
2. Attention scale of unity rather than the usual inverse-sqrt of head dimension — QK-norm already normalizes Q/K magnitude. Control: the conventional scale makes tokens attend too uniformly; unit scale reproduces the golden's self-attention concentration.
3. Non-causal full attention, with sliding-window layers using a fixed window (no effect at small canvas).
4. Per-layer dual RoPE and per-layer head structure — sliding-window layers and global layers differ in rotary base, rotary dimension, head dimension, and the number of query and key/value heads; global layers omit a separate value projection and share the key projection for value. Weight shapes differ per layer; the reader takes per-tensor dimensions.
5. QK-norm plus weightless value-norm — per-head RMSNorm on Q/K with learned weights; per-head RMSNorm on V without a weight.
6. Gemma-style sandwich norms around attention and the FFN entry.
7. Mixed dense-shared plus routed-MoE FFN with a cosine/L2-normalized router and top-k routing. Skipping the router normalization inflates logit magnitude by over an order of magnitude (a load-bearing trap). Branches are post-normed separately, summed, then scaled per layer.
8. Tied output head plus a final tanh-based logit softcap.
9. RMSNorm with a small epsilon and the weight multiplied directly (no 1+w convention).

Cross-backend (negative-control sense): four independent compile paths converging to a bit-identical logits head and argmax is itself a strong cross-implementation control — divergence in any of the nine facts would surface as a logit mismatch.

Bottleneck ablation (honest): dense GEMM offload at very small token counts is flat to roughly 1.06x; two independent GPU investigations reached the same conclusion that wall-clock is dominated by per-expert on-demand 4-bit dequant (hundreds of dequants per forward), tied-output-head streaming (a 262144-row dot), and per-layer weight dequant — not dense GEMM. The 64-token regime is where the GEMM-bound improvement (roughly 1.6x on ROCm) appears.

Self-conditioning gate: with self-conditioning gated off at the first step, the forward is bit-identical, confirming the self-conditioning graph is additive and the off-path equals the verified forward.

5. Threats to validity / limitations

• Forward parity != generation quality. This proves per-token logit fidelity, not that multi-step diffusion decode converges to a correct answer. Self-conditioning quality convergence is unverified (gated by performance: long canvas is slow). The decode demo runs at small canvas and reaches only the start of the reasoning trace.
• Encoder closed on Apple, partial in reference. The production Apple path has a byte-exact tokenizer (self-contained text-to-text). The standalone reference's encode step was still a remaining item (detokenize is self-contained; the demo used llama.cpp templated ids).
• Single-condition measurements. Performance numbers are single runs at fixed conditions, not multi-run or competitive benchmarks; treat as experimental.
• Dequant verification scope. All listed GGUF formats are marked parity-verified, but the explicit element-wise / full-pipeline evidence cited is for the 6-bit and 4-bit K-quant paths.
• GPU offload incomplete per backend. Apple uses the in-house GPU matmul; ROCm/CUDA portable references are CPU host with GPU offload for dense linears only (full expert and output-head GPU kernels are future work). Performance is the open frontier — the real win requires attacking the expert/dequant bottleneck, not dense GEMM.
• Compatibility journey. This line moved the prior compatibility verdict from "needs architecture support" to a forward that is built and parity-verified, plus an Apple production path with handler and CLI; remaining work is performance and ROCm/CUDA GPU offload.

6. Reproducibility (high-level)

• Reference forward / decode (Apple): an independent GGUF reader/dequant module, a forward routine with per-layer parity assertions, and an entropy-bound decode routine. The forward prints cosine and argmax versus the golden; the decode produces text. A 4-bit K-quant GGUF build is required.
• Production integration (Apple): an in-house diffusion model, decode routine, handler, forward CLI product, and a byte-exact tokenizer. Forward parity was rechecked after each change with no regression.
• Cross-backend (ROCm/CUDA): a portable host C++ implementation (no BLAS) plus per-backend GPU variants. Each host obtained the 4-bit K-quant build, compiled with the appropriate toolchain, ran the forward on the fixed input, and recorded parity output. Independence was confirmed by dependency inspection showing no external BLAS (no cuBLAS/rocBLAS).
• Oracle: a self-built llama.cpp (recent open PR) with a patched debug hook emitting full-tensor dumps via its eval-callback tool, used purely as a parity oracle, not a production dependency.