Single-stream int4 decode of a 30B MoE on Apple Silicon: steady-state throughput with byte-identical, vendor-BLAS-free parity
Ran Tao (Octoryn Research)
Abstract
A sovereign Apple-Silicon engine serves a 30B sparse-MoE in packed-int4 (~15.4 GB resident vs ~60 GB at bf16). Folding all decode tokens into one GPU command buffer plus an on-GPU key/value write lifts single-stream greedy decode to a steady-state ~1.87x over a prior per-token path; a silent ~2x regression traced to an optimization gate shipped disabled by default. The path stays greedy byte-identical to a reference forward pass (cosine ~1.0) with zero vendor BLAS. An engineering/correctness result, not benchmark-scale or quality: single-run figures, still behind a popular external runtime.
1. Contribution
Claimed. On a single Apple-Silicon machine, serving a 30B sparse-MoE in packed-int4 (roughly 15.4 GB resident), single-stream greedy decode reaches a steady-state throughput about 1.87x higher than a prior per-token execution path. The improvement comes from two changes: (a) folding all decode tokens into a single GPU command buffer, collapsing the per-token internal synchronizations into a single synchronization point, together with performing the key/value cache write on-GPU rather than on the host; and (b) root-causing a silent roughly 2x serving regression to an optimization gate that shipped disabled by default. The optimized path stays greedy byte-identical to a reference forward pass (cosine approximately 1.0) and uses zero vendor BLAS calls.
NOT claimed. This is an engineering / correctness-validation result, not a novel algorithm and not a benchmark-scale evaluation. It does NOT claim generation quality (byte-identical parity is numerical parity to a reference forward pass, not an evaluation of output quality). It does NOT claim competitiveness with a popular external single-stream runtime (measured roughly 2.2x ahead). It does NOT claim linear multi-stream scaling (measured roughly 1.4x at batch 4). It does NOT claim anything for the bf16 path, sampled (non-greedy) decode, or non-Apple-Silicon hardware. Throughput figures are single-run, single-box, with no reported variance.
2. Methods
| Field | Value |
|---|---|
| Hardware | Apple Silicon Mac (M-series); exact SoC/RAM not recorded in source notes |
| Model | 30B sparse-MoE, served in packed-int4, roughly 15.4 GB resident (versus roughly 60 GB at bf16) |
| Stack / versions | Sovereign Metal-based engine; exact toolchain versions not recorded (recent releases) |
| Oracle / baseline | (parity) reference forward pass on real model shards; (throughput) prior per-token-command-buffer path, plus an external single-stream runtime as a directional reference |
| Procedure | Fold all decode tokens into one GPU command buffer with an on-GPU key/value cache write; enable the previously default-disabled optimization gate; measure steady-state int4 decode throughput and greedy parity on real shards |
| Precision | Packed-int4, dequantized on-chip (no fp16 materialization); greedy (argmax) decode |
| Metric + gate | Throughput: steady-state tokens/sec (excludes a few warmup tokens). Parity: cosine versus reference plus greedy byte-identical; PASS reported |
| Stimulus / inputs | Long-prompt and short-prompt real-shard runs; exact prompt count/text not recorded in source notes |
3. Results
| Measurement | Value |
|---|---|
| Single-stream steady-state (after fix) | roughly 24-25 tok/s |
| Prior per-token path | roughly 14 tok/s |
| Structural speedup | about 1.87x |
| Internal synchronizations | collapsed to a single synchronization per command buffer |
| Regression path (gate disabled) | roughly 12 tok/s; instrumented fast-path never hit |
| Short-generation average (includes warmup) | roughly 11 tok/s |
| Parity (long/short real shard) | cosine approximately 1.0, greedy byte-identical, zero vendor BLAS |
| Parity (production config, real shards) | cosine approximately 1.0, PASS, zero vendor BLAS |
| Concurrency at batch 4 (fused-MoE) | aggregate roughly 1.4x over single-stream |
| Versus external single-stream runtime | roughly 2.2x behind |
| Bandwidth utilization | roughly a quarter of the theoretical memory-bandwidth ceiling |
4. Ablations / Negative controls
| Control | Observation |
|---|---|
| Optimization gate disabled (production serving) | Silent fall-through to the slower per-layer path; instrumented fast-path never engaged across decode layers |
| Optimization gate enabled | Full steady-state throughput, parity byte-identical |
This is a root-cause toggle, not a controlled negative-control sweep; only the two states above are recorded.
5. Threats to validity / Limitations
- Parity is not quality. Cosine approximately 1.0 and greedy byte-identical confirm the optimized path reproduces the reference forward pass numerically. They say nothing about whether the model's outputs are good.
- Sample size / no benchmark scale. Parity rests on long- and short-prompt real-shard runs (exact prompt count not recorded); this is a correctness smoke test, not a benchmark-scale evaluation. Throughput figures are single-run, single-box, with no reported variance or confidence interval.
- Warmup-sensitive headline. The quotable steady-state figure explicitly excludes a few warmup tokens; short generations dominated by warmup average roughly half that. Quoting the steady-state number for short generations would overstate.
- Baseline-comparison weakness. The external-runtime comparison (and the resulting roughly 2.2x gap) carries no recorded version, build, model-quantization match, or run conditions; treat it as directional.
- Concurrency caveat. Single-stream is serialized today; batch-4 aggregate is only about 1.4x (a concurrency reversal, honestly anchored against an earlier verifier's inflated figure). No linear scaling is demonstrated.
- Precision/scope ceiling. Results are bound to packed-int4, greedy decode, on the measured Apple-Silicon machine. bf16, sampled decode, and other hardware are out of scope.
- Evidence level. This rests on a small parity set plus single-run throughput, so "reproduced" should be read as "parity re-confirmed across the fix," not "independently re-benchmarked at scale."
6. Reproducibility / Provenance
- Change set: two structural changes — (1) single-encoder decode folding all tokens into one GPU command buffer with an on-GPU key/value cache write, and (2) enabling the previously default-disabled optimization gate. Exact build identifiers and commands are withheld.
- Version pins: SoC/RAM, toolchain, and external-runtime versions are not recorded in source notes.
- Validate-the-oracle caveat: parity is asserted against a reference forward pass on real shards; the reference's own correctness should be confirmed before trusting the cosine figures.
- Sovereignty check: zero vendor BLAS calls (platform numerics only, no third-party ML framework) on the parity runs; the int4 path dequantizes on-chip from packed int4, keeping the 30B model resident in roughly 15.4 GB rather than roughly 60 GB at bf16.
Claim boundary
The author's explicit scope — what this work does and does not establish — carried over from the Octoryn Research publishing model.
Proves
- Single-stream greedy packed-int4 decode of a 30B sparse-MoE on Apple Silicon reaches a steady-state throughput about 1.87x higher than a prior per-token-command-buffer execution path (steady-state excludes a few warmup tokens).
- The improvement holds with greedy byte-identical output versus a reference forward pass (cosine approximately 1.0) and zero vendor BLAS calls.
- A silent roughly 2x serving regression was root-caused to an optimization gate that shipped disabled by default; enabling it restores the measured steady-state throughput.
Does not prove
- Competitiveness with a popular external single-stream runtime, which was measured roughly 2.2x faster with no recorded version or quantization match.
- Linear multi-stream scaling: measured concurrency at batch 4 is only about 1.4x over single-stream, not linear.
- Generation quality: byte-identical parity to a reference forward pass is numerical parity, not an evaluation of output quality.
- Throughput on non-Apple-Silicon hardware, the bf16 path, or sampled (non-greedy) decode.
Applies when
- Single-stream greedy packed-int4 serving on the measured Apple-Silicon machine, at steady-state with a few warmup tokens excluded.
Does not apply when
- Short generations dominated by warmup, where the average is roughly half the steady-state figure.
- Sampled or non-greedy decode, the bf16 path, batched-throughput claims beyond the measured sub-linear concurrency, or any non-Apple-Silicon hardware.
- Treating single-run, small-parity-set throughput numbers as benchmark-scale or variance-characterized results.
Authors
- Ran Tao — Investigation, Writing
Cite this
Citation
Tao, R., Octoryn Research. (2026). Single-stream int4 decode of a 30B MoE on Apple Silicon: steady-state throughput with byte-identical, vendor-BLAS-free parity (BR-2026-0002). Octopus Research Institute.
BibTeX
@techreport{oribr20260002,
title = {Single-stream int4 decode of a 30B MoE on Apple Silicon: steady-state throughput with byte-identical, vendor-BLAS-free parity},
author = {Tao, Ran and {Octoryn Research}},
institution = {Octopus Research Institute},
year = {2026},
note = {Permanent ID BR-2026-0002. Not peer reviewed.}
}Disclosures
- Funding
- Hardware and infrastructure provided by Octoryn / Octopus Core Pty Ltd.
- Conflicts of interest
- Octoryn ships commercial inference and governance tooling; findings are reported independently.