Resident packed-int4 decode: ~5x over host-streaming on a 24GB-class GPU
Ran Tao (Octoryn Research)
Abstract
A GPU-resident packed-int4 decode path loads all layer weights to the GPU once and dequantizes on-chip, so no fp16 weight is materialized and there is no per-token host-to-device weight copy. On an idle 24GB-class consumer GPU, two layer subsets improve by 5.07x and 5.49x over a streaming baseline, bit-identical to streaming (cosine 1.0, argmax exact) by construction. Scope is narrow: two subsets, single-run, idle GPU, single-stream. The full-model resident figure is analytical only. No generation-quality or concurrency claims are made.
Contribution
Claimed. A single-stream decode path that loads all layer weights to the GPU once in packed int4 form (a dequant-on-chip GEMV step reconstructs values on the fly, so no fp16 weight is ever materialized) and reuses the resident weight pointers, eliminating the per-token host-to-device weight copy of the streaming baseline. On an idle consumer GPU (24GB-class, recent architecture), one layer subset decodes 14.5 -> 73.5 tok/s (5.07x) and a larger layer subset 10.5 -> 57.7 tok/s (5.49x), with output bit-identical to the streaming path (cosine 1.0, argmax exact) by construction. The work is an engineering / correctness-validation result: a memory-placement optimization (resident vs. streamed weights) plus a parity check, not a novel algorithm or kernel.
NOT claimed. This does NOT claim full-model resident throughput — the development box used for measurement holds far less VRAM than the target and cannot host all layers, so only layer subsets were measured; the full-model number is analytical, not benchmarked. It does NOT claim anything about generation quality (parity to the streaming path is bit-identity, not a quality result), throughput under concurrency or load, benchmark-scale evaluation, behaviour on non-idle GPUs, or any other GPU architecture. The speedups are single-run point measurements with no variance reported.
Methods
| Field | Value |
|---|---|
| Hardware | Recent-architecture consumer GPU on the measurement box (idle, well under the target VRAM budget); cannot hold all layers resident |
| Target deployment | 24GB-class consumer GPU |
| Stack / sovereignty | Self-contained GPU build depending only on the GPU runtime (no vendor BLAS / DNN libraries) |
| Weight form | Packed int4, dequantized on-chip by the decode GEMV step (no fp16 materialization) |
| Baseline | Streaming decoder with per-token host-to-device weight copy |
| Procedure | Load all layer weights to GPU once (resident); decode reuses resident pointers with zero per-token host-to-device transfer; a parity self-check runs both streaming and resident paths in one process |
| Precision metric + gate | Cosine vs. streaming path; bit-identical by construction (same bytes uploaded once, same kernels) |
| Stimulus / inputs | Two layer subsets (a smaller and a larger one) with a small number of token positions each |
Results
| Subset | Cosine | Argmax | Streaming tok/s | Resident tok/s | Speedup |
|---|---|---|---|---|---|
| smaller subset | 1.0 | exact | 14.5 | 73.5 | 5.07x |
| larger subset | 1.0 | exact | 10.5 | 57.7 | 5.49x |
Analytical full-model memory (not benchmarked): the packed-int4 layer weights plus a head sit on the order of the high-teens GB, rising into the low-twenties GB once the GPU context and allocator overhead are counted. Both fit within a 24GB budget. The dequant-to-fp16 form is several times larger and does not fit.
Threats to validity / Limitations
- Sample size. Two layer subsets over a small number of token positions is a correctness smoke test, not a benchmark-scale evaluation. No prompt diversity is recorded in the source notes.
- Single-run perf. The 5.07x / 5.49x speedups and the underlying tok/s are single point measurements; no repetitions, variance, or confidence intervals are reported.
- Parity != quality. Cosine 1.0 / argmax-exact establishes the resident path is bit-identical to the streaming path; it says nothing about generation quality versus a reference model. The bit-identity is by construction (same bytes, same kernels), so it is a consistency check, not an independent oracle comparison.
- Subset-only / full-model unmeasured. The measurement box cannot hold all layers; full-model resident tok/s is NOT measured — the fits-in-24GB figure is analytical only.
- Idle-GPU, single-stream scope. Numbers are for an idle GPU in single-stream decode. No concurrency, load, or throughput-under-contention claims hold.
- Hardware/quantization bounds. Results are specific to the tested architecture and to packed-int4; no generalization to other architectures or quantization schemes is shown.
- Evidence level. A skeptical reader should read the reproducibility of this note as applying to the parity property (reproducible by construction), not to the throughput magnitude (single-run) or the full-model projection (analytical).
Reproducibility / Provenance
- Procedure / build & run. The build targets the recent consumer-GPU architecture with an optimizing compile. A parity self-check flag runs the streaming and resident paths in one process and reports cosine + argmax + tok/s.
- Stack versions. Library versions were not recorded in the source notes. Sovereignty was verified by inspecting the dynamic dependencies: only the GPU runtime is linked (no vendor BLAS / DNN / template libraries).
- Validate-the-baseline caveat. Parity here is to the project's own streaming path (bit-identity by construction), not to an external reference implementation; an external-oracle comparison is not part of this note.
Claim boundary
The author's explicit scope — what this work does and does not establish — carried over from the Octoryn Research publishing model.
Proves
- A GPU-resident packed-int4 decode path (on-chip dequant, zero per-token host-to-device weight copy) is 5.07x and 5.49x faster than host-streaming on two layer subsets of an idle consumer GPU, bit-identical by construction (cosine 1.0, argmax exact).
- Analytically, a full mid-size model fits within a 24GB budget in packed-int4 form but not as dequantized fp16.
Does not prove
- Full-model resident throughput — not measurable on the measurement hardware (subset-only; the fits-in-24GB figure is analytical, not benchmarked).
- Generation quality (parity is bit-identity to the streaming path, not a quality result), or throughput under load / concurrency.
- Behaviour on non-idle GPUs or other architectures.
- Statistical robustness of the speedup — values are single-run point measurements with no variance (a correctness smoke test over a small number of token positions).
Applies when
- Single-stream resident packed-int4 decode of small layer subsets on an idle recent-architecture consumer GPU, with a self-contained build depending only on the GPU runtime.
Does not apply when
- Full-model resident throughput claims, concurrency / throughput-under-load claims, other GPU hardware, non-int4 quantization, or any generation-quality claim.
- Multi-prompt / benchmark-scale evaluation or any claim requiring variance over repeated runs.
Authors
- Ran Tao — Investigation, Writing
Cite this
Citation
Tao, R., Octoryn Research. (2026). Resident packed-int4 decode: ~5x over host-streaming on a 24GB-class GPU (BR-2026-0004). Octopus Research Institute.
BibTeX
@techreport{oribr20260004,
title = {Resident packed-int4 decode: ~5x over host-streaming on a 24GB-class GPU},
author = {Tao, Ran and {Octoryn Research}},
institution = {Octopus Research Institute},
year = {2026},
note = {Permanent ID BR-2026-0004. Not peer reviewed.}
}Disclosures
- Funding
- Hardware and infrastructure provided by Octoryn / Octopus Core Pty Ltd.
- Conflicts of interest
- Octoryn ships commercial inference and governance tooling; findings are reported independently.