Continuous batching with paged KV: a dependency-free CUDA throughput path

1. Contribution

Claimed. A dependency-free CUDA inference path (no vendor BLAS/DNN/TLAS libraries) implements two throughput mechanisms: paged KV caching (a global block pool with per-(layer, sequence) block tables, blocks allocated on demand and freed on sequence exit) and iteration-level continuous batching (an active set of B sequences run through one batched forward pass, with dynamic exit and block free). On an idle consumer GPU, batched per-stream argmax output is byte-identical to single-stream decode across small batch sizes and several block sizes, and aggregate throughput modestly exceeds single-serial decode.

NOT claimed. This is a correctness-and-engineering validation, not a novel algorithm and not a benchmark-scale throughput result. It does NOT claim linear-in-batch scaling (aggregate stays roughly flat in batch because expert dispatch and the quantized projections still loop per row, leaving per-iteration cost weight-bound). It does NOT claim parity with an external serving system's absolute throughput. It does NOT claim a full-model aggregate number — the test machine holds only a layer subset resident. Argmax parity is a correctness property; it says nothing about generation quality.

2. Methods

3. Results

4. Threats to validity / Limitations

• Sample size is a correctness smoke test, not a benchmark. Parity is measured over a handful of batch-by-block configurations on short decode runs. This validates byte-identical output, not throughput at benchmark scale or under realistic mixed-length traffic.
• Single-run perf, no variance. The speedup and tokens/sec figures are reported as ranges across configs with no repeated trials, confidence intervals, or variance — they are point observations on an idle GPU.
• Parity is not quality. Argmax exact-match proves the batched/paged path reproduces the single-stream decode bit-for-bit; it says nothing about generation quality or downstream task performance.
• Scope / scaling ceiling is acknowledged. Aggregate is explicitly NOT linear in batch — expert dispatch and the quantized projections loop per row, so the path is weight-bound. The headline framing must be read against this: no external serving system was run, so the comparison is directional, not measured.
• Quantization / hardware bound. Results hold for the resident int4 weight bank on a single-digit-GB consumer GPU holding only a layer subset. Full-model aggregate is unmeasured and requires a substantially larger device-memory budget.
• Oracle/baseline weakness. The baseline is the engine's own single-stream path, which makes parity self-consistent but provides no external throughput reference for the throughput claim.

5. Reproducibility / Provenance

• Build: compiled with the CUDA toolchain targeting the consumer GPU's architecture, optimized C++17 build.
• Version pins (CUDA toolkit, driver, model shard): not recorded.
• Procedure pins (high level): paged KV via a global block pool plus per-(layer, sequence) block tables with allocate-on-demand/free-on-exit; a dedicated scatter-write step and a batched paged-attention decode step with fp32-stable softmax; continuous batching via an iteration-level scheduler over B active sequences; a resident int4 weight bank reused across the path.
• Sovereignty check: confirm the linked libraries report the CUDA runtime plus system libraries only (no vendor BLAS/DNN/TLAS libraries) before trusting the dependency-free claim.
• Next levers (not yet done): batched expert GEMM over active rows, prefill chunking, admission control; full-model aggregate on a larger-memory device.