A scenario-aware routing policy with co-resident dense-small backends: the model bake-off as automatic runtime behavior

1. Contribution

Claimed. A pure, model-free routing policy layer wired into the serving daemon that classifies each incoming request into one of five operator scenarios (RAG-or-code / creative / simple-task / long-context / high-risk) and lowers it to a typed routing decision over four axes: model (large MoE vs small-dense), speculative-decode mode (n-gram vs off), retrieval grounding (on/off), and escalation (none / cloud-frontier / multi-model adjudication) — all driven by a single operator policy table. The decision drives live per-request execution on one daemon, with no environment flag and no restart: model selection across co-resident backends, per-request speculative-decode hot-switch, and a per-request retrieval/epistemic grounding loop. Three concrete sub-results are claimed, each from a single live measured run:

1. Co-resident dense-small backends loaded alongside a 30B-class quantized MoE — measured at a modest total resident footprint for four backends (the 30B-class MoE plus 4B, 1.7B, and 0.6B dense tiers) on a workstation-class host with large unified memory, never a second 30B and never any full-precision copy.
2. Simple-task complexity tiers (mini -> 0.6B / standard -> 1.7B / quality -> 4B) that route the easy majority to a genuinely lighter sub-3B-active dense path; the two smallest tiers measured faster than the MoE path, with the 4B tier an honest tradeoff (slower on single-stream).
3. Per-request speculative-decode hot-switch that is output-identical to greedy with content-dependent steady speedup.

NOT claimed. No aggregate throughput number across mixed traffic; no multi-run / variance / confidence interval; no claim that routing makes single-stream decode faster (it does not — see section 5); no claim that the 4B tier is faster than the MoE; no production-traffic numbers (no live operator deployment). This is the per-request scenario->model/spec/RAG policy engine; it is distinct from the dataplane router + failover work reported in the related report TR-2026-0026.

2. Methods

3. Results

Classification + plan (unit, model-free). A deterministic scenario ladder is used: an explicit task hint wins; otherwise heuristics apply (retrieval/code/high-repetition -> RAG-or-code; context at or above a threshold -> long-context; short classify/extract/rewrite -> simple-task; risk markers -> high-risk; else creative). The unit-test suite grew across the deliveries as tiers and retrieval wiring were added; each route's plan was checked against hand-written reference configs. The safe default is creative -> MoE greedy; an explicit model field is an operator override that wins.

Co-residency (single live run). Four backends co-resident at a modest total footprint on a large-memory host, with ample memory free and swap unchanged; never a second 30B and never any full-precision copy. An unloaded tier degrades gracefully (standard -> quality -> MoE; always served, never a 404). The route-model audit value returned the correct backend per scenario — classify/extract -> 0.6B, translate -> 1.7B, rewrite/summarize -> 4B, creative/code -> MoE — all coherent.

Tier speed (single live micro-benchmark, fused steady decode). The 0.6B tier ran about 1.54x the MoE single-stream path and the 1.7B tier about 1.25x, while the 4B tier ran about 0.71x (slower; served only on an explicit quality request). The MoE path was unchanged, and its greedy output was byte-identical with vs without the dense backends (checksum match). Fused dense decode measured roughly 2x faster than the plain-glue baseline on the 4B model.

Speculative hot-switch (single live run + A/B). Per-request speculative mode executes on a single daemon with the environment-level speculative flag off, alternating RAG-or-code -> n-gram speculative (high acceptance) and creative -> plain greedy (speculative off); an explicit per-request speculative override wins. The speculative path is byte-identical to pure MoE greedy (no divergence) across the tested inputs. Steady speedup is content-dependent, ranging from near break-even on low-repetition text to several-fold on a maximally repetitive input.

Retrieval / memory (single live run). Long-context requests run an epistemic/retrieval grounding loop per request: a grounded answer is served when the evidence supports it, and the model abstains (returns no guess, flagged as requiring external truth) when the evidence contradicts or is unknown, backed by a hash-chained audit record. Non-long-context scenarios (creative / RAG-or-code) skip the loop and are greedy byte-identical across repeats. The epistemic-guard test suite passes with no regression.

4. Ablations / negative controls

• Parity-as-control. MoE greedy output is byte-identical with vs without the co-resident dense backends (checksum match), proving the added backends do not perturb the MoE path. The speculative path is identical to pure greedy (no divergence), proving the hot-switch is exact rather than approximate.
• Honest economics inversion (load-bearing negative result). A 4B dense model reads more 4-bit weight per token than the MoE's roughly-3B active experts, so routing simple-task traffic to the 4B model was not a single-stream win (about 0.71x). Only sub-3B-active tiers (0.6B / 1.7B) read fewer weight bytes per token and are genuinely lighter and faster — the tier design exists because the naive 4B route failed.
• Speculative acceptance floor. Forcing speculative decode on a creative prompt drove acceptance to roughly zero (verify-bound, no win); the policy therefore never routes free-form creative to speculative.
• Degrade path. An unloaded tier and a missing dense backend both degrade through to the MoE (still served), confirming there is no 404 surface.

5. Threats to validity / limitations

• Single-run, no variance. Every speed/coherence number above is from one measured live run with no repetition, no confidence interval, and no independent reproduction. Evidence level is experimental, not reproduced.
• No aggregate / mixed-traffic throughput. No end-to-end number for a realistic mixed request stream is claimed; tier decode rates are isolated micro-benchmark figures.
• Routing does not speed up single-stream decode. Prior work establishes that 30B-class single-stream 4-bit decode is structurally latency/bandwidth-bound on this class of hardware; this policy reduces cost per simple-task by offloading to a lighter model, it does not raise the MoE's own decode rate.
• Speculative speedup is content-dependent and can be near break-even on low-repetition text; the headline several-fold figure is a maximally-repetitive case, not typical.
• Retrieval loop runs only on the non-streaming path and requires the daemon launched in its grounding mode; otherwise the retrieval decision is recorded-and-flagged only (honestly surfaced on the response header). Escalation emits a decision point and hook only — no external cloud call is made (out of scope).
• One audit-header probe did not surface once. A single independent route-tier audit probe failed to surface due to a mis-pointed shard directory; the authoritative evidence is the route-model value across the four live backends, not the tier header.
• Model substitution caveat. The quality tier uses an instruction-tuned 4B variant because the plain 4B is a hybrid thinking model that emits chain-of-thought markers and stalls on one-word answers.

6. Reproducibility

• Architecture. A pure model-free routing policy module classifies requests and emits a typed decision; a routing bridge wires it into the serving daemon's chat handler, batched decode scheduler, and resident-backend manager. Unit tests cover classification, plan byte-equality, tiers, speculative mode, and the epistemic guard.
• Serve configuration. Up to three co-resident dense tiers are loaded via per-tier shard-directory flags plus model ids; a grounding serve mode enables the retrieval loop. Per-request behavior can be overridden via route-model / speculative / retrieval request headers, and decisions are surfaced on route-audit response headers plus a structured log.
• Crash-safety rule from corpus. Only one 30B-class model may be resident at a time on this host; use 4-bit weights, never full precision; gate on available memory before any resident load.
• Not recorded in source. Multi-run variance, mixed-traffic aggregate throughput, and the missed tier-header probe are not recorded; a clean reproduction should add these.