octoryn-router (sovereign DP router)

octoryn-router is a hand-written HTTP reverse-proxy that sits directly in front of a data-parallel inference fleet. There is no third-party proxy in the request path: the routing and failover logic is code you own and can read, rather than behaviour delegated to an opaque load balancer. That is what "sovereign" means here — not a marketing adjective, but the concrete property that every decision about where a request goes is made by inspectable logic under your control.

How it works

The router's defining capability is mid-stream failover. When a chip or replica serving an in-flight generation becomes unavailable, the router can move that request to another replica while keeping the decoded output byte-identical, so a client that is already mid-generation does not observe a broken or divergent stream. This depends on cross-chip decode consistency: the same tokens must decode the same way on the replica that takes over, which is the subject of a related dataset on cross-chip decode consistency. On top of routing, every decision is written to an event-logged audit trail, so the path a request actually took can be reconstructed after the fact.

Why it matters

For teams running self-hosted, multi-chip inference, failover usually forces a trade-off between resilience and control — reaching for a third-party balancer means accepting behaviour you cannot fully see or verify. octoryn-router keeps both the failover and the record of it in your hands. It sits alongside the institute's wider work on local-sovereign-inference and governed-agent-systems, applying the same principle at the serving layer: decisions stay owned, inspectable, and evidenced. Licensed Apache-2.0.