Multi-card inference topology: request-boundary aggregation on a latency-bound LAN

1. Thesis / Contribution

Claimed. Given a latency-bound LAN (the premise: cross-box tensor-parallel decode regresses on a sub-millisecond consumer-Ethernet link per a companion technical report, because the LAN's bandwidth is ample but its latency is decisive), the correct shape for multi-box Apple Silicon inference is to aggregate at the request boundary, never within the layer. The design ranking that follows is: data-parallel (0 hops/token) first, pipeline parallel (~2 hops/token) second, within-layer tensor-parallel rejected for decode on this interconnect.

NOT claimed. This is a design/architecture note, not a measured-throughput report. The headline aggregate figure (a several-tens-of-tokens-per-second range across a small multi-box HTTP daemon) is a design estimate and an honest heterogeneous sum — it is NOT the single-model decode rate of one ~30B model, NOT a verified production measurement, and NOT a linear single-model scale-up. The pipeline multiplier (roughly 1.5-2.5x) is a design estimate, not a measured production figure. No generation-quality, parity, or benchmark-scale claim is made here. The data-parallel path is stated as prototype-stage (near-complete, not finished), i.e. not a completed-and-benchmarked system.

2. Design

The interconnect is bandwidth-ample but latency-bound, so all aggregation is placed at the request boundary.

Iron rule (resident-memory constraint). On a 128GB Apple Silicon box, the data-parallel path must stay single-resident: never co-load two ~30B-class models. The resident set of two such models alone would exceed RAM and trigger swap / the memory watchdog. This is grounded in a real out-of-memory incident (a lesson learned), not a hypothetical.

3. Evidence / Status

There is no Results table in the conventional sense because this note presents no first-party measurements of its own — every quantitative figure is either an estimate or a value imported from a separately-reported incident/report. Presenting an empirical-style results table here would overclaim.

4. Limitations (harsh)

• The headline throughput is an estimate, not a measurement. It carries no run count, no variance, no single-model decomposition. It is explicitly a sum across heterogeneous models of differing quality, so it cannot be read as the decode rate of a single ~30B model.
• The pipeline multiplier is unbuilt. No code, no measurement; it is a paper estimate that depends on a not-yet-implemented self-managed low-latency transport.
• The premise is borrowed. The decisive "tensor-parallel regresses" finding is measured in a companion report, not reproduced here; this note inherits whatever sample-size and single-interconnect limits that report carries.
• Single interconnect. All reasoning is specific to a sub-millisecond consumer-Ethernet LAN between Apple Silicon boxes. On a low-latency interconnect (NVLink-class), within-layer tensor-parallel may be viable and this ranking could invert.
• The experimental evidence level is generous for this note. This note itself contributes a design argument plus one observed incident; its quantitative core is estimates. The label is retained per provenance rules but should be read as "experimental premise, design conclusion," not "experimentally validated throughput."
• Prototype-stage is not finished. The remaining integration, failover, and routing behaviour under concurrent load are unverified here.

5. Provenance / Relation

• Type: Architecture paper. Status: Published.
• Relation: Directly downstream of the companion cross-box tensor-parallel latency technical report (the "tensor-parallel regresses on consumer-Ethernet" premise).
• Hardware: Apple Silicon Mac boxes over a consumer-Ethernet LAN (128GB-class boxes; small multi-box cluster). Build/run commands for the data-parallel daemon: not recorded in source notes.
• Reproducibility caveat: the central premise should be re-validated against the companion latency report before this design ranking is relied upon; the throughput estimates are unvalidated by construction.