Cross-box tensor-parallel decode is latency-bound on commodity 2.5GbE

1. Contribution / Thesis

Claimed. On a commodity 2.5GbE LAN with roughly 0.8 ms round-trip latency, splitting a single model's decode (one token per step) across boxes by tensor parallelism (TP) was originally argued to be structurally a regression rather than a speedup. The argument is a latency budget: decode requires a synchronizing barrier/all-reduce per layer (and per sub-op), estimated at 48-96 barriers per token; at sub-millisecond RTT this is tens of milliseconds of communication per token, which already meets or exceeds the measured single-box per-token time. The decisive variable is latency (paid dozens of times per token), not bandwidth (a single token's activation is tiny). The note argues that aggregation should happen at the request boundary (data-parallel, 0 hops/token) rather than within a layer.

This revision upgrades fidelity through several stages, culminating in a real two-sided heterogeneous GPU all-reduce, and lands on a conditional verdict: the original "structurally dead" magnitude is wrong, but cross-box TP decode is a net loss in its architecturally achievable form unless barriers are fused.

NOT claimed. This note does not claim a generation-quality result, a throughput benchmark, or a controlled end-to-end timed run of a full TP decoder. The compute term remains analytical (a production single-box baseline halved). It does not claim TP decode is dead on low-latency datacenter interconnects, nor that data-parallel aggregate throughput is single-model scale-up. Altitude: a design/feasibility argument with measured communication primitives, not a complete novel system.

2. The latency argument (design)

Decode emits one token per forward step. Tensor parallelism shards each layer's matmuls across boxes and must synchronize (all-reduce/barrier) per layer and per sub-op. The per-token communication cost is therefore approximately (barriers/token) x (round-trip latency):

• Barriers/token: 48-96 (per-layer, per-sub-op synchronization).
• Round-trip latency: roughly 0.8 ms on 2.5GbE (original headline assumption).
• Single-box per-token time: roughly 42 ms (production baseline, this revision).

Because the communication term can approach or exceed the single-box per-token time before any compute, the original headline placed cross-box TP decode at 0.35-0.6x single-box speed. Bandwidth does not enter the binding constraint: one token's activation is small relative to LAN bandwidth; the cost is paying the round-trip latency many times per token.

3. Viable alternatives (design)

Rule of thumb: on a sub-millisecond LAN, prefer to parallelize at the request boundary rather than within the layer, unless the per-layer barrier count can be fused down aggressively.

4. Why the original "structurally dead" framing is too pessimistic

Two structural correctives reshape the original analysis:

• Decode is memory-bandwidth-bound, not compute-bound. Per-token time is dominated by reading the weights from memory. Tensor parallelism splits the weights across boxes, so each box reads roughly half the model per token. The "compute/2" term is really a genuine bandwidth/2 saving — a structural reason 2-box TP can win for decode that a pure FLOPs view overlooks.

• The realistic single-box baseline is faster than first assumed. The production single-stream decode baseline is roughly 42 ms/token (quantized ~30B-class model, full set resident on an Apple Silicon box), which supersedes an earlier higher extrapolation.

With these correctives, the binding question becomes the real per-barrier all-reduce cost and the achievable barrier count, addressed below.

5. Measured communication primitives (this work)

The revision proceeds through three fidelity stages. All measurements are on a fast, low-latency LAN between Apple Silicon and discrete-GPU boxes; precise host, link, and software identifiers are omitted here.

5.1 Network-only sustained sequential-barrier comm

Sustained sequential-barrier communication (not a single bare RTT multiplied by a barrier count) at a small, fixed per-barrier payload on the order of a few KB:

A representative small-payload median RTT on the measured fast link is sub-0.2 ms — roughly 4-5x faster than the ~0.8 ms the original headline assumed. The payload is realistic rather than an underestimate: a single decode-step all-reduce for a ~30B-class model carries only a few KB of activation, set by the hidden width times the activation dtype.

A network-only compute/2 + comm model gives ~38.5 ms/token (96) and ~29.6 ms/token (48) — both under 42 ms, suggesting uniform viability. This projection explicitly excludes GPU-side per-barrier overhead and was flagged as a knife-edge for the 96-barrier case.

5.2 Full barrier cycle with real GPU overhead

The next stage timed a full barrier cycle including real GPU-side overhead — a kernel launch, device synchronization, and device<->host copies wrapped around each host-to-host round-trip — against a CPU-only echo far side. Isolated per-barrier GPU overhead is ~45-46 us/barrier, flat across barrier count, adding roughly 26% on top of the network-only cost.

Recomputing TP_per_token = compute/2 + full_barrier_cycle_comm with compute/2 = 21.0 ms:

• 96-barrier (worst): ~42.55 ms/token -> at/above the 42 ms break-even (~0.99x). The knife-edge crossed.
• 48-barrier (best): ~31.69 ms/token -> comfortably viable (~1.33x).

5.3 Real two-sided heterogeneous GPU all-reduce

The final stage replaces the CPU echo with a real two-sided cross-architecture all-reduce: each box runs a real half-precision matrix-vector kernel and exchanges its partial for a reduce over a single persistent, no-delay (Nagle-disabled) TCP socket. One side runs a CUDA-class discrete GPU, the other a ROCm-class discrete GPU; both kernels self-tested correct against a CPU reference (one side bit-exact, the other within half-precision output-rounding tolerance).

An earlier run failed to make progress because the two sides disagreed on the per-barrier payload width — one side's send format and the other's receive format were sized for different element widths, so a blocking receive waited indefinitely for bytes that were never sent. Aligning both directions to a single half-precision payload width, while keeping the reduction accumulate in higher precision on-device, resolved it; the end-to-end heterogeneous reduce then ran to completion (server-confirmed full barrier count across all runs) — a real, completed two-sided reduce, not an echo and not a stall.

Per-barrier cost is flat across barrier count at ~250 us — the cost of a real two-sided matrix-vector all-reduce (near-side kernel + sync + copy, far-side kernel + sync + copy concurrently, plus the host-to-host exchange each way).

Real-vs-echo: the real two-sided reduce came out roughly 13% higher than the CPU-echo projection (~250 us vs ~222 us per barrier), exactly as the prior "echo is a lower bound" inference predicted — the uplift is the live far-side GPU compute+sync+copy that the echo elided.

5.4 Recomputed 2-box TP per-token with the real all-reduce

Model: TP_per_token = single_box_compute/2 + real_all_reduce_comm, with compute/2 = 21.0 ms (bandwidth-bound decode, weights split across two boxes).

• 48-barrier (fused): 21.0 + 12.00 = ~33.0 ms/token -> below 42 ms -> VIABLE (~1.27x).
• 96-barrier (unfused): 21.0 + 24.17 = ~45.2 ms/token -> at/above 42 ms -> REGRESSION (~0.93x).

5.5 Achievable barrier count is 96

Megatron-style tensor parallelism has two synchronizing all-reduces per transformer layer in the forward pass — one after the attention out-projection, one after the MLP/MoE down-projection — separated by a residual add, normalization, and the entire MLP, which cannot be fused away. (Sequence-parallel variants substitute reduce-scatter + all-gather but keep the same two communication points per layer.) So for a deep decoder the architecturally achievable barrier count is two per layer, i.e. ~96/token for a ~48-layer model, not 48. The 48-barrier "viable" case is a hypothetical lower bound that standard decoder TP cannot reach.

6. Verdict

With a real two-sided all-reduce (not an echo, not a projection), the verdict is MIXED and conditional:

• Fused (<=48 barriers/token): confirmed viable, ~1.27x — backed by a real, completed heterogeneous cross-architecture reduce with correct matrix-vector kernels on both sides.
• Unfused (96 barriers/token): the negative survives at full fidelity, ~0.93x — a ~7.5% regression. The real reduce's ~13% comm uplift over the echo pushed this case from break-even (~42.55 ms) to a decisive regression (~45.2 ms), resolving the earlier knife-edge hedge.

The original blanket negative ("structurally dead / 0.35-0.6x collapse") is wrong for bandwidth-bound decode on a fast link; the realistic worst case is a mild ~7.5% regression, close enough that the bandwidth/2 saving nearly offsets the comm. Barrier fusion is the decisive engineering lever, not the link alone: fuse to <=48 barriers/token and 2-box cross-box TP decode is a real ~1.27x win; pay 96 sequential barriers and it is a real ~0.93x loss.

7. Limitations / Threats to validity

• Not an end-to-end timed TP decoder. The measured primitive is a single-socket two-sided matrix-vector all-reduce; there is no full weight-split end-to-end matmul stack, no production-grade collective (point-to-point reduce over one socket), and no cross-box expert routing. The 21.0 ms compute/2 term remains analytical (production baseline halved), not measured under TP.
• Payload realism vs. larger models. The fixed small per-barrier payload is realistic for a moderate hidden width; a model with larger hidden width would exchange more bytes per barrier, pushing both cases worse.
• Copy cost is a lower bound. Pinned host buffers give a best-case device<->host copy cost, so the ~45-46 us/barrier GPU overhead is a floor; pageable memory would be slower.
• Link- and N-dependent. Results hold for the measured fast interconnect (~250 us real two-sided barrier) and N=2. Slower links (a ~0.8 ms-RTT 2.5GbE cluster) regress in both cases; larger N changes compute/N and the barrier-serialization profile.
• Single-config baselines. The ~42 ms/token baseline and the sustained-barrier comm figures are single-config production/sustained scalars with no recorded variance.
• Data-parallel aggregate is a heterogeneous sum. The aggregate data-parallel throughput aggregates different models across boxes; it is explicitly not single-model scale-up.
• Evidence level: experimental. A real measured two-sided reduce, but not a reproduced end-to-end TP decode benchmark.

8. Operational note

On a 128 GB Apple Silicon box, the data-parallel path obeys a single-resident rule: a box must not co-load two ~30B models, since the resident set alone would exceed RAM and trigger swap. After the TP experiment the discrete-GPU box's serving daemon was restored and verified back to its pre-test resident state, with no residual GPU leak from the experiment binaries.