An honest red result: EOS fixed, but forward degradation persisted
Ran Tao (Octoryn Research)
Abstract
A red checkpoint, kept honestly. On a CUDA stream-resident text decode path using 4-bit quantized weights on consumer-class GPU hardware, an end-of-sequence handling fix made generation stop correctly at the EOS token, but a separate forward-quality degradation persisted, so output stayed wrong. Logged red with no metrics: no trustworthy reference oracle existed yet, because the quantized weights would not load into a standard reference framework without specific older library versions (newer stacks random-initialize experts). Records the before-state; the oracle fix is in a separate note.
1. Contribution / Thesis
Claimed. This is a negative checkpoint documenting the historical state of a CUDA stream-resident text decode path using 4-bit-quantized weights on consumer-class GPU hardware at the recorded time. Two facts are claimed, and only two. First, an end-of-sequence (EOS) handling fix made generation STOP correctly at the EOS token. Second, this fix did not restore output correctness: a separate forward-quality degradation PERSISTED after the EOS fix, so the generated content was still wrong. The note was logged red / unresolved on purpose, not dressed up as a win.
NOT claimed. This note does NOT claim the forward degradation was fixed — it explicitly records the before state. It makes no throughput claim, no generation-quality claim, and no parity-pass claim. The downstream root-cause work (obtaining a trustworthy reference oracle) is described only as context; the actual oracle fix and any subsequent parity result live in a separate parity-methodology note, not here. The altitude is honest: this is a debugging/engineering checkpoint and a discipline artifact (recording reds), not a novel algorithm or a validated result.
2. Method / Context
| Field | Value |
|---|---|
| Path under test | CUDA stream-resident text decode |
| Weights | 4-bit per-expert quantized shards, ~30B-class mixture-of-experts model |
| Build target | consumer-class CUDA GPU architecture |
| Hardware | consumer-class NVIDIA GPU, single-card, modest VRAM |
| Toolchain | direct CUDA compilation at high optimization |
| Sovereignty | runtime depends only on the core CUDA runtime (no vendor BLAS/DNN/template libraries) |
| Symptom before fix | generation did not stop at EOS |
| Change applied | EOS-handling fix (generation now STOPS at end-of-sequence token) |
| Observed after fix | forward-quality degradation PERSISTED (content still wrong) |
| Blocker to root-causing | reference oracle could not load the quantized mixture-of-experts weights |
| Oracle resolution (later, separate note) | pin specific older versions of the loading and quantization libraries |
| Failure mode of wrong versions | newer stacks silently random-initialize the experts |
3. Evidence / Status
This is a research note, not a measured benchmark. The evidence is qualitative and observational:
- EOS fix — confirmed behavioural change: generation stops correctly at the EOS token (was not stopping before).
- Forward degradation — confirmed still present: stopping correctly did not make the content correct; the output remained wrong after the fix. Recorded red.
- Oracle blocker — confirmed and later resolved separately: the reference framework could not load the quantized per-expert weights without specific older library versions; newer loading / quantization stacks silently random-initialize the experts, producing an untrustworthy oracle. The version pin was applied in follow-up work; the actual comparison result is NOT in this note.
No numeric measurement (no cosine, no argmax-match rate, no tokens-per-second) is reported here, because none was recorded for this checkpoint — the forward could not be properly compared until a real oracle existed.
4. Limitations / Threats to Validity
Reviewing this harshly, as the record demands:
- No quantitative data. This note contains zero measurements — no cosine similarity, no argmax-match count, no token-position sample, no throughput. It is a qualitative status log, and should never be cited as evidence of a measured effect size.
- Root cause unisolated at the time. The note asserts the forward degradation is separate from the EOS issue, but without a trustworthy oracle that separation could not be quantitatively verified at the recorded time. It is an observation, not a proven decomposition.
- Oracle absence undermines any correctness claim. Because the reference framework could not load the weights, there was no ground truth to compare against when this was logged — so even "forward is degraded" rests on observed incoherent output, not on a parity gate.
evidenceLevel: experimentalis generous. This is closer to an observational/red checkpoint than a controlled experiment. It is retained for continuity but should be read as "engineering observation, no metrics".- Scope. Findings bind only to this specific configuration (CUDA stream-resident, 4-bit-quantized, consumer-class GPU, ~30B-class mixture-of-experts) at one point in time. Nothing here generalizes to other backends, quantizations, or model sizes.
- Parity != quality, and parity was not even reached. Not only does this note make no quality claim, it could not reach the parity step that would precede such a claim.
5. Provenance / Relation
- Type: intentional red retrospective checkpoint — the before state prior to obtaining a working oracle.
- Version pins (applied in later work): specific older versions of the loading and quantization libraries; wrong versions silently random-initialize experts.
- Relation: the oracle fix and any resulting parity comparison are recorded in a separate parity-methodology note; this note deliberately stops at the unresolved state.
- Hardware: consumer-class NVIDIA GPU, single-card, modest VRAM.
- Sovereignty: runtime depends only on the core CUDA runtime (no vendor BLAS/DNN/template libraries).
- Status: Published as part of rebuilding the project's honest history (recording reds, not only wins).
Claim boundary
The author's explicit scope — what this work does and does not establish — carried over from the Octoryn Research publishing model.
Proves
- At the recorded time, the EOS fix made the CUDA 4-bit-quantized stream-resident generation stop correctly at the EOS token, while a separate forward-quality degradation persisted (logged red).
- The downstream blocker was a non-loadable reference oracle, later addressed by pinning specific older versions of the loading and quantization libraries.
Does not prove
- That the forward degradation was resolved in this note — it was not; this is the before-state.
- Any throughput, generation-quality, or parity-pass claim.
- Any quantitative effect size: this note reports no cosine, argmax-match, or tokens-per-second measurement.
- That the forward degradation is provably independent of the EOS issue — separation was observed, not quantitatively verified absent an oracle.
Applies when
- Recording the historical state of the CUDA 4-bit-quantized stream-resident forward path prior to obtaining a working reference oracle.
Does not apply when
- Citing this as a resolved or positive result — it is intentionally a red checkpoint.
- Treating it as a measured benchmark — it contains no metrics and no parity gate.
- Generalizing beyond the specific CUDA stream-resident, 4-bit-quantized configuration at this point in time.
Authors
- Ran Tao — Investigation, Writing
Cite this
Citation
Tao, R., Octoryn Research. (2026). An honest red result: EOS fixed, but forward degradation persisted (RN-2026-0006). Octopus Research Institute.
BibTeX
@techreport{orirn20260006,
title = {An honest red result: EOS fixed, but forward degradation persisted},
author = {Tao, Ran and {Octoryn Research}},
institution = {Octopus Research Institute},
year = {2026},
note = {Permanent ID RN-2026-0006. Not peer reviewed.}
}Disclosures
- Funding
- Hardware and infrastructure provided by Octoryn / Octopus Core Pty Ltd.
- Conflicts of interest
- Octoryn ships commercial inference and governance tooling; findings are reported independently.