StableLM-2: partial RoPE and a first LayerNorm-with-bias path in a dense ROCm engine
Ran Tao (Octoryn Research)
Abstract
A from-scratch port of StableLM-2-1.6B (Stability AI) onto a dense ROCm engine that avoids vendor BLAS/DNN libraries. It adds two engine firsts: partial RoPE (rotary span 16 of 64 head dims, NeoX split-half) and a first non-RMS normalization path, LayerNorm-with-bias, plus QKV bias. Against a Hugging Face transformers f16 reference on identical token ids, prefill-logits parity reaches cosine 0.999997 (12/12); q8 reaches 0.998910 (11/12). A full-RoPE control collapses to -0.62 (3/12), so the rotary span is load-bearing. Correctness validation only: a small-sample, single-run smoke test.
1. Contribution
Claimed. A from-scratch port of StableLM-2-1.6B (Stability AI) onto a dense ROCm inference engine that does not depend on vendor BLAS/DNN libraries reproduces a Hugging Face transformers f16 prefill-logits reference at cosine 0.999997, argmax 12/12 on the tested prompts. The port required two engine firsts: partial RoPE (only a rotary span of 16 of the 64 head dimensions is rotated, NeoX split-half over that span; the remaining dimensions pass through unrotated) and the engine's first non-RMS normalization path — LayerNorm-with-bias (a new fp16 layer-norm kernel), plus QKV bias. A full-RoPE negative control (rotating the entire head) collapses to cosine -0.62, argmax 3/12, establishing that the partial-RoPE rotary span is load-bearing.
NOT claimed. This is a correctness-validation (numerical-parity) result, not a novel algorithm and not a quality or performance result. It does not claim generation quality, throughput, or any benchmark-scale evaluation. It does not claim that q8 (measured cosine 0.998910, argmax 11/12) matches f16. It does not claim coverage of larger StableLM variants. The parity is a small-sample smoke test against one reference, at f16 (the engine's highest precision), not a statistically powered benchmark.
2. Methods
| Field | Value |
|---|---|
| Model | StableLM-2-1.6B (Stability AI) |
| Hardware | AMD RX 6900 XT (RDNA2, 16 GB) |
| Engine | Dense ROCm engine independent of vendor BLAS/DNN libraries |
| Reference / baseline | Hugging Face transformers (version not recorded in source notes) |
| Procedure | Direct logits parity: identical token ids fed to engine and reference; compare full logit vectors per position |
| Precision | f16 (primary); q8 reported as secondary |
| Metric + gate | Per-position cosine of logit vectors + argmax match; parity gate cosine >= 0.999 (per the project's parity convention) |
| Stimulus / inputs | Prompts yielding 12 compared token positions (exact prompt strings not recorded in source notes) |
3. Results
| Configuration | Cosine | Argmax | Verdict |
|---|---|---|---|
| f16 parity vs transformers | 0.999997 | 12/12 | PASS |
| q8 parity vs transformers | 0.998910 | 11/12 | honest noise (below f16) |
| No-regress: Qwen2.5 | not recorded in source notes | 12/12 | PASS |
| No-regress: OLMo-2 | not recorded in source notes | 12/12 | PASS |
| No-regress: Phi-3 | not recorded in source notes | 12/12 | PASS |
4. Ablations / Negative controls
| Control | Change | Cosine | Argmax | Interpretation |
|---|---|---|---|---|
| Full RoPE | Rotate the entire head (ignore the rotary span) | -0.62 | 3/12 | FAIL — partial-RoPE rotary span is load-bearing; wrong span destroys the forward |
5. Threats to validity / Limitations
- Sample size. Parity is measured over a small set of prompts totalling 12 token positions — a correctness smoke test, not a benchmark-scale evaluation. The exact prompt strings and count are not recorded in the source notes.
- Single run / no variance. No repeated-run statistics or variance bounds are reported for any number; all figures are single-run.
- Parity != quality. Logit cosine and argmax match the reference's numerics; they say nothing about generation quality, coherence, or downstream task behavior.
- Precision ceiling. f16 is the engine's highest precision (no f32 forward), so the residual gap from 1.0 is irreducible here; q8 (0.998910, 11/12) is honestly below f16 and is not claimed to match it.
- Quantization scope. q8 is reported but not validated to f16-equivalence; quantized generation/throughput is out of scope.
- Baseline weakness. The
transformersreference version is not recorded in source notes; reproduction requires pinning and validating the reference independently before trusting the gate. - Scope. Result applies only to StableLM-2-1.6B f16 prefill-logits parity on the target RDNA2 GPU; it does not transfer to other families, larger StableLM variants, generation, or throughput.
- Evidence level. Marked
experimental; this reflects a numerical-parity validation, not a controlled benchmark — readers should not read "experimental" as benchmark-scale.
6. Reproducibility / Provenance
- Reference.
transformers(version not recorded in source notes) with identical token ids; validate the reference first — confirm it loads real weights and produces coherent logits before comparing. - Negative control. Re-run with full-RoPE (rotate the whole head) to reproduce the -0.62 / 3/12 collapse, confirming the partial-RoPE span is load-bearing.
- Library independence. Linked libraries are the HIP runtime plus system only (no vendor BLAS/DNN libraries).
- Scope note. The shared Qwen2-style attention handler was narrowed to exclude the StableLM/LayerNorm path; no-regress was re-checked on Qwen2.5 / OLMo-2 / Phi-3 (12/12 each).
- Build / run. Compiled with the standard HIP toolchain targeting the RDNA2 architecture and linked against the HIP runtime.
Claim boundary
The author's explicit scope — what this work does and does not establish — carried over from the Octoryn Research publishing model.
Proves
- A from-scratch port of StableLM-2-1.6B with partial RoPE (rotary span 16 of 64 head dims) and LayerNorm-with-bias reproduces a Hugging Face transformers f16 prefill-logits reference at cosine 0.999997, argmax 12/12 on the tested prompts.
- A full-RoPE negative control collapses to cosine -0.62 (3/12), showing the partial-RoPE rotary span is load-bearing.
- The engine gains its first non-RMS normalization path (LayerNorm-with-bias).
Does not prove
- Generation quality, coherence, or downstream task behavior.
- Throughput or latency.
- Larger StableLM variants or other model families.
- That q8 (cosine 0.998910) matches f16.
- Benchmark-scale accuracy; parity is a small-sample, single-run smoke test over 12 token positions.
Applies when
- f16 prefill-logits parity for StableLM-2-1.6B on the target RDNA2 GPU versus a Hugging Face transformers reference run on identical token ids.
Does not apply when
- Quantized, generation, or throughput claims, other model families or larger variants, or any conclusion requiring more than the tested small prompt sample.
- The transformers reference version is unpinned or unvalidated.
Authors
- Ran Tao — Investigation, Writing
Cite this
Citation
Tao, R., Octoryn Research. (2026). StableLM-2: partial RoPE and a first LayerNorm-with-bias path in a dense ROCm engine (TR-2026-0016). Octopus Research Institute.
BibTeX
@techreport{oritr20260016,
title = {StableLM-2: partial RoPE and a first LayerNorm-with-bias path in a dense ROCm engine},
author = {Tao, Ran and {Octoryn Research}},
institution = {Octopus Research Institute},
year = {2026},
note = {Permanent ID TR-2026-0016. Not peer reviewed.}
}Disclosures
- Funding
- Hardware and infrastructure provided by Octoryn / Octopus Core Pty Ltd.
- Conflicts of interest
- Octoryn ships commercial inference and governance tooling; findings are reported independently.