TR-2026-0009Technical reportNot peer reviewedExperimentalReleased

RWKV in a self-built engine: a second non-attention paradigm (WKV/RNN)

Ran Tao (Octoryn Research)

This is not peer-reviewed. Treat it as a working document, not a validated result.

Abstract

A correctness-validation port of the RWKV-4-169m hand-written WKV linear-attention/RNN recurrence (pre-LN, token-shift, fp32-state max-trick, time/channel-mix gating; no attention or KV cache) into a self-built dense ROCm engine on one consumer-class AMD GPU. Against a Hugging Face transformers oracle on identical token ids, f16 prefill logits matched at cosine 1.0, argmax 12/12 across 12 positions; a kill-WKV control fell to 0.96 (3/12), so the recurrence is load-bearing. A single-run forward-parity smoke test on one small model; not a quality, throughput, quantized, or larger-model result.

Contribution

Claimed. A hand-written WKV recurrence (RWKV-4 linear-attention / RNN) for RWKV-4-169m, ported into a self-built dense ROCm inference engine, reproduces a Hugging Face transformers f16 prefill-logit oracle at cosine 1.0, argmax 12/12 for the tested prompt(s) and token positions on a single consumer-class AMD GPU (RDNA2, 16 GB). A kill-WKV negative control collapses the same metric to cosine 0.96, argmax 3/12 (FAIL), demonstrating that the WKV recurrence is load-bearing rather than incidental. This is the engine's second non-attention sequence mixer (after a state-space model), implemented directly from the reference equations with no third-party RWKV package and no vendor BLAS in the dependency closure.

NOT claimed. This is a correctness-validation (forward-parity) result, not a novel algorithm and not a performance result. It does not claim: generation quality, decode/throughput numbers, benchmark-scale evaluation, long-context recurrence stability, quantized (q8/q4) parity, correctness of larger RWKV versions (v5/v6), or any hybrid attention+recurrence behavior. The altitude is honest: re-deriving a known recurrence and proving the engine's forward matches an oracle on a small stimulus.

Methods

Field	Value
Hardware	Consumer-class AMD GPU (RDNA2, 16 GB)
Model	RWKV-4-169m
Stack	Self-built dense ROCm engine, hand-written WKV kernel; internal parity harness
Oracle / baseline	Hugging Face `transformers` (recent release)
Procedure	Direct f16 logit parity vs oracle on identical input token ids; per-position cosine + argmax match over 12 rows
Precision	f16 forward; WKV numerator/denominator/max state carried in fp32 (standard max-trick for numerical stability)
Metric + gate	Per-position cosine similarity and argmax agreement (12 positions)
Stimulus / inputs	Prompt(s) tokenized to identical ids vs oracle (exact prompt text not recorded in source notes)

Results

Configuration	Cosine	Argmax match
RWKV-4-169m f16 vs `transformers`	1.0	12/12
Negative control: kill-WKV recurrence	0.96	3/12 (FAIL)

Ablations / Negative controls

Control	Change	Outcome	Interpretation
kill-WKV	Remove the WKV recurrence (the model's memory)	cosine 0.96, argmax 3/12 FAIL	WKV is load-bearing; without it the forward breaks

Threats to validity / Limitations

Sample size is a smoke test, not a benchmark. The evidence is parity over 12 token positions for the tested prompt(s) — a correctness smoke test, not a benchmark-scale evaluation.
Single run, no variance. Reported as one measurement; no repeated trials, seeds, or variance bands.
Parity != quality. Cosine 1.0 / argmax 12/12 establishes forward fidelity against the oracle, NOT generation quality, coherence, or downstream task performance.
Precision ceiling. The engine's highest precision here is f16; there is no f32 forward path. f16-tie / sub-ULP ambiguities, where present, are irreducible.
Scope bound to one tiny model. RWKV-4-169m only. No claim for larger RWKV-4 sizes, v5/v6, long-context recurrence stability, quantized paths, throughput, or hybrid attention+recurrence designs.
Oracle/version not pinned in source. The exact transformers version and prompt text are not recorded in the source notes; the parity is only as trustworthy as the (unrecorded) oracle version. Validate the oracle independently before relying on this.
evidenceLevel = experimental is appropriate; this note does not upgrade beyond a single-box, single-run correctness check.

Reproducibility / Provenance

Model: RWKV-4-169m (ported into the self-built dense ROCm engine).
Implementation: hand-written pre-LN, token-shift (mix current and previous token), WKV recurrence (per-channel numerator/denominator/max in fp32 with the standard max-trick), and time-mix / channel-mix gating; no attention, KV cache, or RoPE.
Sovereignty check: the runtime dependency closure shows only the AMD HIP runtime and system libraries — no rocBLAS / hipBLAS / MIOpen / MKL and no third-party RWKV package.
Build: compiled directly against the AMD HIP runtime for the target RDNA2 architecture; no vendor math libraries linked.
Oracle: Hugging Face transformers (a recent release; exact version not pinned in source notes).

Claim boundary

The author's explicit scope — what this work does and does not establish — carried over from the Octoryn Research publishing model.

Proves

A hand-written, self-built WKV recurrence for RWKV-4-169m reproduces a Hugging Face transformers f16 prefill-logit oracle at cosine 1.0 with argmax agreement on all 12 tested token positions for the tested prompt(s).
A kill-WKV negative control drops the same metric to cosine 0.96 (argmax 3/12), demonstrating the WKV recurrence is load-bearing rather than incidental.
The dense ROCm engine now contains two distinct non-attention sequence mixers: a state-space model and the WKV recurrence.

Does not prove

Generation quality, coherence, decode/throughput, or any downstream task performance of the sequential recurrence.
Correctness for larger RWKV-4 sizes, RWKV v5/v6, or long-context recurrence stability.
Quantized (q8/q4) parity or behavior, or hybrid attention-plus-recurrence designs.
Anything beyond a small single-run correctness smoke test over 12 positions with no variance reporting.

Applies when

Evaluating f16 prefill-logit parity for RWKV-4-169m against a transformers oracle on identical input token ids on a single consumer-class AMD GPU.

Does not apply when

Quantized, generation-quality, or throughput claims are needed, or other RWKV versions are in scope.
Benchmark-scale evaluation or multi-run variance is required.
An f32-precision forward path or long-context recurrence stability is expected.

Authors

Ran Tao — Investigation, Writing

Cite this

Citation

Tao, R., Octoryn Research. (2026). RWKV in a self-built engine: a second non-attention paradigm (WKV/RNN) (TR-2026-0009). Octopus Research Institute.

BibTeX

@techreport{oritr20260009,
  title       = {RWKV in a self-built engine: a second non-attention paradigm (WKV/RNN)},
  author      = {Tao, Ran and {Octoryn Research}},
  institution = {Octopus Research Institute},
  year        = {2026},
  note        = {Permanent ID TR-2026-0009. Not peer reviewed.}
}

Disclosures

Funding: Hardware and infrastructure provided by Octoryn / Octopus Core Pty Ltd.
Conflicts of interest: Octoryn ships commercial inference and governance tooling; findings are reported independently.