Phi-3-mini: fused-tensor direct loading in a sovereign inference engine
Ran Tao (Octoryn Research)
Abstract
A sovereign port of Phi-3-mini-4k into a dense inference engine that direct-loads the model's fused projection tensors — one concatenated Q/K/V and one concatenated gate/up projection — as single tensors, rather than pre-splitting them at load. Under f16 it reproduces a HuggingFace transformers reference's prefill-logits at cosine 1.0 with full top-1 (argmax) agreement on a short stimulus; the 8-bit path reaches cosine 0.999996. Scope is narrow: a forward-pass numerical-fidelity smoke test only — not generation, throughput, latency, or long context (unwired; 4k plain-RoPE only). Mini only.
1. Contribution
Claimed. A sovereign port of Phi-3-mini-4k into a dense inference engine that direct-loads the model's fused weight tensors — a single concatenated Q/K/V projection and a single concatenated gate/up projection — as single tensors via a dedicated fused-projection loading path, rather than pre-splitting them at load time. Under f16 the port reproduces a HuggingFace transformers reference's prefill-logits at cosine 1.0, top-1 (argmax) agreement on a short stimulus; the quantized 8-bit path reaches cosine 0.999996. This is a correctness-validation and engineering result (a new loading path plus a numerical-fidelity smoke test), not a novel algorithm.
NOT claimed. This note does not claim anything about generation quality, decode throughput, or latency — only prefill-logits fidelity. It does not claim long-context correctness: long context is not wired, so only the 4k plain-RoPE regime is covered. It does not cover Phi-3 medium or larger variants, and the small top-1 count is a smoke-test scale, not a benchmark-scale evaluation.
2. Methods
| Field | Value |
|---|---|
| Hardware | Consumer-class GPU (16 GB) |
| Oracle / baseline | A recent HuggingFace transformers release, identical weights and input ids |
| Procedure | Direct-load the fused Q/K/V and gate/up projections as single tensors via a dedicated fused-projection load path; compare engine prefill logits to the oracle |
| Precision | f16 (primary parity); 8-bit (quantized check) |
| Metric + gate | Per-position cosine of logit vectors; top-1 (argmax) agreement count |
| Stimulus / inputs | Prompt yielding a small number of token positions, all top-1-agreeing; SentencePiece tokenizer, no BOS prepend |
| Model specifics | head_dim = 96; untied lm_head |
3. Results
| Configuration | Cosine | Top-1 (argmax) |
|---|---|---|
| f16 vs transformers reference | 1.0 | full agreement on the stimulus |
| 8-bit (quantized) | 0.999996 | not recorded |
| No-regression (other dense models) | pass (qualitative; values not recorded) | — |
4. Threats to Validity / Limitations
- Sample size is a smoke test, not a benchmark. Full top-1 agreement over a single short stimulus is a correctness smoke test; it does not establish distributional fidelity across diverse prompts, lengths, or sampling regimes.
- Single-run, no variance. Parity is reported as point values with no repetition or confidence interval.
- Parity is not quality. Logit-cosine and top-1 agreement bound numerical fidelity of the forward pass; they say nothing about generation quality, coherence, or downstream task performance.
- Long-context out of scope. Long context is not wired; correctness is asserted only for the 4k plain-RoPE regime. Behaviour beyond 4k is unvalidated.
- Precision ceiling. The 8-bit cosine (0.999996) is below f16; its top-1 agreement is not recorded. The engine's highest-precision path here is f16.
- Quantization / variant bounds. Only Phi-3-mini-4k was tested; medium and larger variants are untested. No-regression on other dense models is asserted qualitatively without recorded numbers.
- evidenceLevel = experimental is retained; the limited sample size above is the honest caveat on that label.
5. Reproducibility / Provenance
| Item | Value |
|---|---|
| Oracle pin | A recent transformers release (validate the oracle loads real weights before trusting parity) |
| Loading path | Direct single-tensor load of the fused Q/K/V and gate/up projections |
| Tokenizer | SentencePiece, no BOS prepend |
| Sovereignty | Links against the vendor GPU runtime plus system libraries only (no third-party vendor BLAS or DNN libraries) |
Claim boundary
The author's explicit scope — what this work does and does not establish — carried over from the Octoryn Research publishing model.
Proves
- A sovereign Phi-3-mini-4k port with fused-tensor direct loading (concatenated Q/K/V and gate/up) matches a HuggingFace transformers f16 prefill-logits reference at cosine 1.0 with full top-1 agreement on a short stimulus; 8-bit reaches cosine 0.999996. A forward-pass smoke test.
Does not prove
- Long-context behaviour beyond 4k — long context is not wired.
- Generation quality, coherence, or downstream task performance.
- Decode throughput or latency.
- Phi-3 medium or larger variants.
- Benchmark-scale fidelity — top-1 agreement over one short stimulus is a smoke test, not a distributional evaluation.
- 8-bit top-1 agreement (not recorded).
Applies when
- f16 prefill-logits parity for Phi-3-mini at 4k (plain RoPE) on a consumer-class GPU versus a recent transformers reference with identical input ids.
Does not apply when
- Context beyond 4k, generation/throughput/latency claims, Phi-3 variants other than mini, or any quality (as opposed to numerical-fidelity) claim.
Authors
- Ran Tao — Investigation, Writing
Cite this
Citation
Tao, R., Octoryn Research. (2026). Phi-3-mini: fused-tensor direct loading in a sovereign inference engine (TR-2026-0017). Octopus Research Institute.
BibTeX
@techreport{oritr20260017,
title = {Phi-3-mini: fused-tensor direct loading in a sovereign inference engine},
author = {Tao, Ran and {Octoryn Research}},
institution = {Octopus Research Institute},
year = {2026},
note = {Permanent ID TR-2026-0017. Not peer reviewed.}
}Disclosures
- Funding
- Hardware and infrastructure provided by Octoryn / Octopus Core Pty Ltd.
- Conflicts of interest
- Octoryn ships commercial inference and governance tooling; findings are reported independently.