Status: Active research
Sovereign and Local AI
Local and on-device inference, model optimisation, edge deployment, privacy-aware inference and multi-provider portability.
Scope
We evaluate how far useful inference can run locally or at the edge, how models can be optimised for constrained hardware, and how workloads stay portable across providers — reducing dependence on any single external service.
Programmes in this area
- Status: ExperimentalApple Silicon Inference Runtimes
Kernel design, memory movement and bandwidth ceilings for local inference on Apple Silicon and other accelerators — consolidated from earlier Octoryn Research work.
- Status: Active researchLocal and Sovereign Inference
On-device and edge inference, model optimisation for constrained hardware, and multi-provider portability to reduce single-vendor dependence.
Open questions
- RQ-001HypothesisCan Apple GPUs support persistent-kernel global synchronisation?
Publications
- Apple-Silicon-friendly LLM architecture: substrate laws reverse-engineered from a model bake-off
- Gemma4-26B-A4B on Apple Silicon: a drop-in isomorphism falsified (NO-GO), with a conditional new-port ceiling
- A falsify-first root cause for a concurrent 4-bit decode crash: batch-composition KV-pool wipe trips a re-seed precondition
- DiffusionGemma-26B compatibility assessment: NEEDS_ARCH_SUPPORT on Apple Silicon (pre-port, external llama.cpp baseline)
- Diffusion-LLM Decode Quality: An Honest Regression, Negative Evidence, and a Canvas-Geometry Reattribution
- Federation continuity primitives: an event-sourced session log with deterministic replay, cross-machine replication, and self-healing recovery
- Sovereign gpt-oss-20B (MXFP4) on Apple Silicon: a fine-grained-MoE small-matrix occupancy ceiling
- Per-stream KV-slot isolation for parallel low-bit decode on Apple Silicon: crash fix with byte-identical solo parity and no-regression
- Batched resident MoE experts on Apple Silicon: a flat parallel expert pass, ~1.6x decode, parity bit-identical (single-run)
- Batched-concurrent 4-bit serving on Apple Silicon: weight-stream amortization with an honest sublinear ceiling
- Amortizing the per-token host-sync setup tax in fused all-layer q4 decode on Apple M1 Ultra (1.57x), then occupancy-bound
- A sovereign ARM-NEON CPU inference backend (Raspberry Pi 4B): Q8 and multicore are complementary, not additive
- Independent GGUF Forward Pass and Bit-Identical Cross-Backend Consistency for a 26B Text-Diffusion Model (forward only, not full decode)
- Multi-card inference topology: request-boundary aggregation on a latency-bound LAN
- Single-stream int4 decode of a 30B MoE on Apple Silicon: steady-state throughput with byte-identical, vendor-BLAS-free parity
- Cross-chip determinism: 96.875% byte-identical decode across two chips, live
- Resident packed-int4 decode: ~5x over host-streaming on a 24GB-class GPU
- FP8 grouped-expert prefill with on-chip int4-to-e4m3 dequant
- A DiT self-attention flash kernel on RDNA2: 9.4x op, 4.67x end-to-end
- An honest red result: EOS fixed, but forward degradation persisted
- ASR phase 1: a sovereign Whisper-tiny path landed
- A sovereign dense engine reproduces HF logits across 16 transformer families
- Cross-box tensor-parallel decode is latency-bound on commodity 2.5GbE
- Porting Mamba (selective-scan SSM) into a sovereign sequence-mixing engine
- RWKV in a self-built engine: a second non-attention paradigm (WKV/RNN)
- Bloom in a sovereign engine: ALiBi as the third positional paradigm
- Falcon in a sovereign engine: the first parallel attention+MLP block
- Fully-sovereign text-to-video on Apple Silicon (Wan2.1), parity to diffusers
- OLMoE: the first sparse-MoE in a dense sovereign engine (and an eps trap)
- Granite: four scalar multipliers, and what a negative control cannot see
- OLMo-2: reordered (post) normalization and a loader-clobber root-cause
- StableLM-2: partial RoPE and a first LayerNorm-with-bias path in a dense ROCm engine
- Phi-3-mini: fused-tensor direct loading in a sovereign inference engine
- The direct HF-logits parity harness (and the compressed-tensors version trap)
- Continuous batching with paged KV: a dependency-free CUDA throughput path
- A multi-week debugging retrospective on an Apple Silicon 4-bit inference engine: command-buffer folding, an encoder leak, and cold-bank reservation
- True standalone text-to-video on gfx1030, sovereign (no vendor BLAS)
- True CUDA text-to-video with live UMT5 encode, dependency-minimal (no cuBLAS)
- 1024px image generation end-to-end on a consumer RDNA2 GPU, and a NULL-deref segfault root-caused
- Wan 3D-VAE decode parity on CPU (Swift): pinning the layout contract
- Loading image/video weights directly from HF safetensors (portability fix)
- A sovereign DP router with mid-stream byte-identical failover
- On the Practical Limits of Local Inference: A Research Note