Loading image/video weights directly from HF safetensors (portability fix)
Ran Tao (Octoryn Research)
Abstract
Portability fix: the CUDA image/video generation pipelines previously required box-specific prebaked weight layouts and could not run on a fresh consumer-class (sm_89) CUDA box. The video DiT and VAE now load directly from public Hugging Face safetensors (byte-identical keys), and the image pipeline rebuilds its weight root from public HF SDXL on any box. Verified parity vs the prior pipeline: video frames cosine 1.0, image 1024px cosine 0.99999995, self-contained runtime. The image VAE needs the fp32 shard (fp16 degrades decode). A correctness smoke test, not a throughput or quality result.
Contribution
Claimed. This is an engineering / portability fix that removes a hard blocker: the CUDA image and video generation pipelines previously depended on prebaked, box-specific weight layouts and were therefore unrunnable on a fresh consumer-class (sm_89) CUDA box. After this change, (a) the video DiT and VAE load directly from public Hugging Face safetensors because the keys are byte-identical to the prebaked layout, gated behind a single load-mode flag, and (b) the image pipeline rebuilds its full weight root from public HF SDXL weights via a portable preparation step on any box. Both paths were validated on a consumer-class sm_89 GPU against the prior pipeline outputs: video frames cosine 1.0, image 1024px cosine 0.99999995, with a self-contained runtime dependency graph (no vendor BLAS library linked).
NOT claimed. This is a weight-loading / portability correctness result, not a novel algorithm and not a performance or quality result. It does not establish any throughput or token-rate number, does not measure or assert generation quality beyond the numeric parity, and does not establish portability to non-CUDA backends. The parity figures are a correctness smoke test (one image resolution and a frames-level video check), not a benchmark-scale evaluation. Altitude: low — this is plumbing that makes already-validated pipelines reproducible from public weights.
Methods
| Field | Value |
|---|---|
| Hardware | Consumer-class NVIDIA GPU (sm_89, 8 GB) |
| Stack / versions | Not recorded in source notes (CUDA pipeline; self-contained build) |
| Oracle / baseline | Prior prebaked-layout pipeline outputs (same engine, pre-portability weights) |
| Baseline version | Not recorded in source notes |
| Procedure (video) | Load DiT and VAE directly from public HF safetensors via a load-mode flag; keys byte-identical to the prebaked layout, no re-baking |
| Procedure (image) | Portable preparation step rebuilds the full weight root from public HF SDXL on the box |
| Precision | Image VAE: fp32 shard (required); other precisions not recorded in source notes |
| Metric + gate | Cosine vs prior pipeline output; no explicit numeric gate recorded in source notes |
| Stimulus / inputs | Image: 1024px generation. Video: frame output. Specific prompts not recorded in source notes |
Results
| Pipeline | Metric | Value |
|---|---|---|
| Video (DiT+VAE from HF) | Frames cosine vs prior pipeline | 1.0 |
| Image (root from HF SDXL) | 1024px cosine vs prior pipeline | 0.99999995 |
| Both | Runtime dependency graph | Self-contained (no vendor BLAS library) |
Ablations / Negative controls
| Variable | Setting | Outcome |
|---|---|---|
| Image VAE shard | fp16 shard | Degrades the decode (incorrect) |
| Image VAE shard | fp32 shard | Correct decode (required for the 0.99999995 parity) |
This is the single load-bearing observation the notes record: the image VAE must use the fp32 shard, not fp16.
Threats to validity / Limitations
- Sample size is a smoke test, not a benchmark. The image result is a single resolution (1024px) and the video result is a frames-level cosine; the source records no multi-prompt, multi-seed, or multi-resolution sweep. Treat both as correctness spot-checks.
- Single run, no variance. Each cosine is a point measurement; no repeated runs, confidence interval, or seed variance is recorded.
- Parity is not quality. Cosine 1.0 / 0.99999995 against the prior pipeline shows the portable load reproduces the prior numeric output; it says nothing about perceptual generation quality, which was not assessed here.
- Cosine 0.99999995 is not bit-exact. The image path is near-identical but not byte-identical (unlike the byte-identical video keys); the residual is unexplained in the source.
- Scope bound to CUDA sm_89. Verified only on a consumer-class sm_89 target; non-CUDA backends are out of scope and unproven.
- fp32 image-VAE constraint. The fp16 image-VAE shard is known to degrade decode; the portability claim holds only with the fp32 shard.
- Stack versions absent. Exact CUDA / diffusers / HF-weight-revision pins are not recorded in the source notes, weakening exact reproducibility.
- Evidence level. Given the single-resolution / single-run nature, readers should read the result as "the portable path was re-run and matched the prior pipeline," not as an independent multi-environment reproduction.
Reproducibility / Provenance
- Video: enable the HF-direct load mode to load the DiT and VAE directly from public HF safetensors (keys byte-identical to the prebaked layout).
- Image: run the portable preparation step to rebuild the full weight root from public HF SDXL; the VAE step must use the fp32 shard.
- Validate first: confirm against the prior prebaked-layout pipeline output (the oracle here) before trusting a fresh-box run.
- Build: compiled for the sm_89 target with standard optimization flags.
- Self-containment: the runtime dependency graph links only the core CUDA runtime, with no vendor BLAS / DNN library.
- Exact run commands and version pins: not recorded in source notes.
Claim boundary
The author's explicit scope — what this work does and does not establish — carried over from the Octoryn Research publishing model.
Proves
- Video DiT+VAE load directly from public HF safetensors (byte-identical keys) and the image pipeline rebuilds its root from public HF SDXL, making both portable to a fresh CUDA box; verified parity vs the prior pipeline (video frames cosine 1.0, image 1024px cosine 0.99999995).
- The image VAE must use the fp32 shard for a correct decode; the fp16 shard degrades it.
Does not prove
- Any throughput, token-rate, or perceptual generation-quality result beyond the numeric parity figures.
- Portability to non-CUDA backends, which would require separate work.
- Benchmark-scale correctness: the evidence is a single 1024px image cosine and a frames-level video cosine, single-run, with no seed or resolution sweep.
Applies when
- Reproducing the image/video CUDA generation pipelines from public HF weights on a fresh consumer-class (sm_89) CUDA box.
- Treating cosine parity against a prior pipeline output as a weight-loading correctness spot-check.
Does not apply when
- Making throughput or perceptual-quality claims, using the fp16 image-VAE shard, targeting non-CUDA backends, or extrapolating beyond the single tested image resolution and frames-level video check.
Authors
- Ran Tao — Investigation, Writing
Cite this
Citation
Tao, R., Octoryn Research. (2026). Loading image/video weights directly from HF safetensors (portability fix) (TR-2026-0025). Octopus Research Institute.
BibTeX
@techreport{oritr20260025,
title = {Loading image/video weights directly from HF safetensors (portability fix)},
author = {Tao, Ran and {Octoryn Research}},
institution = {Octopus Research Institute},
year = {2026},
note = {Permanent ID TR-2026-0025. Not peer reviewed.}
}Disclosures
- Funding
- Hardware and infrastructure provided by Octoryn / Octopus Core Pty Ltd.
- Conflicts of interest
- Octoryn ships commercial inference and governance tooling; findings are reported independently.