Tag the ltxv and wan GGUF archs text-to-video so cached video checkpoints
actually surface in the Video picker (they were classed unsupported and
hidden everywhere). Adopt the loaded family's default clip length instead
of silently keeping the 25-frame pre-load fallback, and derive steps and
guidance from the picked GGUF filename so a distilled variant gets its
few-step schedule. Suppress the error toast for the user's own Cancel and
disable the Video nav item on chat-only hosts with a hint, matching Train.
The video backend never cleared FBCache residuals between clips, so with the
step cache engaged a second generation at a different resolution would hit
stale state from the first. Mirrors the image backend fix from #6872: call
the transformer level _reset_stateful_cache (reset_stateful_hooks only exists
on the HookRegistry in diffusers 0.39), covering transformer_2 for the Wan
dual expert, only when a cache is engaged.
The ideogram-ai/ideogram-4-fp8 repo stores its two DiTs and the Qwen3-VL text
encoder in a vendor float8 layout that diffusers 0.39.0 (and diffusers main)
cannot read, so a stock Ideogram4Pipeline.from_pretrained produced a pipeline
with randomly initialized attention weights left on the meta device: the load
then died at pipe.to(device) with "Cannot copy out of meta tensor", and any load
that got past that would have generated noise.
Two things broke:
- The DiT attention is stored FUSED as attention.qkv.weight ([3*hidden, hidden],
Q/K/V rows stacked) plus attention.o.weight, while the diffusers transformer has
split to_q/to_k/to_v/to_out.0. from_pretrained mapped neither name and left them
meta + random.
- Every quantized weight is float8_e4m3 with a per-output-channel weight_scale;
the real weight is fp8.float() * weight_scale[:, None]. diffusers dropped the
scales and loaded the raw fp8 values (range +-448) as the weights, so even the
weights that did map were wrong.
load_ideogram4_transformer now reads the shards, dequantizes every scaled weight,
splits the fused qkv into to_q/to_k/to_v and renames o to to_out.0, then loads the
result into a config-constructed model. It fails loudly if any key stays unmatched
so a partly random model can never ship. The dequantized fp8 projections match the
byte-identical -nf4 export (already in the diffusers split layout with a bnb
quantization_config) to cosine ~0.997, so the split order and scale axis are
confirmed. The conversion is gated on the fp8 marker (a *.weight_scale key) read
from the shard header only, so the -nf4 repos skip it and load through the stock
from_pretrained path without a wasteful full-shard read.
The fp8 text encoder needed the same float8 dequant (its keys already match the
transformers Qwen3-VL module, so no rename). load_ideogram4_text_encoder handles
the fp8 repo and delegates the bnb-4bit and dense repos to the shared krea shim.
One more incompatibility was in the diffusers pipeline itself: it calls
transformers create_causal_mask(inputs_embeds = ...) with no cache_position, but
on transformers 4.57.6 the parameter is spelled input_embeds and cache_position is
required. _patch_create_causal_mask installs a signature-aware wrapper that renames
the kwarg and supplies cache_position, and is self-disabling on a matching signature.
Adds unit tests for the fp8 dequant/split conversion and the causal-mask patch.
Verified live on a B200: ideogram-4-fp8 (both CFG paths), ideogram-4-nf4-diffusers,
and krea-2 with the retroanime LoRA all load and generate coherent images.
The Lightricks/LTX-2.3-fp8 checkpoints store float8 weights with
per-tensor weight_scale and input_scale companions (verified from the
file headers: 1496 F8_E4M3 tensors, 2924 scale tensors). A plain dtype
cast would silently corrupt every quantized layer, so the 2.3 assembly
now detects the companions and raises with a pointer to the GGUF quants,
which offer comparable fidelity through the supported path. Dequantizing
the scaled fp8 layout is a possible follow-up.
diffusers 0.39 ships every LTX-2.3 model class but its single-file loader
maps all LTX-2 checkpoints to the 2.0 config, so 2.3 checkpoints (9-row
modulation tables, gated attention, per-modality connectors) fail a shape
check at load. The community transformer-only GGUFs also lack the text
projections, VAEs, and vocoder that 2.3 moved out of the transformer.
New core/inference/video_ltx2.py detects a 2.3 checkpoint from its header
(6 vs 9 modulation rows, no weight data read) and assembles the full
pipeline: the DiT through from_single_file with the 2.3 config overrides
and the prompt_adaln key renames the stock converter lacks, the 8-layer
per-modality connectors from the same checkpoint plus the text projection
companion file, and the 2.3 video VAE, audio VAE, and BWE vocoder from
the companion files in unsloth/LTX-2.3-GGUF. Configs and rename tables
mirror diffusers' own scripts/convert_ltx2_to_diffusers.py, which the
library loader has not absorbed yet. Assembled through the constructor
because the base repo pins LTX2Vocoder while 2.3 needs LTX2VocoderWithBWE
and the from_pretrained type gate rejects the substitution.
Verified on a B200: distilled-1.1 Q4_K_M GGUF loads in 37s, generates a
49-frame 768x512 clip with synchronized audio in 18s (8 steps), frames
on-prompt and non-black, container decodes fully. Meta-tensor validation
confirms exact key and shape match for all five converted components.
Unit tests cover 2.3 detection (gguf + safetensors headers), combined
checkpoint partitioning, and companion-set choice.
Measured on the fresh linux x64 prebuilt (z-image Q8_0, sd-cli, 192 CPU
threads, 512x512, 9 steps, steady state): sampling 56.1s vs 51.3s (about
9 percent faster), VAE decode unchanged, peak RSS identical. The sd.cpp
engine only serves the no-GPU tier, so the default profile now matches
max: --diffusion-fa plus --diffusion-conv-direct.
Add a Video generation page that mirrors the Images feature's create
workflow. It loads a text-to-video model, generates a clip, and plays it
back inline with the gallery of past clips.
- src/features/video/api.ts: typed client for the /api/inference/video
routes (load, load-progress, generate, generate-progress, cancel,
status, unload, gallery CRUD, and an auth-protected MP4 blob fetch).
- src/features/video/video-page.tsx: the page. Curated model picker
(LTX 2.3 distilled GGUF, LTX 2 base pipeline), prompt and negative
prompt, resolution preset select, duration select over the family's
temporal lattice, fixed fps display, steps and guidance sliders seeded
from per-model defaults, seed box. Generate polls per-step progress
with a phase label and ETA and a Cancel button, then plays the result
in a video player with a download button and an audio badge. Gallery
strip below with per-card delete and clear all. Right-docked Advanced
panel for memory, speed, attention, and step-cache with Auto badges
fed from the resolved status.
- Register the page: router child, /video route, sidebar nav item with a
video icon after Images, and the __root keep-alive mount so an
in-flight generation survives leaving the tab.
- Add the text-to-video task to the model picker so video models never
appear in the chat picker.
- Add the video nav label to the en and zh-CN locales.
routes/video.py mirrors the /images/* routes one-for-one: validate-before-evict
load ordering (a bad pick must not evict a working chat model and then 400),
the training-active interlock, the device-gated GPU arbiter handoff with the
new VIDEO owner, the exact-match sentinel mapping (VIDEO_NOT_LOADED_MSG /
VIDEO_CANCELLED_MSG to 409, ValueError/FileNotFoundError to 400 with native
paths redacted, everything else a sanitized 500), and the gallery CRUD shape
with fetch-one-extra has_more paging. Generate persists the encoded MP4 plus
its full recipe through video_gallery.save and returns the gallery record; the
file endpoint serves video/mp4 with an immutable Cache-Control, 404 on any id
that fails the containment check.
models/inference.py gains the video request/response set (VideoLoadRequest,
VideoGenerateRequest/Response, GalleryVideo, gallery list, both progress
shapes, VideoGenerationDefaults nested in VideoStatusResponse), reusing
DiffusionResolvedControl for the resolved-provenance badges. The router is
registered in main.py after the images router under the same /api/inference
prefix and auth dependency.
Tests: 20 route tests on a stubbed backend + real tmp gallery (load happy path
and arbiter acquisition, 400/409 mappings, generate persistence round trip,
mp4 file serving + 404, delete/clear, unload releases ownership); the
diffusion route suite stays green after the shared models edit.
Text-to-video lands as a SIBLING of the image diffusion backend, not a mode of
it: video pipelines take frame/fps arguments, return frame stacks plus, for
LTX-2, synchronized audio, and persist MP4s -- none of the image module's
img2img/inpaint/ControlNet/LoRA surface applies. The image backend's hardware
and optimisation layers are imported unchanged (device/dtype resolution, memory
planning + offload tiers, attention backends, speed profiles, FBCache), and the
load-token/cancel-event concurrency skeleton is copied verbatim so lifecycle
behaviour cannot diverge.
core/inference/video_families.py: a pure VideoFamily registry (no torch) with
the ltx-2 entry -- LTX2Pipeline + LTX2VideoTransformer3DModel, base
Lightricks/LTX-2, unsloth/LTX-2.3-GGUF as the curated GGUF source, audio on,
frame lattice k*8+1, /32 resolutions with a vertical preset, and measured bf16
component sizes (the Gemma3-27B text encoder outweighs the 19B DiT itself).
MoE fields (transformer_2, guidance_scale_2) are declared now so the Wan2.2
A14B family lands later without churning the schema.
core/inference/video.py: VideoBackend with async begin_load + cache-scan
download progress, GGUF / single-file / full-pipeline loads (the GGUF DiT
assembles onto the base repo exactly like the image path), generation with
frame/size snapping BEFORE latents allocate, per-step progress + ETA and
cooperative cancel via the standard diffusers callback, and MP4 (H.264) export
through diffusers' PyAV encoder with the audio track muxed when the family
produces one. VAE tiling is always on: decoding a 100+ frame clip is the
memory peak, and the frames-aware estimate_video_runtime_mib (new, in
diffusion_memory) feeds the planner where the pixel-area image estimate would
badly undershoot. Loads are gated to unsloth/*, the official Lightricks base
repos, or local paths; PyAV availability is checked at load time so a missing
encoder cannot fail a clip after a multi-minute denoise.
core/inference/video_gallery.py: {id}.mp4 + {id}.json recipe sidecar pairs
under studio_root()/videos (an MP4 has no PNG text chunk to embed the recipe
in), with the image gallery's id/containment guards, newest-first listing that
skips orphans, delete/clear.
gpu_arbiter gains the VIDEO owner: ownership is exclusive, so the existing
evict-the-current-owner already generalises to chat/image/video all evicting
each other. The av (PyAV) dependency joins requirements/studio.txt.
Tests: video family detection/snapping/defaults, backend lifecycle on a faked
torch/diffusers runtime (GGUF assembly, shape snapping, distilled defaults,
cancel/progress, sentinel), gallery roundtrip/containment/orphans. 52 new
tests green plus the arbiter suite.
Ideogram 4 (diffusers 0.39 Ideogram4Pipeline) as a new image family. The vendor
publishes no bf16 checkpoint, so ideogram-ai/ideogram-4-fp8 (raw float8 DiTs,
upcast by from_pretrained) is the family base and ideogram-4-nf4-diffusers is
the bnb-4bit pipeline artifact (ideogram-4-nf4 is byte-identical and detects to
the same family). All three repos join the trusted non-GGUF allowlist and the
frontend safetensors catalog.
Family specifics handled:
- Dual-branch CFG runs through a SEPARATE unconditional_transformer, so the
auto-policy size table entry counts two ~9.3B DiTs (37.2 GB bf16), and the
pipeline-kind memory plan now takes max(cached bytes, family table) for the
family base repo: the fp8 repo's cached bytes undershoot the bf16-resident
footprint by ~2x, which would let auto planning pick a resident placement
that OOMs.
- The pipeline accepts EITHER guidance_scale OR a per-step guidance_schedule
(its default: the recommended 45x7.0 + 3x3.0 taper, valid only at 48 steps)
and raises when both are set. At the advertised defaults (48 steps, guidance
7) generate() drops the constant so the recommended taper engages; any other
request nulls the schedule so the constant broadcasts legally.
- Generation defaults per the model card: 48 steps, guidance 7 (both tables).
tencent/HunyuanImage-3.0 is deliberately excluded: it has no diffusers pipeline
(an 80B autoregressive MoE behind trust_remote_code). A structured exclusion
map now surfaces that reason verbatim from validate_load_request instead of
the generic unknown-family error.
The curated diffusion LoRA catalog gains the nine official krea/Krea-2-LoRA-*
style adapters (family-tagged krea-2, explicit weight filenames), so they show
up in the picker instead of requiring a typed repo id.
Tests: new test_diffusion_more_families.py (detection, trust, defaults, size
table, exclusion reason, curated catalog + family filter), two generate()
tests for the guidance_scale/guidance_schedule pairing, and the local-scan
LoRA test updated for a non-empty curated list. Backend suite + CI-sim
(block_diffusers/block_torchao) green; frontend builds.
An unset transformer_quant used to mean off (run the GGUF as-is), so the
hardware ladder only engaged when auto was explicitly chosen and the panel
showed Off as the default. Unset (or auto) now hands the decision to the
ladder: a dense-capable GPU gets at least int8, data-center silicon fp8,
falling back to the GGUF when the device, VRAM, family deny table or disk
cannot take it. An explicit none/off pins GGUF-as-is and is now
expressible in the API (previously only omission meant off, so pinned-off
and unset were indistinguishable); an explicit scheme pins that scheme.
The dense candidate also gains a free-disk gate: with auto as the default
the bf16 base download (up to ~40 GB) must never wedge a nearly-full
model-cache disk, so the candidate is dropped (GGUF build kept) when free
space cannot hold it plus a 10 GiB margin. Unprobeable disk passes.
Frontend: the Dtype select defaults to Auto (fastest for GPU), keeps Off
as an explicit choice, and sends none through instead of omitting it.
Suite: 622 diffusion tests green (default-load test rewritten to the new
contract, explicit-off short-circuit covered), CI-sim green.
