Tag the ltxv and wan GGUF archs text-to-video so cached video checkpoints
actually surface in the Video picker (they were classed unsupported and
hidden everywhere). Adopt the loaded family's default clip length instead
of silently keeping the 25-frame pre-load fallback, and derive steps and
guidance from the picked GGUF filename so a distilled variant gets its
few-step schedule. Suppress the error toast for the user's own Cancel and
disable the Video nav item on chat-only hosts with a hint, matching Train.
Review round on the video backend:
- the resident memory check now budgets transformer plus companions like the
image backend, instead of letting auto pick a resident placement that OOMs
while the LTX text encoder and VAEs load
- a new load waits for the signalled in flight generation to exit before
tearing down the old pipeline, so two models never share VRAM during a swap
- the load worker rechecks its token right before placement, narrowing the
window where a cancelled load could put weights on a GPU the arbiter
already handed to another backend
- the step cache installs before the speed profile and compile now keys
fullgraph off an active cache, matching the image order; compiling
fullgraph first crashed the first cached generation
- teardown uninstalls the process wide compiled GGUF dequantizer so a later
speed off load gets the bit identical path
- explicit base_repo goes through the same trust gate as non GGUF repo ids,
and local checkpoint paths are verified during validation, before the
route evicts a resident model
- status reports only the speed optimisations that actually engaged
- the gallery file route streams via FileResponse with range support instead
of buffering whole clips
- the build step reuses the checkpoint path resolved during planning
Review follow ups on the more-families branch: the per channel scale now
broadcasts rank aware instead of assuming 2D (all shipped tensors are 2D,
verified across all three fp8 components, but a future non 2D quantized
tensor would have mis broadcast silently), the fused qkv split asserts the
expected 3x hidden row count so a GQA style export fails loudly, fp8
detection scans every shard header rather than the first, and the excluded
model match uses the segment aware token helper with a hunyuanimage-3
token so a future HunyuanImage 2.x is not blocked with a 3.0 reason.
The generic Studio config dict path can deliver these flags as strings, and a
non-empty string like "false" is truthy, so an opt-out silently no-ops (the
latent cache still builds, TF32 stays on). Coerce them the same way
gradient_checkpointing already is.
The wheel-only pip install for an optional attention kernel ran inside
load_pipeline under _lock and _generate_lock, so a slow or hanging install
blocked unload and cancellation for up to the 600s timeout. Resolve and install
the kernel before taking the locks (only an explicit backend ever pulls a
package, and its resolution ignores the speed tier); the in-lock apply call is
then a fast no-op. Also decode and log pip's stderr on a failed install so the
fallback to native is diagnosable instead of showing only the exit code.
The raw speed_mode string was forwarded to _enable_fp16_accumulation, so a
case-variant like MAX failed the speed_mode != SPEED_MAX check and wrongly
disabled fp16 accumulation on float16 pipelines. Forward the normalized mode
and cover the case-insensitive path in the test.
The video backend never cleared FBCache residuals between clips, so with the
step cache engaged a second generation at a different resolution would hit
stale state from the first. Mirrors the image backend fix from #6872: call
the transformer level _reset_stateful_cache (reset_stateful_hooks only exists
on the HookRegistry in diffusers 0.39), covering transformer_2 for the Wan
dual expert, only when a cache is engaged.
The ideogram-ai/ideogram-4-fp8 repo stores its two DiTs and the Qwen3-VL text
encoder in a vendor float8 layout that diffusers 0.39.0 (and diffusers main)
cannot read, so a stock Ideogram4Pipeline.from_pretrained produced a pipeline
with randomly initialized attention weights left on the meta device: the load
then died at pipe.to(device) with "Cannot copy out of meta tensor", and any load
that got past that would have generated noise.
Two things broke:
- The DiT attention is stored FUSED as attention.qkv.weight ([3*hidden, hidden],
Q/K/V rows stacked) plus attention.o.weight, while the diffusers transformer has
split to_q/to_k/to_v/to_out.0. from_pretrained mapped neither name and left them
meta + random.
- Every quantized weight is float8_e4m3 with a per-output-channel weight_scale;
the real weight is fp8.float() * weight_scale[:, None]. diffusers dropped the
scales and loaded the raw fp8 values (range +-448) as the weights, so even the
weights that did map were wrong.
load_ideogram4_transformer now reads the shards, dequantizes every scaled weight,
splits the fused qkv into to_q/to_k/to_v and renames o to to_out.0, then loads the
result into a config-constructed model. It fails loudly if any key stays unmatched
so a partly random model can never ship. The dequantized fp8 projections match the
byte-identical -nf4 export (already in the diffusers split layout with a bnb
quantization_config) to cosine ~0.997, so the split order and scale axis are
confirmed. The conversion is gated on the fp8 marker (a *.weight_scale key) read
from the shard header only, so the -nf4 repos skip it and load through the stock
from_pretrained path without a wasteful full-shard read.
The fp8 text encoder needed the same float8 dequant (its keys already match the
transformers Qwen3-VL module, so no rename). load_ideogram4_text_encoder handles
the fp8 repo and delegates the bnb-4bit and dense repos to the shared krea shim.
One more incompatibility was in the diffusers pipeline itself: it calls
transformers create_causal_mask(inputs_embeds = ...) with no cache_position, but
on transformers 4.57.6 the parameter is spelled input_embeds and cache_position is
required. _patch_create_causal_mask installs a signature-aware wrapper that renames
the kwarg and supplies cache_position, and is self-disabling on a matching signature.
Adds unit tests for the fp8 dequant/split conversion and the causal-mask patch.
Verified live on a B200: ideogram-4-fp8 (both CFG paths), ideogram-4-nf4-diffusers,
and krea-2 with the retroanime LoRA all load and generate coherent images.
The Lightricks/LTX-2.3-fp8 checkpoints store float8 weights with
per-tensor weight_scale and input_scale companions (verified from the
file headers: 1496 F8_E4M3 tensors, 2924 scale tensors). A plain dtype
cast would silently corrupt every quantized layer, so the 2.3 assembly
now detects the companions and raises with a pointer to the GGUF quants,
which offer comparable fidelity through the supported path. Dequantizing
the scaled fp8 layout is a possible follow-up.
diffusers 0.39 ships every LTX-2.3 model class but its single-file loader
maps all LTX-2 checkpoints to the 2.0 config, so 2.3 checkpoints (9-row
modulation tables, gated attention, per-modality connectors) fail a shape
check at load. The community transformer-only GGUFs also lack the text
projections, VAEs, and vocoder that 2.3 moved out of the transformer.
New core/inference/video_ltx2.py detects a 2.3 checkpoint from its header
(6 vs 9 modulation rows, no weight data read) and assembles the full
pipeline: the DiT through from_single_file with the 2.3 config overrides
and the prompt_adaln key renames the stock converter lacks, the 8-layer
per-modality connectors from the same checkpoint plus the text projection
companion file, and the 2.3 video VAE, audio VAE, and BWE vocoder from
the companion files in unsloth/LTX-2.3-GGUF. Configs and rename tables
mirror diffusers' own scripts/convert_ltx2_to_diffusers.py, which the
library loader has not absorbed yet. Assembled through the constructor
because the base repo pins LTX2Vocoder while 2.3 needs LTX2VocoderWithBWE
and the from_pretrained type gate rejects the substitution.
Verified on a B200: distilled-1.1 Q4_K_M GGUF loads in 37s, generates a
49-frame 768x512 clip with synchronized audio in 18s (8 steps), frames
on-prompt and non-black, container decodes fully. Meta-tensor validation
confirms exact key and shape match for all five converted components.
Unit tests cover 2.3 detection (gguf + safetensors headers), combined
checkpoint partitioning, and companion-set choice.
Measured on the fresh linux x64 prebuilt (z-image Q8_0, sd-cli, 192 CPU
threads, 512x512, 9 steps, steady state): sampling 56.1s vs 51.3s (about
9 percent faster), VAE decode unchanged, peak RSS identical. The sd.cpp
engine only serves the no-GPU tier, so the default profile now matches
max: --diffusion-fa plus --diffusion-conv-direct.
Add a Video generation page that mirrors the Images feature's create
workflow. It loads a text-to-video model, generates a clip, and plays it
back inline with the gallery of past clips.
- src/features/video/api.ts: typed client for the /api/inference/video
routes (load, load-progress, generate, generate-progress, cancel,
status, unload, gallery CRUD, and an auth-protected MP4 blob fetch).
- src/features/video/video-page.tsx: the page. Curated model picker
(LTX 2.3 distilled GGUF, LTX 2 base pipeline), prompt and negative
prompt, resolution preset select, duration select over the family's
temporal lattice, fixed fps display, steps and guidance sliders seeded
from per-model defaults, seed box. Generate polls per-step progress
with a phase label and ETA and a Cancel button, then plays the result
in a video player with a download button and an audio badge. Gallery
strip below with per-card delete and clear all. Right-docked Advanced
panel for memory, speed, attention, and step-cache with Auto badges
fed from the resolved status.
- Register the page: router child, /video route, sidebar nav item with a
video icon after Images, and the __root keep-alive mount so an
in-flight generation survives leaving the tab.
- Add the text-to-video task to the model picker so video models never
appear in the chat picker.
- Add the video nav label to the en and zh-CN locales.
routes/video.py mirrors the /images/* routes one-for-one: validate-before-evict
load ordering (a bad pick must not evict a working chat model and then 400),
the training-active interlock, the device-gated GPU arbiter handoff with the
new VIDEO owner, the exact-match sentinel mapping (VIDEO_NOT_LOADED_MSG /
VIDEO_CANCELLED_MSG to 409, ValueError/FileNotFoundError to 400 with native
paths redacted, everything else a sanitized 500), and the gallery CRUD shape
with fetch-one-extra has_more paging. Generate persists the encoded MP4 plus
its full recipe through video_gallery.save and returns the gallery record; the
file endpoint serves video/mp4 with an immutable Cache-Control, 404 on any id
that fails the containment check.
models/inference.py gains the video request/response set (VideoLoadRequest,
VideoGenerateRequest/Response, GalleryVideo, gallery list, both progress
shapes, VideoGenerationDefaults nested in VideoStatusResponse), reusing
DiffusionResolvedControl for the resolved-provenance badges. The router is
registered in main.py after the images router under the same /api/inference
prefix and auth dependency.
Tests: 20 route tests on a stubbed backend + real tmp gallery (load happy path
and arbiter acquisition, 400/409 mappings, generate persistence round trip,
mp4 file serving + 404, delete/clear, unload releases ownership); the
diffusion route suite stays green after the shared models edit.
Text-to-video lands as a SIBLING of the image diffusion backend, not a mode of
it: video pipelines take frame/fps arguments, return frame stacks plus, for
LTX-2, synchronized audio, and persist MP4s -- none of the image module's
img2img/inpaint/ControlNet/LoRA surface applies. The image backend's hardware
and optimisation layers are imported unchanged (device/dtype resolution, memory
planning + offload tiers, attention backends, speed profiles, FBCache), and the
load-token/cancel-event concurrency skeleton is copied verbatim so lifecycle
behaviour cannot diverge.
core/inference/video_families.py: a pure VideoFamily registry (no torch) with
the ltx-2 entry -- LTX2Pipeline + LTX2VideoTransformer3DModel, base
Lightricks/LTX-2, unsloth/LTX-2.3-GGUF as the curated GGUF source, audio on,
frame lattice k*8+1, /32 resolutions with a vertical preset, and measured bf16
component sizes (the Gemma3-27B text encoder outweighs the 19B DiT itself).
MoE fields (transformer_2, guidance_scale_2) are declared now so the Wan2.2
A14B family lands later without churning the schema.
core/inference/video.py: VideoBackend with async begin_load + cache-scan
download progress, GGUF / single-file / full-pipeline loads (the GGUF DiT
assembles onto the base repo exactly like the image path), generation with
frame/size snapping BEFORE latents allocate, per-step progress + ETA and
cooperative cancel via the standard diffusers callback, and MP4 (H.264) export
through diffusers' PyAV encoder with the audio track muxed when the family
produces one. VAE tiling is always on: decoding a 100+ frame clip is the
memory peak, and the frames-aware estimate_video_runtime_mib (new, in
diffusion_memory) feeds the planner where the pixel-area image estimate would
badly undershoot. Loads are gated to unsloth/*, the official Lightricks base
repos, or local paths; PyAV availability is checked at load time so a missing
encoder cannot fail a clip after a multi-minute denoise.
core/inference/video_gallery.py: {id}.mp4 + {id}.json recipe sidecar pairs
under studio_root()/videos (an MP4 has no PNG text chunk to embed the recipe
in), with the image gallery's id/containment guards, newest-first listing that
skips orphans, delete/clear.
gpu_arbiter gains the VIDEO owner: ownership is exclusive, so the existing
evict-the-current-owner already generalises to chat/image/video all evicting
each other. The av (PyAV) dependency joins requirements/studio.txt.
Tests: video family detection/snapping/defaults, backend lifecycle on a faked
torch/diffusers runtime (GGUF assembly, shape snapping, distilled defaults,
cancel/progress, sentinel), gallery roundtrip/containment/orphans. 52 new
tests green plus the arbiter suite.
