Review round on the video backend:
- the resident memory check now budgets transformer plus companions like the
image backend, instead of letting auto pick a resident placement that OOMs
while the LTX text encoder and VAEs load
- a new load waits for the signalled in flight generation to exit before
tearing down the old pipeline, so two models never share VRAM during a swap
- the load worker rechecks its token right before placement, narrowing the
window where a cancelled load could put weights on a GPU the arbiter
already handed to another backend
- the step cache installs before the speed profile and compile now keys
fullgraph off an active cache, matching the image order; compiling
fullgraph first crashed the first cached generation
- teardown uninstalls the process wide compiled GGUF dequantizer so a later
speed off load gets the bit identical path
- explicit base_repo goes through the same trust gate as non GGUF repo ids,
and local checkpoint paths are verified during validation, before the
route evicts a resident model
- status reports only the speed optimisations that actually engaged
- the gallery file route streams via FileResponse with range support instead
of buffering whole clips
- the build step reuses the checkpoint path resolved during planning
Review follow ups on the more-families branch: the per channel scale now
broadcasts rank aware instead of assuming 2D (all shipped tensors are 2D,
verified across all three fp8 components, but a future non 2D quantized
tensor would have mis broadcast silently), the fused qkv split asserts the
expected 3x hidden row count so a GQA style export fails loudly, fp8
detection scans every shard header rather than the first, and the excluded
model match uses the segment aware token helper with a hunyuanimage-3
token so a future HunyuanImage 2.x is not blocked with a 3.0 reason.
logs/ (a 1.3 MB ComfyUI object_info dump plus stale PID files), temp/ (PR body
and commit message scratch), and async_task_outputs/ (agent task transcripts)
are environment specific runtime artifacts that were committed by accident and
carry stale local state into every checkout. Remove them and gitignore the
directories so they cannot be re-added.
The generic Studio config dict path can deliver these flags as strings, and a
non-empty string like "false" is truthy, so an opt-out silently no-ops (the
latent cache still builds, TF32 stays on). Coerce them the same way
gradient_checkpointing already is.
The wheel-only pip install for an optional attention kernel ran inside
load_pipeline under _lock and _generate_lock, so a slow or hanging install
blocked unload and cancellation for up to the 600s timeout. Resolve and install
the kernel before taking the locks (only an explicit backend ever pulls a
package, and its resolution ignores the speed tier); the in-lock apply call is
then a fast no-op. Also decode and log pip's stderr on a failed install so the
fallback to native is diagnosable instead of showing only the exit code.
The raw speed_mode string was forwarded to _enable_fp16_accumulation, so a
case-variant like MAX failed the speed_mode != SPEED_MAX check and wrongly
disabled fp16 accumulation on float16 pipelines. Forward the normalized mode
and cover the case-insensitive path in the test.
The video backend never cleared FBCache residuals between clips, so with the
step cache engaged a second generation at a different resolution would hit
stale state from the first. Mirrors the image backend fix from #6872: call
the transformer level _reset_stateful_cache (reset_stateful_hooks only exists
on the HookRegistry in diffusers 0.39), covering transformer_2 for the Wan
dual expert, only when a cache is engaged.
The 1.22.0 release adds _sandbox.py (the client for HF Jobs sandboxes,
including the bootstrap that downloads HF's own sandbox server binary)
and the scanner flags it plus three long-standing while True retry and
pagination loops as CRITICAL. Reviewed all four against the upstream
repo and the 1.22.0 wheel: legitimate library code. Entries generated
with scan_packages.py --write-baseline and verified to suppress with
exit 0.
The ideogram-ai/ideogram-4-fp8 repo stores its two DiTs and the Qwen3-VL text
encoder in a vendor float8 layout that diffusers 0.39.0 (and diffusers main)
cannot read, so a stock Ideogram4Pipeline.from_pretrained produced a pipeline
with randomly initialized attention weights left on the meta device: the load
then died at pipe.to(device) with "Cannot copy out of meta tensor", and any load
that got past that would have generated noise.
Two things broke:
- The DiT attention is stored FUSED as attention.qkv.weight ([3*hidden, hidden],
Q/K/V rows stacked) plus attention.o.weight, while the diffusers transformer has
split to_q/to_k/to_v/to_out.0. from_pretrained mapped neither name and left them
meta + random.
- Every quantized weight is float8_e4m3 with a per-output-channel weight_scale;
the real weight is fp8.float() * weight_scale[:, None]. diffusers dropped the
scales and loaded the raw fp8 values (range +-448) as the weights, so even the
weights that did map were wrong.
load_ideogram4_transformer now reads the shards, dequantizes every scaled weight,
splits the fused qkv into to_q/to_k/to_v and renames o to to_out.0, then loads the
result into a config-constructed model. It fails loudly if any key stays unmatched
so a partly random model can never ship. The dequantized fp8 projections match the
byte-identical -nf4 export (already in the diffusers split layout with a bnb
quantization_config) to cosine ~0.997, so the split order and scale axis are
confirmed. The conversion is gated on the fp8 marker (a *.weight_scale key) read
from the shard header only, so the -nf4 repos skip it and load through the stock
from_pretrained path without a wasteful full-shard read.
The fp8 text encoder needed the same float8 dequant (its keys already match the
transformers Qwen3-VL module, so no rename). load_ideogram4_text_encoder handles
the fp8 repo and delegates the bnb-4bit and dense repos to the shared krea shim.
One more incompatibility was in the diffusers pipeline itself: it calls
transformers create_causal_mask(inputs_embeds = ...) with no cache_position, but
on transformers 4.57.6 the parameter is spelled input_embeds and cache_position is
required. _patch_create_causal_mask installs a signature-aware wrapper that renames
the kwarg and supplies cache_position, and is self-disabling on a matching signature.
Adds unit tests for the fp8 dequant/split conversion and the causal-mask patch.
Verified live on a B200: ideogram-4-fp8 (both CFG paths), ideogram-4-nf4-diffusers,
and krea-2 with the retroanime LoRA all load and generate coherent images.
The Lightricks/LTX-2.3-fp8 checkpoints store float8 weights with
per-tensor weight_scale and input_scale companions (verified from the
file headers: 1496 F8_E4M3 tensors, 2924 scale tensors). A plain dtype
cast would silently corrupt every quantized layer, so the 2.3 assembly
now detects the companions and raises with a pointer to the GGUF quants,
which offer comparable fidelity through the supported path. Dequantizing
the scaled fp8 layout is a possible follow-up.
diffusers 0.39 ships every LTX-2.3 model class but its single-file loader
maps all LTX-2 checkpoints to the 2.0 config, so 2.3 checkpoints (9-row
modulation tables, gated attention, per-modality connectors) fail a shape
check at load. The community transformer-only GGUFs also lack the text
projections, VAEs, and vocoder that 2.3 moved out of the transformer.
New core/inference/video_ltx2.py detects a 2.3 checkpoint from its header
(6 vs 9 modulation rows, no weight data read) and assembles the full
pipeline: the DiT through from_single_file with the 2.3 config overrides
and the prompt_adaln key renames the stock converter lacks, the 8-layer
per-modality connectors from the same checkpoint plus the text projection
companion file, and the 2.3 video VAE, audio VAE, and BWE vocoder from
the companion files in unsloth/LTX-2.3-GGUF. Configs and rename tables
mirror diffusers' own scripts/convert_ltx2_to_diffusers.py, which the
library loader has not absorbed yet. Assembled through the constructor
because the base repo pins LTX2Vocoder while 2.3 needs LTX2VocoderWithBWE
and the from_pretrained type gate rejects the substitution.
Verified on a B200: distilled-1.1 Q4_K_M GGUF loads in 37s, generates a
49-frame 768x512 clip with synchronized audio in 18s (8 steps), frames
on-prompt and non-black, container decodes fully. Meta-tensor validation
confirms exact key and shape match for all five converted components.
Unit tests cover 2.3 detection (gguf + safetensors headers), combined
checkpoint partitioning, and companion-set choice.
Measured on the fresh linux x64 prebuilt (z-image Q8_0, sd-cli, 192 CPU
threads, 512x512, 9 steps, steady state): sampling 56.1s vs 51.3s (about
9 percent faster), VAE decode unchanged, peak RSS identical. The sd.cpp
engine only serves the no-GPU tier, so the default profile now matches
max: --diffusion-fa plus --diffusion-conv-direct.
routes/video.py mirrors the /images/* routes one-for-one: validate-before-evict
load ordering (a bad pick must not evict a working chat model and then 400),
the training-active interlock, the device-gated GPU arbiter handoff with the
new VIDEO owner, the exact-match sentinel mapping (VIDEO_NOT_LOADED_MSG /
VIDEO_CANCELLED_MSG to 409, ValueError/FileNotFoundError to 400 with native
paths redacted, everything else a sanitized 500), and the gallery CRUD shape
with fetch-one-extra has_more paging. Generate persists the encoded MP4 plus
its full recipe through video_gallery.save and returns the gallery record; the
file endpoint serves video/mp4 with an immutable Cache-Control, 404 on any id
that fails the containment check.
models/inference.py gains the video request/response set (VideoLoadRequest,
VideoGenerateRequest/Response, GalleryVideo, gallery list, both progress
shapes, VideoGenerationDefaults nested in VideoStatusResponse), reusing
DiffusionResolvedControl for the resolved-provenance badges. The router is
registered in main.py after the images router under the same /api/inference
prefix and auth dependency.
Tests: 20 route tests on a stubbed backend + real tmp gallery (load happy path
and arbiter acquisition, 400/409 mappings, generate persistence round trip,
mp4 file serving + 404, delete/clear, unload releases ownership); the
diffusion route suite stays green after the shared models edit.
Text-to-video lands as a SIBLING of the image diffusion backend, not a mode of
it: video pipelines take frame/fps arguments, return frame stacks plus, for
LTX-2, synchronized audio, and persist MP4s -- none of the image module's
img2img/inpaint/ControlNet/LoRA surface applies. The image backend's hardware
and optimisation layers are imported unchanged (device/dtype resolution, memory
planning + offload tiers, attention backends, speed profiles, FBCache), and the
load-token/cancel-event concurrency skeleton is copied verbatim so lifecycle
behaviour cannot diverge.
core/inference/video_families.py: a pure VideoFamily registry (no torch) with
the ltx-2 entry -- LTX2Pipeline + LTX2VideoTransformer3DModel, base
Lightricks/LTX-2, unsloth/LTX-2.3-GGUF as the curated GGUF source, audio on,
frame lattice k*8+1, /32 resolutions with a vertical preset, and measured bf16
component sizes (the Gemma3-27B text encoder outweighs the 19B DiT itself).
MoE fields (transformer_2, guidance_scale_2) are declared now so the Wan2.2
A14B family lands later without churning the schema.
core/inference/video.py: VideoBackend with async begin_load + cache-scan
download progress, GGUF / single-file / full-pipeline loads (the GGUF DiT
assembles onto the base repo exactly like the image path), generation with
frame/size snapping BEFORE latents allocate, per-step progress + ETA and
cooperative cancel via the standard diffusers callback, and MP4 (H.264) export
through diffusers' PyAV encoder with the audio track muxed when the family
produces one. VAE tiling is always on: decoding a 100+ frame clip is the
memory peak, and the frames-aware estimate_video_runtime_mib (new, in
diffusion_memory) feeds the planner where the pixel-area image estimate would
badly undershoot. Loads are gated to unsloth/*, the official Lightricks base
repos, or local paths; PyAV availability is checked at load time so a missing
encoder cannot fail a clip after a multi-minute denoise.
core/inference/video_gallery.py: {id}.mp4 + {id}.json recipe sidecar pairs
under studio_root()/videos (an MP4 has no PNG text chunk to embed the recipe
in), with the image gallery's id/containment guards, newest-first listing that
skips orphans, delete/clear.
gpu_arbiter gains the VIDEO owner: ownership is exclusive, so the existing
evict-the-current-owner already generalises to chat/image/video all evicting
each other. The av (PyAV) dependency joins requirements/studio.txt.
Tests: video family detection/snapping/defaults, backend lifecycle on a faked
torch/diffusers runtime (GGUF assembly, shape snapping, distilled defaults,
cancel/progress, sentinel), gallery roundtrip/containment/orphans. 52 new
tests green plus the arbiter suite.
Ideogram 4 (diffusers 0.39 Ideogram4Pipeline) as a new image family. The vendor
publishes no bf16 checkpoint, so ideogram-ai/ideogram-4-fp8 (raw float8 DiTs,
upcast by from_pretrained) is the family base and ideogram-4-nf4-diffusers is
the bnb-4bit pipeline artifact (ideogram-4-nf4 is byte-identical and detects to
the same family). All three repos join the trusted non-GGUF allowlist and the
frontend safetensors catalog.
Family specifics handled:
- Dual-branch CFG runs through a SEPARATE unconditional_transformer, so the
auto-policy size table entry counts two ~9.3B DiTs (37.2 GB bf16), and the
pipeline-kind memory plan now takes max(cached bytes, family table) for the
family base repo: the fp8 repo's cached bytes undershoot the bf16-resident
footprint by ~2x, which would let auto planning pick a resident placement
that OOMs.
- The pipeline accepts EITHER guidance_scale OR a per-step guidance_schedule
(its default: the recommended 45x7.0 + 3x3.0 taper, valid only at 48 steps)
and raises when both are set. At the advertised defaults (48 steps, guidance
7) generate() drops the constant so the recommended taper engages; any other
request nulls the schedule so the constant broadcasts legally.
- Generation defaults per the model card: 48 steps, guidance 7 (both tables).
tencent/HunyuanImage-3.0 is deliberately excluded: it has no diffusers pipeline
(an 80B autoregressive MoE behind trust_remote_code). A structured exclusion
map now surfaces that reason verbatim from validate_load_request instead of
the generic unknown-family error.
The curated diffusion LoRA catalog gains the nine official krea/Krea-2-LoRA-*
style adapters (family-tagged krea-2, explicit weight filenames), so they show
up in the picker instead of requiring a typed repo id.
Tests: new test_diffusion_more_families.py (detection, trust, defaults, size
table, exclusion reason, curated catalog + family filter), two generate()
tests for the guidance_scale/guidance_schedule pairing, and the local-scan
LoRA test updated for a non-empty curated list. Backend suite + CI-sim
(block_diffusers/block_torchao) green; frontend builds.
