Review follow-ups on the image-generation PR:
- ControlNet: resolve_controlnet accepts a bare owner/name repo without the
non-GGUF base trust gate, and _controlnet_pipe hands it straight to
from_pretrained. A malicious pickle .bin would deserialize on load, so run
the same Hugging Face malware preflight (evaluate_file_security) the chat and
export loaders use before any remote ControlNet load; local dirs are exempt.
- Dataset thumbnails: key the cache on the full filename instead of the stem so
sample.png and sample.jpg no longer collide on one .thumbs file (which could
serve or delete the wrong image); the delete cleanup globs the same key.
- Diffusion training start: mirror start_training's API-key guard so an API
client cannot start training (which frees VRAM by unloading chat) while an
inference request is streaming; it now returns 409 before any GPU is freed.
- Model picker: include the curated safetensors row keys in the recommended
roving key list so arrow-key navigation reaches those rows instead of hitting
the duplicate option-missing id.
Tests: ControlNet malware gate (remote blocked before from_pretrained, local
skipped), thumbnail same-stem cache separation, API-key diffusion-start 409
before GPU free. Full diffusion suites green.
- Free reserved VRAM in the diffusion load worker's failure path: a load-time OOM
never commits _state and the next load's _unload_locked early-returns, so nothing
else reclaimed the half-built pipeline's memory
- Use a monotonic clock for the denoise ETA rate
- Sync _GENERATION_DEFAULTS with the UI table: kontext, flux.2-dev, sdxl-turbo and
SDXL base rows so /v1/images/generations stops falling back to 9 steps / CFG 0
- 400 (not sanitized 500) when /v1/images/generations hits an edit-only model
- Fail fast on pre-Ampere CUDA in the DiT trainer instead of dying in model load
- Run the trainer trust gate in the diffusion training route before freeing GPU
residents so an untrusted base cannot tear down loaded chat/Images models
- Protect native sd.cpp companion VAE/text-encoder repos from cache deletion while
a load is downloading them
- Exempt the task-scoped Images picker from the chat-only GGUF/MLX format gate so
local diffusers pipelines stay selectable on no-GPU hosts
Tag the ltxv and wan GGUF archs text-to-video so cached video checkpoints
actually surface in the Video picker (they were classed unsupported and
hidden everywhere). Adopt the loaded family's default clip length instead
of silently keeping the 25-frame pre-load fallback, and derive steps and
guidance from the picked GGUF filename so a distilled variant gets its
few-step schedule. Suppress the error toast for the user's own Cancel and
disable the Video nav item on chat-only hosts with a hint, matching Train.
Register two new video families and wire them through the backend, routes,
and frontend picker: wan2.2-ti2v-5b (single DiT) and wan2.2-t2v-a14b (the
dual-expert MoE). Both share diffusers' WanPipeline + WanTransformer3DModel
+ AutoencoderKLWan, which the VideoFamily dataclass already reserved fields
for (transformer2_class, is_moe, cfg2_kwarg).
Verified against the installed diffusers 0.39.0 before writing code:
- WanPipeline, WanTransformer3DModel, and AutoencoderKLWan are all exported
from top-level diffusers 0.39.0.
- WanPipeline.__call__ (pipeline_wan.py:383) defaults to num_frames=81,
num_inference_steps=50, guidance_scale=5.0. guidance_scale_2 DOES exist
in 0.39 (line 392) and its check_inputs raises if it is passed when the
pipeline's boundary_ratio is None (line 322), so the second guidance is
threaded ONLY for the MoE family and only when inspect.signature accepts
it (the same gate frame_rate already uses).
- The Wan VAE temporal factor is 4 (autoencoder_kl_wan.py scale_factor_temporal),
and the pipeline snaps num_frames to 4k+1 (line 493), so frame_step is 4,
unlike LTX-2's 8k+1. Sizes patchify at spatial 8 * patch 2 = 16, so
resolution_multiple is 16.
- boundary_ratio and transformer_2 come from model_index.json: TI2V-5B ships
boundary_ratio=null and transformer_2=[null,null] (single DiT), while A14B
ships boundary_ratio=0.875 and transformer_2=WanTransformer3DModel (dual
DiT). boundary_ratio lives in the pipeline config, so it needs no per-call
plumbing.
- WanTransformer3DModel declares _repeated_blocks=["WanTransformerBlock"] and
inherits CacheMixin (transformer_wan.py:508/551), so regional compile and
First-Block-Cache both work.
bf16-resident component sizes, measured from each diffusers repo's on-disk
safetensors (all stored bf16), feed the auto memory table:
TI2V-5B: transformer 20.0, UMT5 text encoder 11.4, VAE 2.8 GB.
A14B: two experts 57.2 each (114.3 total), text encoder 11.4, VAE 0.5 GB.
Backend changes make the optimisation layers dual-DiT aware: a small
_SecondDiTView proxy presents transformer_2 as pipe.transformer so the
existing single-DiT helpers (apply_speed_optims, apply_attention_backend,
apply_step_cache, quantize_transformer) cover BOTH experts on an is_moe load
without forking any helper; single-DiT loads are unchanged (views is just
(pipe,)). The two Wan base repos are added to the trusted non-GGUF allowlist.
A transformer_quant option is added to the load path, mirroring the image
backend's dense torchao fast path: on a pipeline-kind load the dense DiT(s)
are quantised in place onto the low-precision tensor cores and the engaged
scheme is surfaced in status. generate() threads guidance_2 through the
family's cfg2_kwarg when the loaded pipeline accepts it.
Routes and Pydantic models gain the optional transformer_quant (load /
status) and guidance_2 (generate) fields. The frontend picker gains the two
Wan models with 50-step / CFG 5.0 defaults; fps is supplied per family by
the backend.
Tests extend the fake runtime with WanPipeline and per-DiT transformer fakes
(single-DiT and dual-DiT), and cover family detection for both repos, 4k+1
frame snapping, default application, dual-DiT speed/cache/attention/quant
coverage on both experts, cfg2 threading gated on the pipeline signature,
trusted-repo validation, and the new route fields. Both the standard and the
diffusers/torchao-blocked CI-sim runs are green.
Add a Video generation page that mirrors the Images feature's create
workflow. It loads a text-to-video model, generates a clip, and plays it
back inline with the gallery of past clips.
- src/features/video/api.ts: typed client for the /api/inference/video
routes (load, load-progress, generate, generate-progress, cancel,
status, unload, gallery CRUD, and an auth-protected MP4 blob fetch).
- src/features/video/video-page.tsx: the page. Curated model picker
(LTX 2.3 distilled GGUF, LTX 2 base pipeline), prompt and negative
prompt, resolution preset select, duration select over the family's
temporal lattice, fixed fps display, steps and guidance sliders seeded
from per-model defaults, seed box. Generate polls per-step progress
with a phase label and ETA and a Cancel button, then plays the result
in a video player with a download button and an audio badge. Gallery
strip below with per-card delete and clear all. Right-docked Advanced
panel for memory, speed, attention, and step-cache with Auto badges
fed from the resolved status.
- Register the page: router child, /video route, sidebar nav item with a
video icon after Images, and the __root keep-alive mount so an
in-flight generation survives leaving the tab.
- Add the text-to-video task to the model picker so video models never
appear in the chat picker.
- Add the video nav label to the en and zh-CN locales.
Ideogram 4 (diffusers 0.39 Ideogram4Pipeline) as a new image family. The vendor
publishes no bf16 checkpoint, so ideogram-ai/ideogram-4-fp8 (raw float8 DiTs,
upcast by from_pretrained) is the family base and ideogram-4-nf4-diffusers is
the bnb-4bit pipeline artifact (ideogram-4-nf4 is byte-identical and detects to
the same family). All three repos join the trusted non-GGUF allowlist and the
frontend safetensors catalog.
Family specifics handled:
- Dual-branch CFG runs through a SEPARATE unconditional_transformer, so the
auto-policy size table entry counts two ~9.3B DiTs (37.2 GB bf16), and the
pipeline-kind memory plan now takes max(cached bytes, family table) for the
family base repo: the fp8 repo's cached bytes undershoot the bf16-resident
footprint by ~2x, which would let auto planning pick a resident placement
that OOMs.
- The pipeline accepts EITHER guidance_scale OR a per-step guidance_schedule
(its default: the recommended 45x7.0 + 3x3.0 taper, valid only at 48 steps)
and raises when both are set. At the advertised defaults (48 steps, guidance
7) generate() drops the constant so the recommended taper engages; any other
request nulls the schedule so the constant broadcasts legally.
- Generation defaults per the model card: 48 steps, guidance 7 (both tables).
tencent/HunyuanImage-3.0 is deliberately excluded: it has no diffusers pipeline
(an 80B autoregressive MoE behind trust_remote_code). A structured exclusion
map now surfaces that reason verbatim from validate_load_request instead of
the generic unknown-family error.
The curated diffusion LoRA catalog gains the nine official krea/Krea-2-LoRA-*
style adapters (family-tagged krea-2, explicit weight filenames), so they show
up in the picker instead of requiring a typed repo id.
Tests: new test_diffusion_more_families.py (detection, trust, defaults, size
table, exclusion reason, curated catalog + family filter), two generate()
tests for the guidance_scale/guidance_schedule pairing, and the local-scan
LoRA test updated for a non-empty curated list. Backend suite + CI-sim
(block_diffusers/block_torchao) green; frontend builds.
An unset transformer_quant used to mean off (run the GGUF as-is), so the
hardware ladder only engaged when auto was explicitly chosen and the panel
showed Off as the default. Unset (or auto) now hands the decision to the
ladder: a dense-capable GPU gets at least int8, data-center silicon fp8,
falling back to the GGUF when the device, VRAM, family deny table or disk
cannot take it. An explicit none/off pins GGUF-as-is and is now
expressible in the API (previously only omission meant off, so pinned-off
and unset were indistinguishable); an explicit scheme pins that scheme.
The dense candidate also gains a free-disk gate: with auto as the default
the bf16 base download (up to ~40 GB) must never wedge a nearly-full
model-cache disk, so the candidate is dropped (GGUF build kept) when free
space cannot hold it plus a 10 GiB margin. Unprobeable disk passes.
Frontend: the Dtype select defaults to Auto (fastest for GPU), keeps Off
as an explicit choice, and sends none through instead of omitting it.
Suite: 622 diffusion tests green (default-load test rewritten to the new
contract, explicit-off short-circuit covered), CI-sim green.
GET /api/inference/images/info returns each family's bf16 component sizes and
the estimated resident GB under bf16/int8/fp8/mxfp8/nvfp4, computed purely from
the auto-policy tables (no GPU probing, torch-free), so the panel can show the
Dtype tradeoff before anything is loaded.
DiffusionStatusResponse gains an additive resolved field: per-control
{value, source, reason} provenance the loader already records. The Advanced
panel renders a muted Auto: X pill next to Speed / Dtype / Attention / Memory /
Step cache / CPU offload when the backend decided that control (source auto),
with the reason as the tooltip; an explicit user choice renders no badge.
The always-visible description under the select is gone (the hint tooltip keeps the
full detail) and the no-model gallery placeholder now reads 'Select a diffusion model
to load'.
- Trainers emit the pre-clip gradient norm; the service keeps a bounded
grad_norm history and the Train tab renders a Grad Norm chart next to
Loss and LR
- Completed runs show 'Training complete' with a celebratory marker in
the success color instead of a plain status word
- metadata.jsonl caption keys now match on Windows (as_posix relative
paths) in both the trainer discovery and the dataset image records
- RMSNorm eager patch skips installation on torch builds without
F.rms_norm instead of failing at forward time
- GGUF compute description no longer says the GGUF is dequantised: the
INT8/FP8/FP4 modes load the base model's bf16 transformer and quantise
that directly; label no longer wraps in the Advanced panel
- list_diffusion_runs skips wrong-shape records and the runs route tolerates
per-record ValidationError so one bad file never breaks the panel
- max_steps: 0 epoch-mode sentinel no longer trips train_steps validation
before epochs are resolved
- numberField keeps an explicit 0 (Seed, LR warmup) instead of falling back
- previous-runs list refetches once more shortly after a terminal status so
the just-finished run appears even if the record write races the fetch
- Reset mixed precision to bf16 when the family changes to a DiT: an fp16/no
value left over from SDXL rode along in the DiT start payload and the backend
rejected it (dense base precisions require bf16 compute).
- Gate the dense base precisions behind the selected base: a bnb-4bit repo
disables bf16/int8/fp8 with a hint, and a dense selection auto-flips to auto so
the run does not fail at the validator.
- Re-arm the run-completion notification in onStart, so a second run notifies
even when its running phase is never observed by the poll.
