Two live-test findings on the video progress endpoints:
- load-progress downloaded_bytes froze mid-download: the counter used
scan_cache_dir, which skips in-flight *.incomplete blobs, so it sat at the
last completed blob for the whole multi-GB shard pull while the disk kept
filling. Count the repo's cache directory directly (completed plus incomplete
blobs, snapshot symlinks skipped so nothing is double-counted).
- generate-progress reported total_steps=null / fraction=0 while step advanced:
the video API only carried the native total field while the image API exposes
total_steps and fraction, so one poller could not work against both. Derive
the image-compatible aliases in generate_progress and declare them on the
response model; the native total stays for back-compat.
Wan-AI/Wan2.2-I2V-A14B-Diffusers is the image-to-video sibling of the already
supported T2V-A14B: the same dual-expert WanTransformer3DModel pair (boundary_ratio
0.9 in the pipeline config) behind WanImageToVideoPipeline, conditioning through the
VAE latent (no CLIP-vision image encoder in this 2.2 variant).
- New wan2.2-i2v-a14b family: image_conditioned flag, card recipe defaults (40 steps,
CFG 3.5, 81 frames at 16 fps), the T2V memory table (57.2 GB both experts bf16),
fp32-pinned VAE, and a wan2.2-i2v generation-defaults key ahead of the generic wan
50/5.0 entry.
- Source-image plumbing: /video/generate takes init_image (base64/data URL);
begin_generate 400s synchronously when an image-to-video family has no image or a
text-only family is given one, and generate() decodes and resizes the image to the
snapped output size before threading it as the pipeline's image kwarg. status()
reports image_input so the UI can gate the control.
- Trust the official -Diffusers repo for pipeline loads and transfer the measured wan
quant recipes: fp8 keeps condition_embedder in bf16 (zero padding-row scale), mxfp8
and nvfp4 stay denied, the UMT5 auto TE quant resolves dense, and the balanced
FBCache pin carries over. All tables share the T2V DiT profile.
- Tests: family detection/aliases/defaults, the dual-DiT image pipeline load, the
image gates on both begin_generate and generate, init_image route pass-through, and
the quant exclude/deny/auto entries.
GPU-verified on a B200: bf16 resident load (offload none, 72.8 GB peak) animating a
conditioning image at 832x480/33f in 28.7 s with first-frame MAE 5.7 vs the source,
and an int8 load with both experts quantized (43.1 GB peak, clean output).
Adapters are baked at load time: they attach to the dense transformer,
then quantize_ converts only the frozen base linears (the lora_ side
path is excluded by name), then the loader compiles. Post-quant PEFT
injection is not possible on a manually quantized module, so the
prequant shortcut is skipped for a baked load and the memory plan is
sized for the dense build (force_dense on the quant candidate).
At generation time the baked topology is frozen: weight tweaks and
disabling (scale 0 reproduces the quantized base exactly) go through
set_adapters, while adding or removing adapters returns a clean 400
telling the client to reload with the new selection.
supports_lora now returns True for int8/fp8 diffusers loads (checked
before the gguf-kind early return, since the quant fast path keeps the
picker kind); nvfp4/mxfp8 and GGUF-via-diffusers stay blocked. The
load request model takes an optional loras list, threaded through
begin_load on both engines (native ignores it and keeps applying LoRA
at generation).
Verified end to end on GPU: Z-Image GGUF picker + int8 + trained
adapter loads through the API, bake marker logged, weight 1.0 vs 0
renders differ visibly, weight 0.5 accepted live, unknown adapter
rejected as 400. Affected suites: 304 passed.
Cuts the shipped default's LPIPS vs the bit-exact reference from 0.224 to 0.139
while going faster (24.9 s to 21.2 s at 720p/33f/30 steps, 22.7x vs reference),
and makes the remaining speed/accuracy trade a user knob.
- inductor precision parity: set emulate_precision_casts=True for the regional
compile (fused pointwise kernels kept fp32 intermediates where eager rounds to
bf16 between ops); full-clip LPIPS vs bit-exact 0.221 to 0.052 at zero speed
cost. Snapshot/restored with the other process-wide backend flags.
- cache x compile composition fix: diffusers cache hooks are
torch.compiler.disable'd, so every COMPUTED step ran eager (1.69 vs 1.09
s/step) under MagCache/FBCache in both enable orders. Re-point each hook's
fn_ref.original_forward at a torch.compile'd wrapper of the same bound method
(armed only where the speed layer compiled the block; restored before every
disable_cache so the uncached path stays pristine). Balanced MagCache at 50
steps: 1.48x to 2.17x, identical skip counts, bit-identical uncached rerun
after enable/disable cycles.
- transformer_cache_quality knob (quality|balanced|fast; API + UI + bench)
mapping to (threshold, max_skip_steps, retention_ratio). Auto resolves to the
near-lossless quality preset (0.06, 2, 0.3; 1.63-1.64x at pairwise LPIPS
0.05-0.09) for the HunyuanVideo-1.5 families and to balanced (the pre-knob
values, byte-identical behaviour) everywhere else.
- TE auto-quant resolves dense for HunyuanVideo-1.5: TE fp8_dynamic alone moves
the clip to LPIPS 0.236 vs bit-exact for zero speed win (the quantised encoder
perturbs the conditioning and the trajectory amplifies it chaotically); VAE
fp8 stays in auto (0.053, at the compile floor). Explicit schemes honored.
- dual-GPU CFG branch parallelism (new diffusion_cfg_parallel.py): transformer
proxy + DiT replica on the most-free second CUDA device + worker thread,
branch-routed off the pipeline's own cache_context names. Auto engages only
where measured bit-identical (eager tier: max abs diff 0.0, 1.66x); the
compiled stack is explicit cfg_parallel=on (1.52x over the sequential
default; per-device compiled artifacts differ by 1 bf16 ulp/step, documented
in the resolved record). Fail-soft gates: family allowlist, guider CFG,
pipeline kind, dense DiT, no offload, free-VRAM check; single-GPU loads are
untouched and the memory plan stays single-device.
- video API: the transformer_cache literal now accepts auto/magcache (an
explicit magcache request was rejected at the pydantic layer); the mxfp8
family deny records the round-2 measurement (block-32 MX scaling fixes the
zero-row collapse, no black frames, but is latency-neutral at LPIPS 0.37:
fails both ship bars).
Measured on B200 via the production lever path (video_speedmem_bench.py, which
gained a --cache-quality lever and companion-quant isolation configs). Tests:
441 passing across the video inference suite (32 new for cfg-parallel, 20 for
presets/arming, 3 for the inductor flag, 2 for TE auto-dense); ruff clean.
POST /video/generate previously held the response open for the whole
generation (multi-minute for 720p), so in --secure mode the Cloudflare
quick tunnel's ~100s origin-response cap returned a 524 while the server
kept generating, and the frontend treated the run as failed.
Generation now follows the same return-at-once pattern as /video/load:
begin_generate validates synchronously (409 on no model or on a second
concurrent generate via a new busy sentinel) and runs the existing
generate + gallery-persist pipeline, with the route's exact error
mapping, on a daemon thread. GET /video/generate-progress gains optional
terminal fields: phase completed carries the saved gallery record, phase
failed a client-safe error; active only drops together with a terminal
phase. The cancel event is registered before the worker starts so
/video/generate/cancel keeps working across the whole job.
VideoGenerateResponse becomes an accepted acknowledgement (status
started, video kept as an always-null compat field). The video page
fires the POST, then drives completion off the progress poll it already
runs (completed prepends the clip, failed surfaces the error, the
cancelled sentinel stays toast-free). The API-key training-start guards
now also probe the video backend for an in-flight background clip, since
it is no longer visible as an in-flight HTTP request to the keep-warm
counter.
Route tests keep the fake backend for load/generate/status but inherit
the real job machinery, covering immediate accept, concurrent 409, the
terminal completed record, sanitized/ValueError/cancelled failures, and
cancel of a running job.
Review round follow-ups:
- Drop the machine-specific HF_HOME defaults from the four bench /
reproduction scripts (fp8_layer_ablation, hunyuan_int8_profile,
quant_accuracy_sweep, video_speedmem_bench); they pointed at a private
workspace cache and broke the scripts on any other machine. The
standard HF_HOME env override still applies.
- Correct the vae_quant 'auto' descriptions (image + video request
fields, select_vae_quant_scheme docstring, loader comment) to match
the shipped ladder: auto engages layerwise fp8 only; fp8_dynamic is an
explicit opt-in and is never picked automatically.
- Enforce _TE_FAMILY_SCHEME_DENY on the explicit text-encoder path too,
gating the final concrete mode (so an int8 -> fp8 fallback is
re-checked), matching the table's documented contract and the VAE
module's behavior. Covered by a new test.
Also merges origin/image-generation (single-GPU fit-budget fix) to keep
the stacked head self-consistent.
* Studio /v1/messages: accept thinking and unknown content blocks
The Anthropic-compatible /v1/messages endpoint modeled a message's content as
Union[str, list[{text|image|tool_use|tool_result}]], so any other block type
made Pydantic reject the whole request with
`messages.N.content.str: Input should be a valid string`. Resuming a Claude
session commonly replays assistant turns that carry `thinking` (extended
thinking) blocks, and sometimes a null content for a tool-only turn, both of
which tripped this and returned a 400.
Accept them:
- Add a permissive AnthropicUnknownBlock fallback (any block whose type is not
one of the four known ones), so thinking/redacted_thinking/provider-specific/
future blocks validate. A validator keeps known types on their typed models,
so a malformed known block (e.g. a tool_use without id) still fails cleanly.
- Coerce a null message (and tool_result) content to "" so the converter's
`for block in content` stays safe.
The converter already drops block types it does not translate, so a thinking
block is not forwarded to the model.
* Studio /v1/messages: keep user content validation strict
Make the thinking/null leniency role-aware so it never silently drops real
user input. Assistant turns (replayed history) still accept unknown/thinking
blocks and coerce a null tool-only turn to empty. User turns keep the strict
boundary: a null user content is rejected, and a content block the converter
cannot translate is rejected instead of being dropped into an empty prompt.
Also remove an empty file committed by accident.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio /v1/messages: coalesce resumed user turns and tighten content checks
- The /v1/messages count and generation paths now coalesce the adjacent user
turns that dropping an empty or null assistant turn can leave behind, so a
strict GGUF chat template no longer 400s on non-alternating roles.
- A user content block with a non-string type (list / dict) is rejected as a
clean 400 instead of raising TypeError and escaping as a 500.
- The assistant null-to-empty coercion only applies to an explicit null; an
assistant turn that omits content entirely still fails required-field
validation instead of being silently coerced to an empty string.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio /v1/messages: tighten comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The transformer and text encoder auto-quantize; the VAE stayed dense. VAEs are
convolutional, so torchao int8 (Linear/2D-only) does not apply, but
Float8DynamicActivationFloat8WeightConfig quantizes Conv2d/Conv3d weights with
PerTensor granularity (auto-skipping convs whose channels are not a multiple of 16,
so the 3-channel RGB head stays dense). New diffusion_vae_quant.py offers two
schemes: fp8_dynamic (torchao conv compute fp8, cc>=8.9, resident) and fp8
(diffusers layerwise storage cast, any conv, survives offload); no int8 (no Conv3d
int8 kernel). select_vae_quant_scheme walks (fp8_dynamic, fp8) with a live conv
smoke probe, an offload gate, a per-family deny list, and a force_fp32 gate; the
image + video loaders map unset vae_quant to auto, skip the vae_force_fp32 Wan
families, and record the engaged scheme. Guards _align_vae_dtype to skip the
img2img/inpaint re-cast when the VAE is quantized (its fp8 tensor subclasses reject
.to(dtype=)). Verified on a B200: %16 Conv2d/Conv3d/Linear -> Float8Tensor, conv_out
dense, forward runs.
The transformer already defaults to auto-quant (fp8/int8); the companion text
encoder was opt-in and stayed dense bf16 unless a scheme was named, even though it
is often the largest resident component. Add an auto policy mirroring the
transformer's ladder: select_te_quant_scheme walks a per-capability ladder
(data-center fp8-GEMM: fp8_dynamic -> int8 -> layerwise fp8; Ampere: int8 -> fp8),
reorders int8 first on consumer GDDR parts, falls to layerwise fp8 under group
offload (the only offload-safe cast), only picks int8 for a family with a measured
keep-bf16 schedule, honors a per-family deny list, and smoke-probes the torchao
kernel so a missing build degrades gracefully. The image + video loaders now map an
unset text_encoder_quant to auto (explicit none/off stays dense; a named scheme is
forced), so the shipped default quantizes the encoder to the fastest accurate
scheme for the GPU. Records the engaged scheme in the image resolved-record too.
Verified on a B200: auto -> fp8_dynamic, offload -> layerwise fp8.
Fold PR #6872's image-generation fixes into the branch, deduped against the
round-12 dataset-upload and gallery integrity work already on image-generation.
Fixes carried forward from #6872:
- fp8 single-file transformer memory estimate: an fp8 checkpoint loads with no
quantization_config and diffusers upcasts it to bf16 (~2x resident), so budget
it accordingly in _plan_memory and estimate_safetensors_dense_mib.
- dense-quant OOM-evict preflight: when the GGUF fits resident but the dense bf16
transformer this path materializes does not, skip the fast path up front rather
than evict the current pipeline and OOM in finalization. Combined with the
existing offload->resident candidate re-plan so both the family-table estimate
and the on-disk shard measurement gate engagement (unified on the
transformer_resident_override_mib plan override).
- ControlNet: evict the previous module and its from_pipe wrapper before loading a
new one so swapping ControlNets within a base-model load cannot accumulate to OOM.
- ControlNet union_control_mode: raise on an unknown control type instead of
silently defaulting to canny.
- edit-family mask rejection: raise instead of silently dropping a mask on an
image-editing model that has no inpaint pipeline.
- companion cache: walk the snapshot dir and exclude transformer/ so the
dense-quant prefetch's cached shards do not inflate the companion total and
wrongly force offload.
- training: drop piecewise_constant from the LR scheduler enum and force bf16 for
fp16-incompatible families.
- dataset upload: batch-atomic staging with the same-stem duplicate guard.
- images page: guard negative-prompt restore on guidance>0, clear stale ControlNet
selection on restore, and revert an optimistic quant label when a pipeline load
never starts.
- uninstall (sh + ps1): keep the owner-marker guard on sd.cpp removal.
Conflicts resolved in favour of image-generation's evolved memory system,
loadSpecFor catalog, and stop-and-save (lora_path) run detection; #6872's fp8 and
dense-preflight fixes carried forward on top. All affected backend tests pass
(test_diffusion_backend, test_diffusion_training, test_diffusion_lora_trainer,
test_video_gallery, test_diffusion_controlnet).
Follow-up to removing piecewise_constant from the trainable scheduler allow-list: the
DiffusionTrainingStartRequest.lr_scheduler Literal still advertised it, so a client that
picked it straight from the schema passed request validation and then hit the 400 from
normalized(). Remove it from the enum too so the API only offers schedulers the trainers
can actually run, and add a test asserting the enum never advertises a scheduler outside
the validation allow-list.
* Studio: apply presence_penalty on the safetensors and MLX inference paths
The safetensors and MLX generate paths resolved the inference config and
then dropped presence_penalty before generation, so the same model applied
the configured value under GGUF and 0 under safetensors/MLX. Thread the
already-resolved presence_penalty through the orchestrator command, worker
gen_kwargs, and the safetensors/MLX generate calls, and apply it with a
small logits processor (subtract once per distinct completion token,
prompt excluded, presence not frequency, zero is a no-op, negatives raise).
Backwards compatible: presence_penalty defaults to 0.0 (byte-identical
output when unset) and the GGUF path is unchanged. Also forward min_p on
the legacy /generate/stream route and add the missing min_p field to
GenerateRequest.
* Studio: bound presence_penalty generated ids to valid vocab range on both paths
The presence-penalty logits processors index by generated token ids. The
torch path filtered only the upper bound (seen < vocab_size), so a negative
id would silently wrap to the wrong row; the MLX path had no bound at all,
and MLX out-of-bounds indexing is documented undefined behavior (crash or
memory corruption on Apple Silicon), unlike torch's harmless negative wrap.
Bound generated ids to [0, vocab) consistently on both paths:
- torch: seen[(seen >= 0) & (seen < vocab_size)] (zero-regression safety net;
real completion tokens are always in range).
- MLX: route out-of-range/negative ids to a discarded scratch slot via
mx.where and a (vocab + 1)-wide scatter-assign mask, then subtract. MLX has
no boolean-mask filtering (data-dependent output shape), so this keeps a
fixed shape, stays on-device, and preserves once-per-distinct-token
semantics without any torch/numpy dependency.
Add torch tests for out-of-range and negative ids (only in-range distinct
ids penalized, stray ids ignored, no wrong-index wrap) and a bound-documenting
MLX test that runs on the arm64 macOS CI.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: client-tool passthrough healing for safetensors and MLX
PR 6801 made response-side tool-call healing default-on for the client-tool
passthrough, but only on the GGUF path: the passthrough branch in
/v1/chat/completions is gated on using_gguf, and the safetensors section never
reads payload.tools, so a client-tools request against a safetensors or MLX
model silently dropped the tool schemas and returned prose with no tool_calls.
Add the missing leg. When a non-GGUF model is loaded, the request declares
client tools (or carries tool-role history), server-side tools are off, and the
template supports tools, the route now:
- renders the tools into the chat template for a single turn via the existing
backend.generate_chat_response(..., tools=...) seam (worker templating
already accepts role=tool and assistant.tool_calls messages, normalized with
_openai_messages_for_passthrough);
- non-streaming: promotes text-form calls with heal_openai_message, honors the
opt-in nudge single retry (nudge_should_retry / nudge_messages), caps healed
calls when parallel_tool_calls=false (covers the nudge retry too), and sets
finish_reason=tool_calls with content null on a pure tool-call turn;
- streaming: derives deltas from the worker's cumulative snapshots and feeds
StreamToolCallHealer, emitting healed tool-call deltas and the correct
finish chunk, guarded against repeated or shrinking snapshots.
heal_gate semantics are identical to the GGUF passthrough: default on,
auto_heal_tool_calls=false or UNSLOTH_DISABLE_TOOL_CALL_HEALING=1 relays
verbatim, tool_choice narrows promotion, undeclared names stay text. MLX rides
the same orchestrator seam, so both local backends gain the behavior.
CompletionMessage.content becomes Optional so a promoted pure tool-call turn
matches the OpenAI contract (content null when only tool_calls return).
Adds tests/test_sf_client_tools_passthrough.py (22 cases: healing, gating,
opt-outs, streaming deltas, tool-role history, dict-arguments history, forced
tool_choice, parallel cap, usage, nudge on/off/double-failure, generator error
hygiene, disconnect reset, empty output, MLX path).
* Address review: tool_choice none, developer folding, retry fallback, monitor reply
Four review follow-ups on the safetensors/MLX client-tool passthrough leg:
- tool_choice="none" keeps the tool-history templating but no longer
advertises the tools, so a forced final-answer turn is not prompted into
emitting markup that the (correctly disabled) healer would relay as prose.
Mirrors the GGUF passthrough where llama-server honors tool_choice itself.
- OpenAI "developer" messages fold into a single leading system message via
_set_or_prepend_system_message before templating; local templates reject the
role and the fallback formatter drops it.
- A nudge retry that fails or is cancelled after the original answer exists
falls back to the first response instead of surfacing a 500, matching the
GGUF nudge path.
- The API monitor records the healed tool call summary instead of the raw
markup on a promoted turn.
Adds four regression tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: forced tool_choice templating, content-part flattening, stream monitor parity
- A forced tool_choice function is now the only schema rendered into the
local template, so the advertised tools and the healer allowlist can no
longer disagree (llama-server enforces tool_choice itself on the GGUF path).
- Content-part lists are flattened to their text parts before templating.
Remote image URLs are not decodable locally, so such requests reached this
path with part lists that raise inside apply_chat_template on text-only
templates; the plain non-GGUF path has always flattened them.
- The streaming monitor entry is now fed from the healed events the client
actually receives, recording promoted calls as the [tool_calls] summary
the non-streaming path records.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: gate passthrough on the engaged server path, deserialize templated arguments
- The client-tools gate now keys on _sf_use_tools (whether the server-side
tool path actually claimed the request) instead of the raw mcp_enabled
flag: with an empty MCP registry or a CLI --disable-tools policy, a client
that sets mcp_enabled while declaring its own tools fell through to plain
generation with the tools silently dropped. The GGUF passthrough gate has
no mcp_enabled clause either.
- New _structured_tool_history_for_local_template deserializes assistant
tool_calls[].function.arguments JSON strings into mappings for the
templated copy only: spec-compliant clients send strings, but local chat
templates iterate arguments as a mapping or raise on strings, which
crashed or misrendered multi-turn tool history. The HTTP response and the
GGUF wire shape keep strings.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments and docstrings in the client-tools passthrough
* Report first-attempt usage when a nudge retry is discarded
When nudge_should_retry fires but the retry produces no healable tool call
(or raises), the first response is still delivered to the client. The retry's
generate() had already overwritten stats_holder, so _monitor_usage recorded
the unseen retry's token counts against the request instead of the first
attempt that was actually returned. Capture the first attempt's stats before
the retry and restore them on both the no-heal and exception paths so the
monitor reports the usage of the response the caller received.
* Do not promote buffered tool markup when a stream is cancelled
The streaming client-tool heal path breaks out of the token loop when
cancel_event is set (the registry "Stop" path), but then still fell through to
healer.finalize(), which heals incomplete tool markup at EOF (allow_incomplete)
and emits a tool_calls delta plus finish_reason=tool_calls. Because the Stop
request only sets the event and leaves the SSE socket open, the client received
that promoted call and executed a tool the user had just cancelled. The disconnect
path already returns before finalize; guard finalize and the finish_reason on
cancel_event too, so a cancelled stream ends with finish_reason=stop and no tool
call. Adds a regression test driving a Stop mid-emission with buffered markup.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments in the client-tools passthrough
* Trim client-tools passthrough comments further
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Auto policies: deferred dense compile, video compile default, step cache and precision auto
Image dense loads with speed unset no longer sit at plain off: the load stays
bit-identical eager, and the 3rd generation in a session engages the default
compile profile plus the cuDNN attention upgrade mid-session (a one-off image
never pays the warmup, repeated use amortises it). Video dense loads resolve
straight to the default profile since a clip denoise amortises the compile
within a single run, and never to max.
Video also gains the image backend's tri-state auto policies: unset step cache
now decides from the default schedule and re-checks the actual step count per
generation, and unset precision (transformer_quant) hands the decision to the
hardware ladder instead of staying off. Memory badge reason now says plainly
that everything fits when no offload is planned.
* Rename Dtype to Precision, add the video Precision control, step cache Auto option
The images Advanced panel's Dtype row is now Precision (same control, clearer
name), and the video Advanced panel gains the matching Precision select wired
to the load route's existing transformer_quant field, gated to full-pipeline
loads the way the image control gates to GGUF. Step cache selects on both
pages gain an explicit Auto option as the default (the previous Off default
silently behaved as auto and never let anyone pin off), and the Speed and
Attention tooltips now state the deferred dense compile and the SageAttention
black-frame caveat.
* Model catalog: canonical diffusion model groups with device-aware routing
One canonical name per image/video model, its published artifacts (GGUF, FP8,
bnb-4bit, official BF16) as data, and pure routing helpers: suffix-stripped
canonical keys (owner-preserving; cross-owner merges only via explicit
aliases), group/artifact lookups, a flat back-compat options shim, load-spec
resolution replacing the pages' lookup tables, search matching over old ids
and format tokens, the GGUF fit ladder extracted from the variant expander,
and pickDefaultArtifact/pickDefaultQuant deciding what a bare group click
loads (downloaded first, then the best quality that fits 70 percent of VRAM,
GGUF as the safe fallback). Checked by npm run catalog:check, following the
i18n:check pattern.
* Picker: one canonical row per diffusion model with a format second level
The Images and Video pickers now render the curated catalog as one row per
model in Recommended: clicking loads the best artifact for the device (the
routed GGUF quant, a prequant FP8/bnb-4bit that fits, or the official BF16),
and a chevron opens the per-format list, with the GGUF row nesting the usual
quant expander. Live HF listing rows that belong to a group are deduplicated,
search collapses member repos into their group (old ids and format tokens
still match), and the On Device sections group cached member repos under the
same canonical name with the per-repo rows inside. Curated groups render from
the catalog rather than the HF listing, which finally surfaces LTX-2.3 in the
video Recommended list (its hub pipeline_tag is image-to-video, so the
text-to-video listing always missed it) and exposes the HunyuanVideo 720p
repack next to 480p.
Backend: /cached-models now tags trusted video-family repos text-to-video
instead of blanket text-to-image, and the pickers admit catalog-known
non-unsloth repos On Device, so cached Lightricks/Wan/Hunyuan pipelines
finally appear in the Video picker. Chat pickers pass no catalog and are
unchanged.
* Download formats, tab icons, plain-language train tips, 3-loop autoplay
The image Download button becomes a menu: PNG saves the original bytes with
the embedded recipe, JPEG and WebP re-encode client-side from the fetched
blob (JPEG flattened onto white). The video Download button gains MP4
(original, keeps audio), WebM and GIF; the latter two transcode server-side
from the stored MP4 via PyAV (VP9 realtime profile for WebM, ~12 fps adaptive
palette for GIF) behind a new gallery export route that 501s with a readable
message when a codec is missing.
Generated clips no longer loop forever: the player replays a clip three times
per selection, then pauses with controls up; a new generation or a refresh
gets its own three plays. The Create/Train tabs reuse the sidebar's New Chat
and Train icons (TestTubeOutlineIcon moved to a shared lib module), and every
Train tab helper text is now one plain sentence.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the Create/Train tab icon and label on one line
TabsTrigger renders its children inside a plain inline span and the
Tailwind preflight gives svg display:block, so the HugeiconsIcon forced
the label onto a second line. Wrap icon plus label in their own
inline flex row inside each trigger.
* Strip -int8 and -nvfp4 prequant suffixes in the model catalog key
canonicalKeyFor already lowercases before matching, so -GGUF/-FP8 in any
case were covered; -int8 and -nvfp4 were not in the suffix table, so
such repos rendered as standalone rows in Recommended and On Device
instead of standardizing into their base-name group and routing through
pickDefaultArtifact. Added both suffixes plus case-insensitivity and
routing assertions to the catalog check.
* Standardize non-catalog picker rows to their base model name
The curated catalog already collapses its own groups, but hub listing
rows and cached repos outside the catalog (ERNIE-Image, FLUX.2-klein,
Qwen-Image-Edit-2509, FLUX.2-dev) still rendered raw ids with -GGUF /
-FP8 style suffixes in Recommended and On Device.
- model-catalog.ts: new stripArtifactSuffixesForDisplay, a
case-preserving twin of canonicalKeyFor's stripping that keeps the
owner prefix and original casing for display.
- pickers.tsx: recommended hub rows and the downloaded GGUF/model rows
pass their labels through it when a catalog is present, so only the
diffusion pickers change; chat rows keep raw ids. Click targets keep
the full repo id, and the format badge still shows the artifact kind.
- Catalog check covers the new helper across GGUF/FP8/int8/nvfp4 in
both cases plus no-op and suffix-only names.
* Offer official BF16/FP8 artifacts per model group and fix gallery label clipping
Model picker changes so groups are not limited to unsloth quant repos:
- model-catalog.ts: each image group that has an official vendor pipeline
now carries its BF16 (official) artifact as the top (highest quality)
entry - Tongyi-MAI/Z-Image-Turbo, Qwen/Qwen-Image, Qwen/Qwen-Image-2512,
Qwen/Qwen-Image-Edit-2511, black-forest-labs/FLUX.1-dev, FLUX.1-schnell
and FLUX.1-Kontext-dev. The LTX-2.3 video group now lists Lightricks'
own bf16 and fp8 distilled single-file checkpoints alongside the GGUF.
Resident sizes are set from the actual weight totals (FLUX ships a
duplicate single-file that from_pretrained ignores, so FLUX bf16 is ~32
GB not 54). The repos that used to be aliases are now real artifacts.
- The router already prefers the highest-quality artifact that fits the
0.7 x GPU budget, so a datacenter GPU now defaults to official BF16
while consumer GPUs still route to the fitting quant or GGUF. That is
why bnb-4bit was the Z-Image-Turbo default before: it was the only
non-GGUF artifact and it was already downloaded.
- diffusion.py: allowlist the four official image repos not previously
trusted (qwen/qwen-image-2512, qwen/qwen-image-edit-2511,
black-forest-labs/flux.1-schnell, flux.1-kontext-dev). All verified as
safetensors-only diffusers model_index pipelines. The LTX-2.3
checkpoints are already on the video trust list.
- catalog check: BF16-wins-on-datacenter, quant-wins-on-consumer, and the
single-file load specs for the LTX-2.3 checkpoints.
Also fixes the video gallery thumbnail caption: the leading duration was
clipped by the rounded corner and selection border, so the strip now has
enough left/bottom padding to clear the curve.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* video gallery: guard export transcode against a stream-less clip
_transcode_webm and _transcode_gif indexed src.streams.video[0] before
checking the stream list, so a container with no video stream raised a bare
IndexError that the broad handlers then re-labeled as a missing libvpx or
decoder. Raise an explicit RuntimeError naming the real cause in both the
WebM and GIF paths.
* Studio: honor explicit attention/format choices, fix distilled-LTX defaults and On Device catalog routing
* Remove stray planning notes accidentally committed to the branch
* video: add transformerQuant to the load callback deps
handleLoad reads transformerQuant but omitted it from the useCallback dep array,
so after the user changes only Precision and then selects a model or clicks
Reapply, the memoized callback keeps the stale closure and loads the previous
precision. The image page's equivalent callback already lists it.
* model picker: honor the format filter when routing catalog clicks; add catalog rows to the roving list
- routedArtifactFor now scopes a group's artifacts to the active format filter
(the same matchesFormatFilter predicate the visibility check uses) before
pickDefaultArtifact, so a group shown only because it owns a GGUF no longer
routes a click to a large non-GGUF download. Covers both the Recommended and
On Device grouped paths.
- hubOptionKeys now includes the catalog-group, search-catalog-group, and grouped
On Device row keys in exact render order, so arrow/Home/End roving reaches the
catalog rows instead of giving them a duplicate missing id and skipping them.
* model picker: don't treat a partial base cache as downloaded
A partially-cached base repo (a cancelled download that left only some weights)
was counted as downloaded, so an On Device click routed to a fresh multi-GB
re-download instead of the complete GGUF. The picker's endpoint (/api/models/
cached-models) did not carry a partial flag at all, so a frontend-only guard
could not see it. Surface partial from that endpoint by reusing the hub inventory
scan's snapshot-partial detector, plumb it through CachedModelRepo (backend +
frontend types), and skip partial base repos when building the downloaded set.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* model picker + diffusion: drop partial/unloadable cached rows, skip defer-compile before a LoRA gen
- On Device (cached non-GGUF) rows filtered partial-download snapshots back in: sortedCachedModels
gated on passesTaskGate + a groupForRepoId key match but, unlike downloadedSet, never checked
c.partial, so an incomplete unsloth snapshot showed as a loadable On Device row (click errors or
silently re-fetches multi-GB). It also admitted repos that only match the catalog by group KEY
(a base / uncurated-quant sibling like Qwen/Qwen-Image-2512) which have no loadable artifact and
dead-end at the trust gate. Add !c.partial and gate on artifactForRepoId (what loadSpecFor
resolves) instead of groupForRepoId, so a cached row shows only when the backend can load it.
- Deferred speed-auto engaged the compile profile on the 3rd generation BEFORE _apply_loras. A
compiled transformer rejects LoRA (supports_lora is False) and _apply_loras raises before its
unchanged-selection no-op, so once compile engaged every LoRA generation on that load failed
permanently. Skip the deferral when a LoRA is requested (compile and LoRA are mutually exclusive)
and let it engage on a later LoRA-free generation.
* Scope the cached-model partial probe to the listed snapshot dir
list_cached_models builds each row from the largest/complete copy across HF cache
roots, but _cached_repo_partial probed is_snapshot_partial with no repo_cache_dir,
so the scan spanned every root: a stale .incomplete copy in one root would flag a
complete copy in another as partial and hide the usable model from the picker (the
click then routes to a re-download). Forward the winning snapshot's repo_path so all
three partial signals are scoped to that copy, matching the sibling inventory paths
(models/dataset cache_inventory, local_inventory).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Do not auto-route to gated repos, prefer complete cached copies, defer compile past attached LoRA, scope group expand keys
Four fixes:
- pickDefaultArtifact's not-downloaded ladder returned the gated BF16 FLUX.1-dev / Kontext-dev
before the open GGUF on a large GPU, so a bare group click routed to a repo the user may lack
license/token access to. Add a gated flag and skip gated artifacts in the not-downloaded ladder
(an already-downloaded gated artifact is still returned).
- list_cached_models picked the largest duplicate cache copy and computed partial only on it, so a
larger partial copy shadowed a smaller complete one; since partial rows are dropped from the
picker the usable model vanished. Prefer completeness, then size.
- the deferred-speed compile engaged on a no-LoRA generation while an adapter from a prior
generation was still attached, baking it into the compiled graph (the later unload is swallowed
on a compiled pipe); also defer while adapters remain attached.
- routeGroupClick's GGUF fallback toggled the context-free canonicalId while the chevron toggles
the context-scoped expandKey, leaving the format list un-collapsible in one context, dead in the
other, and risking cross-context expansion; thread expandKey through.
* Guard video pipeline repos from deletion, drop the always-failing LTX FP8 artifact, prefer 720p Hunyuan
Three round-6 fixes:
- cached non-GGUF video repos now surface in the Video On-Device picker with the normal delete
action, but /delete-cached only guarded chat + the Images engine, so a loaded/loading Wan / LTX /
Hunyuan pipeline could have its HF snapshot removed from under it. Add a VideoBackend
loading_repo_ids accessor and a video loaded/loading guard mirroring the Images one.
- the catalog advertised Lightricks/LTX-2.3-fp8 as loadable, but the LTX-2.3 loader refuses the
official scaled-FP8 single file (.weight_scale/.input_scale) and points to GGUF/BF16, so a pick
routed to a ~76 GB download that always fails on load. Remove the FP8 artifact.
- pickDefaultArtifact only sorts by format, so the HunyuanVideo group's 480p (listed first) beat
the 720p even on GPUs where 720p fits the budget. List 720p first so the fit loop prefers it and
falls back to 480p only on smaller cards.
* diffusion: add compute int8/fp8_dynamic text-encoder quant, wire into video
Add two torchao compute text-encoder quant modes to the diffusion precision
engine, alongside the existing layerwise fp8 and weight-only nvfp4:
- int8: per-token activation + per-channel weight (torch._int_mm), with per-layer
keep-bf16 selection. int8 degrades on large encoders unless the most
quant-sensitive decoder blocks stay bf16, so it engages only for families with
a measured keep-bf16 schedule (qwen-image / qwen-image-edit keep first+last 6,
flux.2-dev keeps first 3); a family without one falls back to fp8.
- fp8_dynamic: per-row fp8 compute (torch._scaled_mm), keeping the matmul in fp8
on the tensor cores instead of upcasting each forward like the layerwise fp8.
The selective int8 caster reuses the committed transformer-quant factory
(_make_quant_config / make_filter_fn / exclude_tokens_for_scheme) plus a small
structural first/last-N block skip, so it depends only on committed APIs.
Wire text-encoder quant into the video backend, which previously loaded the
companion encoder (Gemma3 / UMT5 / Qwen2.5-VL) dense bf16 while quantising only
the DiT. text_encoder_quant is plumbed through the load request, validation, the
load chain, the resolved record, and status, mirroring the image backend; it
applies for every load kind (the encoder is dense regardless of how the DiT was
sourced). Widen the image and video load request Literals and add the video
status field.
Tests: int8 family-schedule routing and fp8 fallback, fp8_dynamic routing,
hardware gates (int8 sm_80+, fp8_dynamic sm_89+), the structural block selection,
the real int8 filter closure (keeps the first blocks plus the vision tower /
lm_head / T5 wo dense), and the video route threading and 422 validation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* text-encoder quant: skip the torchao modes under offload (both backends)
quantize_text_encoders applied int8-with-schedule / fp8_dynamic / nvfp4 (all torchao) to the
text encoder regardless of the offload policy. An offload placement then moves the quantized encoder
with Module.to(), which torchao tensor subclasses reject (aten._has_compatible_shallow_copy_type is
unimplemented) -- a hard crash, the same one the DiT path already skips torchao quant under offload to
avoid. Add offload_active to quantize_text_encoders and skip the torchao modes when set; layerwise fp8
is not torchao and still streams under offload. Both the video and image loaders pass
offload_active = (offload policy != none).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* diffusion: skip non-bf16 linears for scaled_mm quant schemes
The fp8 / mxfp8 / nvfp4 schemes run on torch._scaled_mm and the fp4 / mx GEMMs,
which assert a bfloat16 input weight. On a mixed-precision DiT that keeps some
linears in fp32 for numerical stability (the Wan and Hunyuan video transformers
do this), quantize_ hits the first fp32 linear, raises, and the best-effort
wrapper swallows it to None, so the whole transformer stays dense with no error
and no speedup or memory saving.
Add a require_bf16 gate to make_filter_fn and pass it for the scaled_mm schemes
in quantize_transformer (and the fp8_dynamic text-encoder caster). The gate
skips non-bf16 linears so the scheme engages on the bf16 ones. int8 uses
torch._int_mm, which quantizes fp32/fp16 weights fine, so it leaves the gate off
and keeps its current coverage.
Verified on Wan2.2-TI2V-5B: fp8 and mxfp8 now quantize 303 linears via the
committed quantize_transformer path where they previously engaged 0.
* prequant builder: mirror the scaled-mm bf16 gate offline
The runtime DiT quantizer skips non-bf16 Linears for the scaled_mm schemes (fp8,
nvfp4, mxfp8) so the scheme engages on a mixed-precision transformer instead of
aborting on the first fp32 Linear. The offline prequant builder reused make_filter_fn
without that gate, so building an fp8/nvfp4/mxfp8 checkpoint for a mixed-precision DiT
(Wan, Hunyuan keep _keep_in_fp32_modules in fp32 even under torch_dtype=bf16) would hit
the same fp32 Linear and abort, breaking the builder's stated offline == runtime,
LPIPS-0 invariant. Thread require_bf16 = scheme in _SCALED_MM_SCHEMES through the builder,
record it in the checkpoint metadata, and verify it on load (mirrors the existing
exclude_name_tokens guard) so a future _SCALED_MM_SCHEMES change cannot silently load a
checkpoint built under the old filter.
* Keep nvfp4 fp32 linears quantised (bf16 gate is fp8/mxfp8 only)
Verified on torchao 0.17 / B200: fp8 per-row asserts 'PerRow quantization only
works for bfloat16 precision input weight' and mxfp8 asserts 'Only supporting bf16
out dtype', but NVFP4's high-precision conversion quantises an fp32 weight fine
(forward included). So the bf16 skip-gate must be fp8/mxfp8 only, not all scaled_mm
schemes -- otherwise nvfp4 leaves large fp32 projections dense, losing the intended
memory/speed gain. Rename _SCALED_MM_SCHEMES -> _REQUIRE_BF16_SCHEMES = (fp8, mxfp8)
and thread it through the runtime filter, the offline builder, and the loader
require_bf16 verification (offline == runtime preserved).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
- _dense_quant_prefetch_needed widened the transformer/ prefetch to pull the base
repo's full dense bf16 shards even when a prequant checkpoint is configured
(candidate.prequant), contradicting its own docstring. That both defeats the
prequant download savings and can hard-fail begin_load on a disk-full (no GGUF
fallback there). Only widen for a real dense build (candidate is not None and
not candidate.prequant).
- DiffusionStatusResponse declared no 'resolved' field, so Pydantic's default
extra='ignore' silently dropped the per-control auto-policy provenance the
backend records (build_resolved_record / state.resolved) -- the plumbing never
reached any client. Declare the field so it round-trips.
- DiffusionFamily gains deploy_base_repo (krea/Krea-2-Turbo): deploying a LoRA
trained on Raw now previews it on Turbo, not the non-distilled Raw checkpoint.
Scoped to a same-precision override so it never turns an nf4 train base into a
larger bf16 deploy load; exposed through family_train_infos -> the Train UI's
onDeployClick / historical-run deploy resolve the deploy base.
- _GENERATION_DEFAULTS gains a Krea entry (8 steps, 0 CFG) so the OpenAI
/v1/images/generations route matches the Create UI's documented distilled recipe
instead of falling through to the generic (9, 0.0).
Re-plan memory with the quant steady factor when the bf16 table forces
offload a quantised DiT would not need, mirroring the image dense-quant
path, and fall back to the bf16 plan when quant does not engage. Stream
the second expert under group offload (model and sequential already hook
every module). Fail the load cleanly when quant engages on only one
expert instead of running mixed precision with quant reported off.
Persist guidance_2 in the gallery recipe so A14B clips are reproducible.
Register two new video families and wire them through the backend, routes,
and frontend picker: wan2.2-ti2v-5b (single DiT) and wan2.2-t2v-a14b (the
dual-expert MoE). Both share diffusers' WanPipeline + WanTransformer3DModel
+ AutoencoderKLWan, which the VideoFamily dataclass already reserved fields
for (transformer2_class, is_moe, cfg2_kwarg).
Verified against the installed diffusers 0.39.0 before writing code:
- WanPipeline, WanTransformer3DModel, and AutoencoderKLWan are all exported
from top-level diffusers 0.39.0.
- WanPipeline.__call__ (pipeline_wan.py:383) defaults to num_frames=81,
num_inference_steps=50, guidance_scale=5.0. guidance_scale_2 DOES exist
in 0.39 (line 392) and its check_inputs raises if it is passed when the
pipeline's boundary_ratio is None (line 322), so the second guidance is
threaded ONLY for the MoE family and only when inspect.signature accepts
it (the same gate frame_rate already uses).
- The Wan VAE temporal factor is 4 (autoencoder_kl_wan.py scale_factor_temporal),
and the pipeline snaps num_frames to 4k+1 (line 493), so frame_step is 4,
unlike LTX-2's 8k+1. Sizes patchify at spatial 8 * patch 2 = 16, so
resolution_multiple is 16.
- boundary_ratio and transformer_2 come from model_index.json: TI2V-5B ships
boundary_ratio=null and transformer_2=[null,null] (single DiT), while A14B
ships boundary_ratio=0.875 and transformer_2=WanTransformer3DModel (dual
DiT). boundary_ratio lives in the pipeline config, so it needs no per-call
plumbing.
- WanTransformer3DModel declares _repeated_blocks=["WanTransformerBlock"] and
inherits CacheMixin (transformer_wan.py:508/551), so regional compile and
First-Block-Cache both work.
bf16-resident component sizes, measured from each diffusers repo's on-disk
safetensors (all stored bf16), feed the auto memory table:
TI2V-5B: transformer 20.0, UMT5 text encoder 11.4, VAE 2.8 GB.
A14B: two experts 57.2 each (114.3 total), text encoder 11.4, VAE 0.5 GB.
Backend changes make the optimisation layers dual-DiT aware: a small
_SecondDiTView proxy presents transformer_2 as pipe.transformer so the
existing single-DiT helpers (apply_speed_optims, apply_attention_backend,
apply_step_cache, quantize_transformer) cover BOTH experts on an is_moe load
without forking any helper; single-DiT loads are unchanged (views is just
(pipe,)). The two Wan base repos are added to the trusted non-GGUF allowlist.
A transformer_quant option is added to the load path, mirroring the image
backend's dense torchao fast path: on a pipeline-kind load the dense DiT(s)
are quantised in place onto the low-precision tensor cores and the engaged
scheme is surfaced in status. generate() threads guidance_2 through the
family's cfg2_kwarg when the loaded pipeline accepts it.
Routes and Pydantic models gain the optional transformer_quant (load /
status) and guidance_2 (generate) fields. The frontend picker gains the two
Wan models with 50-step / CFG 5.0 defaults; fps is supplied per family by
the backend.
Tests extend the fake runtime with WanPipeline and per-DiT transformer fakes
(single-DiT and dual-DiT), and cover family detection for both repos, 4k+1
frame snapping, default application, dual-DiT speed/cache/attention/quant
coverage on both experts, cfg2 threading gated on the pipeline signature,
trusted-repo validation, and the new route fields. Both the standard and the
diffusers/torchao-blocked CI-sim runs are green.
routes/video.py mirrors the /images/* routes one-for-one: validate-before-evict
load ordering (a bad pick must not evict a working chat model and then 400),
the training-active interlock, the device-gated GPU arbiter handoff with the
new VIDEO owner, the exact-match sentinel mapping (VIDEO_NOT_LOADED_MSG /
VIDEO_CANCELLED_MSG to 409, ValueError/FileNotFoundError to 400 with native
paths redacted, everything else a sanitized 500), and the gallery CRUD shape
with fetch-one-extra has_more paging. Generate persists the encoded MP4 plus
its full recipe through video_gallery.save and returns the gallery record; the
file endpoint serves video/mp4 with an immutable Cache-Control, 404 on any id
that fails the containment check.
models/inference.py gains the video request/response set (VideoLoadRequest,
VideoGenerateRequest/Response, GalleryVideo, gallery list, both progress
shapes, VideoGenerationDefaults nested in VideoStatusResponse), reusing
DiffusionResolvedControl for the resolved-provenance badges. The router is
registered in main.py after the images router under the same /api/inference
prefix and auth dependency.
Tests: 20 route tests on a stubbed backend + real tmp gallery (load happy path
and arbiter acquisition, 400/409 mappings, generate persistence round trip,
mp4 file serving + 404, delete/clear, unload releases ownership); the
diffusion route suite stays green after the shared models edit.
