21 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
pre-commit-ci[bot]
|
0d378bc496 |
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci |
||
|
Daniel Han
|
61a5c91461 | Merge branch 'image-generation' into video-diffusion-improvements | ||
|
Daniel Han
|
d58141b611 |
perf(video): generalize round-2 levers to Wan2.2 and LTX-2: per-family step cache, per-expert MagCache, TE quant audit
Extends the HunyuanVideo-1.5 round-1/2 optimization levers to wan2.2-ti2v-5b, wan2.2-t2v-a14b (dual-expert MoE) and ltx-2, shipping only what beats the incumbent on the measured accuracy-speed frontier (B200, LPIPS(AlexNet) pairwise vs the same uncached compiled stack at identical seed/settings). - Wan2.2-TI2V-5B auto step cache switches FBCache to calibrated MagCache: balanced (0.12, 3, 0.2) measures 1.65x at pairwise LPIPS 0.034 vs the incumbent FBCache 0.08 at 1.49x/0.031, and 1.73x/0.044 vs 1.71x/0.083 at the fast points (FBCache error grows unboundedly past its threshold while MagCache's budget caps it). A 50-step calibrated curve ships; cond/uncond branches agree within 0.0008 so one curve serves both CFG contexts. - Per-expert MagCache plumbing for dual-expert MoEs: the experts split the schedule at the boundary timestep (Wan2.2-A14B: 16 + 34 of 50) and the hook counts each expert's own forwards from 0, so a shared full-schedule curve would be misaligned for both. apply_step_cache / maybe_toggle_step_cache / the loader now thread an expert name; a second expert resolves family::transformer_2 curves and sub-curves scale their configured step count by steps/50. Single-DiT behaviour unchanged. - Wan2.2-A14B keeps FBCache: with per-expert curves, FBCache 0.12 at 2.88x/0.128 dominates balanced MagCache (1.80x/0.145) and FBCache 0.08 sits at 1.28x/0.098; the 16-step high-noise expert starves MagCache's skip budget. No calibrated curve ships, so an explicit magcache request runs uncached with a warning instead of engaging a measured-worse mode. - Wan2.2-A14B TE auto quant resolves dense: TE fp8_dynamic alone costs pairwise LPIPS 0.1195 for a 1.03x once-per-generation encode (146.7 to 142.7 s e2e). Wan2.2-TI2V-5B shares the UMT5 encoder but stays quantized (0.0396 pairwise at a real 1.09x on its much faster DiT). - LTX-2 TE fp8_dynamic family-denied: torchao per-row compute fp8 on the Gemma3-27B encoder black-frames the whole clip (mean luma 137.9 to 0.0, LPIPS 0.78; reproduced compiled and eager), while layerwise fp8 is near-lossless (pairwise 0.0043) at the same shrink, so auto falls through to it and explicit fp8_dynamic requests are refused. - LTX-2 step caching deliberately stays unregistered, now documented on _EXTRA_BLOCK_METADATA: the block returns a joint (video, audio) stream pair and both cache hook families would substitute text embeddings into the audio slot on every skipped step; a dual-stream cache is required, and the distilled checkpoints run below FBCACHE_MIN_STEPS anyway. - Compile parity (emulate_precision_casts) verified family-neutral and kept global: wan5b 1.75x/0.0029 on vs 1.54x/0.0082 off; ltx2 1.308x/0.0013 vs 1.307x/0.0025; a14b 2711 vs 2717 ms/step. Cache-hook compile arming verified to generalize (wan5b fb@0.04 armed 1.216x vs raw 1.048x). Dual-GPU CFG stays HunyuanVideo-1.5-only: LTX-2 runs batch-CFG in one forward and the Wan pipelines consume each branch inline with no guider combine hook. - video_speedmem_bench gains epc_off (compile-parity isolation) and fbcache_explicit / magcache_explicit configs plus expert-aware cache application mirroring the loader. Measured via scripts/video_speedmem_bench.py and the round-3 single-load probes; full data and per-family decision table in outputs/video_families_optim_round3.md (workspace). Tests: 235 passing across the five video inference suite files (9 new: per-expert curve resolution and step scaling, uncalibrated-expert refusal, toggle expert threading, wan5b magcache auto load/toggle, ltx2 deny auto+explicit, a14b TE auto-dense); ruff clean. |
||
|
Daniel Han
|
de2f22df2b |
perf(image): compile numeric parity, cache-hook compile arming, FBCache toggle crash fix, TE fp8 zero-row guard
Applies the video round-2 accuracy findings to the image diffusion stack and fixes two real image-path bugs found while measuring. All numbers B200, production settings (family default steps/guidance, 1024px, seed 42, 4 fixed prompts), LPIPS (AlexNet) via the new scripts/image_speedmem_bench.py, which drives the production lever functions in the loader's own order. - inductor precision parity: emulate_precision_casts=True on the regional-compile path (fused pointwise kernels keep fp32 intermediates where eager rounds to bf16 between ops). Pairwise LPIPS of the compiled tier vs the same-stack eager tier: Qwen-Image 0.019 to 0.006 at identical speed (72.4 vs 72.5 ms/step), FLUX.1-dev 0.046 to 0.029 at +2% step time (69.8 vs 68.3, reproduced), FLUX.2-klein-4B 0.018 to 0.017 at identical speed. Snapshot/restored with the other process-wide backend flags so an off load never inherits it. - cache x compile composition: re-point each cache hook's fn_ref.original_forward at a torch.compile'd wrapper of the same bound method (armed only where the speed layer compiled the block; restored before every disable_cache and before the partial-hook cleanup). Qwen-Image FBCache computed steps 91.8 to 71.2 ms (back at the uncached compiled rate), 1.21x end to end (7.36 to 6.06 s per 4 images); FLUX.1-dev already traced through its FBCache hook and is measured neutral (same-process armed vs unarmed latents bit-identical). Skip counts within noise (13 vs 11 of 76; pairwise LPIPS 0.005). - FBCache mid-session toggle crash: diffusers 0.39 caches the HookRegistry child list on first cache_context use, so an uncached generation followed by a 20+-step generation (the auto toggle path) enabled hooks the context never reached and crashed with "No context is set" (reproduced live on FLUX.1-dev). Invalidate the stale child cache after every enable_cache. - TE fp8_dynamic zero-row guard: torchao per-row fp8 derives a per-output-channel scale from the row amax, so an all-zero weight row is 0/0 = NaN. SDXL's text_encoder_2 (OpenCLIP bigG) ships exactly such a row, and every explicit fp8_dynamic SDXL render came out black; keep zero-row Linears dense (LPIPS 0.976 black to 0.096 working). Other families' encoders have no such rows and are byte-identical. - No AUTO TE quant exists on the image branch (text_encoder_quant defaults dense, explicit-only), so the video round's auto-dense retune has no image analogue; the explicit lever's cost is now measured (TE fp8_dynamic alone, LPIPS vs bit-exact: Qwen-Image 0.038, FLUX.1-dev 0.084, SDXL 0.096; no speed win, VRAM -6.5 GB on Qwen-Image) for the docs. Tests: 96 passing across the cache/speed/precision suites (11 new arming, 2 child-registry, 2 zero-row, 4 inductor-flag); ruff clean. |
||
|
Daniel Han
|
7dbdd28161 |
perf(video): accuracy-first round 2 for HunyuanVideo-1.5: compile parity, cache quality presets, dual-GPU CFG
Cuts the shipped default's LPIPS vs the bit-exact reference from 0.224 to 0.139 while going faster (24.9 s to 21.2 s at 720p/33f/30 steps, 22.7x vs reference), and makes the remaining speed/accuracy trade a user knob. - inductor precision parity: set emulate_precision_casts=True for the regional compile (fused pointwise kernels kept fp32 intermediates where eager rounds to bf16 between ops); full-clip LPIPS vs bit-exact 0.221 to 0.052 at zero speed cost. Snapshot/restored with the other process-wide backend flags. - cache x compile composition fix: diffusers cache hooks are torch.compiler.disable'd, so every COMPUTED step ran eager (1.69 vs 1.09 s/step) under MagCache/FBCache in both enable orders. Re-point each hook's fn_ref.original_forward at a torch.compile'd wrapper of the same bound method (armed only where the speed layer compiled the block; restored before every disable_cache so the uncached path stays pristine). Balanced MagCache at 50 steps: 1.48x to 2.17x, identical skip counts, bit-identical uncached rerun after enable/disable cycles. - transformer_cache_quality knob (quality|balanced|fast; API + UI + bench) mapping to (threshold, max_skip_steps, retention_ratio). Auto resolves to the near-lossless quality preset (0.06, 2, 0.3; 1.63-1.64x at pairwise LPIPS 0.05-0.09) for the HunyuanVideo-1.5 families and to balanced (the pre-knob values, byte-identical behaviour) everywhere else. - TE auto-quant resolves dense for HunyuanVideo-1.5: TE fp8_dynamic alone moves the clip to LPIPS 0.236 vs bit-exact for zero speed win (the quantised encoder perturbs the conditioning and the trajectory amplifies it chaotically); VAE fp8 stays in auto (0.053, at the compile floor). Explicit schemes honored. - dual-GPU CFG branch parallelism (new diffusion_cfg_parallel.py): transformer proxy + DiT replica on the most-free second CUDA device + worker thread, branch-routed off the pipeline's own cache_context names. Auto engages only where measured bit-identical (eager tier: max abs diff 0.0, 1.66x); the compiled stack is explicit cfg_parallel=on (1.52x over the sequential default; per-device compiled artifacts differ by 1 bf16 ulp/step, documented in the resolved record). Fail-soft gates: family allowlist, guider CFG, pipeline kind, dense DiT, no offload, free-VRAM check; single-GPU loads are untouched and the memory plan stays single-device. - video API: the transformer_cache literal now accepts auto/magcache (an explicit magcache request was rejected at the pydantic layer); the mxfp8 family deny records the round-2 measurement (block-32 MX scaling fixes the zero-row collapse, no black frames, but is latency-neutral at LPIPS 0.37: fails both ship bars). Measured on B200 via the production lever path (video_speedmem_bench.py, which gained a --cache-quality lever and companion-quant isolation configs). Tests: 441 passing across the video inference suite (32 new for cfg-parallel, 20 for presets/arming, 3 for the inductor flag, 2 for TE auto-dense); ruff clean. |
||
|
pre-commit-ci[bot]
|
b9b1a4b8de |
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci |
||
|
Daniel Han
|
726c0b63a1 |
fix(review): portable bench scripts, accurate VAE auto docs, explicit TE deny
Review round follow-ups: - Drop the machine-specific HF_HOME defaults from the four bench / reproduction scripts (fp8_layer_ablation, hunyuan_int8_profile, quant_accuracy_sweep, video_speedmem_bench); they pointed at a private workspace cache and broke the scripts on any other machine. The standard HF_HOME env override still applies. - Correct the vae_quant 'auto' descriptions (image + video request fields, select_vae_quant_scheme docstring, loader comment) to match the shipped ladder: auto engages layerwise fp8 only; fp8_dynamic is an explicit opt-in and is never picked automatically. - Enforce _TE_FAMILY_SCHEME_DENY on the explicit text-encoder path too, gating the final concrete mode (so an int8 -> fp8 fallback is re-checked), matching the table's documented contract and the VAE module's behavior. Covered by a new test. Also merges origin/image-generation (single-GPU fit-budget fix) to keep the stacked head self-consistent. |
||
|
Daniel Han
|
4ca24886a6 |
fix(te-quant): probe the weight-only NVFP4 kernel for explicit TE nvfp4
The explicit text_encoder_quant=nvfp4 path gated on the transformer smoke probe, which builds the dynamic-activation NVFP4 config, while the TE caster _cast_nvfp4 applies weight-only NVFP4WeightOnlyConfig. On a Blackwell build that carries the weight-only FP4 path but not the dynamic FP4 GEMM, the probe would fail and the encoder would silently stay dense even though the caster would run. Add a dedicated weight-only NVFP4 smoke probe (mirroring _cast_nvfp4's config) and route TE nvfp4 through it; int8 / fp8_dynamic keep the dynamic transformer probe since their TE casters are also dynamic-activation. |
||
|
pre-commit-ci[bot]
|
65d33c85bb |
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci |
||
|
Daniel Han
|
be04ba00f4 |
video/image: honor explicit Speed=off for companions + trim, probe explicit TE kernels, bench fidelity
Address the Codex review round on the video/quant work: - Companion auto-quant now honors an explicit Speed=off. Both loaders already pin the DiT dense under an explicit off (bit-exact reference), but the unset text-encoder / VAE quant still promoted to auto and silently fp8/int8'd the companions, breaking the bit-exact request. An UNSET speed still auto-quantises; an explicit companion scheme still forces it. - The HunyuanVideo joint-attention trim is a speed lever (it swaps to the fused SDPA kernel), so gate it on a non-off speed tier exactly like the adjacent attention-backend selection -- the off path keeps the stock dense-mask attention. - Explicit torchao text-encoder modes (int8 / fp8_dynamic / nvfp4) now run the same kernel smoke test the auto ladder uses. They could clear the capability gate yet fail the real GEMM on a build where quantize_ wraps the encoder but the kernel is broken; the caster's try/except only covers the cast, not the first forward, so the load would report engaged then crash at generation. Now it falls back to dense. Layerwise fp8 has no torchao GEMM, so the probe is a no-op for it. - The trim pre-hook's fallback restores the caller's original kwargs (it may have emptied the image stream / trimmed a text stream before failing), so the stock dense-mask path runs on exactly what it expects, matching the empty-prompt guard. - video_speedmem_bench mirrors the loader: installs the Hunyuan trim before the backend set (gated on an active tier) and skips the auto int8 quant when it is the fp8-denied memory fallback and dense fits resident, so the shipped/auto rows measure what the loader actually runs. Tests: TE explicit-mode kernel probe (+ layerwise-fp8 bypass), trim mid-trim restore, and loader-level speed=off companion suppression + trim skip for both backends. 262 backend tests pass; ruff clean. |
||
|
pre-commit-ci[bot]
|
e7168ef34c |
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci |
||
|
Daniel Han
|
40b285aa7c |
Auto-select text-encoder quantization by GPU and family (default on)
The transformer already defaults to auto-quant (fp8/int8); the companion text encoder was opt-in and stayed dense bf16 unless a scheme was named, even though it is often the largest resident component. Add an auto policy mirroring the transformer's ladder: select_te_quant_scheme walks a per-capability ladder (data-center fp8-GEMM: fp8_dynamic -> int8 -> layerwise fp8; Ampere: int8 -> fp8), reorders int8 first on consumer GDDR parts, falls to layerwise fp8 under group offload (the only offload-safe cast), only picks int8 for a family with a measured keep-bf16 schedule, honors a per-family deny list, and smoke-probes the torchao kernel so a missing build degrades gracefully. The image + video loaders now map an unset text_encoder_quant to auto (explicit none/off stays dense; a named scheme is forced), so the shipped default quantizes the encoder to the fastest accurate scheme for the GPU. Records the engaged scheme in the image resolved-record too. Verified on a B200: auto -> fp8_dynamic, offload -> layerwise fp8. |
||
|
pre-commit-ci[bot]
|
785cedbcd7 |
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci |
||
|
Daniel Han
|
3d27fa7817 |
Bound img2img/inpaint init resolution and exclude the VLM vision tower from NVFP4 text-encoder quant
img2img and inpaint take their output size from the uploaded image and only snap it to a multiple of 16, so an ordinary phone photo (up to the 4096/side decode cap, 4x the txt2img 2048 ceiling and ~16x the area) drove an OOM-scale latent and an opaque 500 on a normal card, while txt2img, upscale, edit, and FLUX.2-klein inpaint are all already megapixel-bounded. Clamp the init longest side to 2048 (the txt2img ceiling) before deriving width/height; edit is exempt since its pipeline resizes to ~1MP internally. _cast_nvfp4 quantized every nn.Linear with no filter, unlike the int8 and fp8 torchao text-encoder modes which exclude the VLM vision tower / lm_head / T5 wo. On qwen-image / qwen-image-edit that 4-bit quantized the Qwen2.5-VL image tower, degrading the edit/image conditioning the sibling schemes protect. Apply the same make_filter_fn exclusion (require_bf16, mirroring _cast_fp8_dynamic). |
||
|
Daniel Han
|
186f381bc6 |
Studio: diffusion UX polish and stronger auto policies (images + video) (#6885)
* Auto policies: deferred dense compile, video compile default, step cache and precision auto Image dense loads with speed unset no longer sit at plain off: the load stays bit-identical eager, and the 3rd generation in a session engages the default compile profile plus the cuDNN attention upgrade mid-session (a one-off image never pays the warmup, repeated use amortises it). Video dense loads resolve straight to the default profile since a clip denoise amortises the compile within a single run, and never to max. Video also gains the image backend's tri-state auto policies: unset step cache now decides from the default schedule and re-checks the actual step count per generation, and unset precision (transformer_quant) hands the decision to the hardware ladder instead of staying off. Memory badge reason now says plainly that everything fits when no offload is planned. * Rename Dtype to Precision, add the video Precision control, step cache Auto option The images Advanced panel's Dtype row is now Precision (same control, clearer name), and the video Advanced panel gains the matching Precision select wired to the load route's existing transformer_quant field, gated to full-pipeline loads the way the image control gates to GGUF. Step cache selects on both pages gain an explicit Auto option as the default (the previous Off default silently behaved as auto and never let anyone pin off), and the Speed and Attention tooltips now state the deferred dense compile and the SageAttention black-frame caveat. * Model catalog: canonical diffusion model groups with device-aware routing One canonical name per image/video model, its published artifacts (GGUF, FP8, bnb-4bit, official BF16) as data, and pure routing helpers: suffix-stripped canonical keys (owner-preserving; cross-owner merges only via explicit aliases), group/artifact lookups, a flat back-compat options shim, load-spec resolution replacing the pages' lookup tables, search matching over old ids and format tokens, the GGUF fit ladder extracted from the variant expander, and pickDefaultArtifact/pickDefaultQuant deciding what a bare group click loads (downloaded first, then the best quality that fits 70 percent of VRAM, GGUF as the safe fallback). Checked by npm run catalog:check, following the i18n:check pattern. * Picker: one canonical row per diffusion model with a format second level The Images and Video pickers now render the curated catalog as one row per model in Recommended: clicking loads the best artifact for the device (the routed GGUF quant, a prequant FP8/bnb-4bit that fits, or the official BF16), and a chevron opens the per-format list, with the GGUF row nesting the usual quant expander. Live HF listing rows that belong to a group are deduplicated, search collapses member repos into their group (old ids and format tokens still match), and the On Device sections group cached member repos under the same canonical name with the per-repo rows inside. Curated groups render from the catalog rather than the HF listing, which finally surfaces LTX-2.3 in the video Recommended list (its hub pipeline_tag is image-to-video, so the text-to-video listing always missed it) and exposes the HunyuanVideo 720p repack next to 480p. Backend: /cached-models now tags trusted video-family repos text-to-video instead of blanket text-to-image, and the pickers admit catalog-known non-unsloth repos On Device, so cached Lightricks/Wan/Hunyuan pipelines finally appear in the Video picker. Chat pickers pass no catalog and are unchanged. * Download formats, tab icons, plain-language train tips, 3-loop autoplay The image Download button becomes a menu: PNG saves the original bytes with the embedded recipe, JPEG and WebP re-encode client-side from the fetched blob (JPEG flattened onto white). The video Download button gains MP4 (original, keeps audio), WebM and GIF; the latter two transcode server-side from the stored MP4 via PyAV (VP9 realtime profile for WebM, ~12 fps adaptive palette for GIF) behind a new gallery export route that 501s with a readable message when a codec is missing. Generated clips no longer loop forever: the player replays a clip three times per selection, then pauses with controls up; a new generation or a refresh gets its own three plays. The Create/Train tabs reuse the sidebar's New Chat and Train icons (TestTubeOutlineIcon moved to a shared lib module), and every Train tab helper text is now one plain sentence. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep the Create/Train tab icon and label on one line TabsTrigger renders its children inside a plain inline span and the Tailwind preflight gives svg display:block, so the HugeiconsIcon forced the label onto a second line. Wrap icon plus label in their own inline flex row inside each trigger. * Strip -int8 and -nvfp4 prequant suffixes in the model catalog key canonicalKeyFor already lowercases before matching, so -GGUF/-FP8 in any case were covered; -int8 and -nvfp4 were not in the suffix table, so such repos rendered as standalone rows in Recommended and On Device instead of standardizing into their base-name group and routing through pickDefaultArtifact. Added both suffixes plus case-insensitivity and routing assertions to the catalog check. * Standardize non-catalog picker rows to their base model name The curated catalog already collapses its own groups, but hub listing rows and cached repos outside the catalog (ERNIE-Image, FLUX.2-klein, Qwen-Image-Edit-2509, FLUX.2-dev) still rendered raw ids with -GGUF / -FP8 style suffixes in Recommended and On Device. - model-catalog.ts: new stripArtifactSuffixesForDisplay, a case-preserving twin of canonicalKeyFor's stripping that keeps the owner prefix and original casing for display. - pickers.tsx: recommended hub rows and the downloaded GGUF/model rows pass their labels through it when a catalog is present, so only the diffusion pickers change; chat rows keep raw ids. Click targets keep the full repo id, and the format badge still shows the artifact kind. - Catalog check covers the new helper across GGUF/FP8/int8/nvfp4 in both cases plus no-op and suffix-only names. * Offer official BF16/FP8 artifacts per model group and fix gallery label clipping Model picker changes so groups are not limited to unsloth quant repos: - model-catalog.ts: each image group that has an official vendor pipeline now carries its BF16 (official) artifact as the top (highest quality) entry - Tongyi-MAI/Z-Image-Turbo, Qwen/Qwen-Image, Qwen/Qwen-Image-2512, Qwen/Qwen-Image-Edit-2511, black-forest-labs/FLUX.1-dev, FLUX.1-schnell and FLUX.1-Kontext-dev. The LTX-2.3 video group now lists Lightricks' own bf16 and fp8 distilled single-file checkpoints alongside the GGUF. Resident sizes are set from the actual weight totals (FLUX ships a duplicate single-file that from_pretrained ignores, so FLUX bf16 is ~32 GB not 54). The repos that used to be aliases are now real artifacts. - The router already prefers the highest-quality artifact that fits the 0.7 x GPU budget, so a datacenter GPU now defaults to official BF16 while consumer GPUs still route to the fitting quant or GGUF. That is why bnb-4bit was the Z-Image-Turbo default before: it was the only non-GGUF artifact and it was already downloaded. - diffusion.py: allowlist the four official image repos not previously trusted (qwen/qwen-image-2512, qwen/qwen-image-edit-2511, black-forest-labs/flux.1-schnell, flux.1-kontext-dev). All verified as safetensors-only diffusers model_index pipelines. The LTX-2.3 checkpoints are already on the video trust list. - catalog check: BF16-wins-on-datacenter, quant-wins-on-consumer, and the single-file load specs for the LTX-2.3 checkpoints. Also fixes the video gallery thumbnail caption: the leading duration was clipped by the rounded corner and selection border, so the strip now has enough left/bottom padding to clear the curve. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * video gallery: guard export transcode against a stream-less clip _transcode_webm and _transcode_gif indexed src.streams.video[0] before checking the stream list, so a container with no video stream raised a bare IndexError that the broad handlers then re-labeled as a missing libvpx or decoder. Raise an explicit RuntimeError naming the real cause in both the WebM and GIF paths. * Studio: honor explicit attention/format choices, fix distilled-LTX defaults and On Device catalog routing * Remove stray planning notes accidentally committed to the branch * video: add transformerQuant to the load callback deps handleLoad reads transformerQuant but omitted it from the useCallback dep array, so after the user changes only Precision and then selects a model or clicks Reapply, the memoized callback keeps the stale closure and loads the previous precision. The image page's equivalent callback already lists it. * model picker: honor the format filter when routing catalog clicks; add catalog rows to the roving list - routedArtifactFor now scopes a group's artifacts to the active format filter (the same matchesFormatFilter predicate the visibility check uses) before pickDefaultArtifact, so a group shown only because it owns a GGUF no longer routes a click to a large non-GGUF download. Covers both the Recommended and On Device grouped paths. - hubOptionKeys now includes the catalog-group, search-catalog-group, and grouped On Device row keys in exact render order, so arrow/Home/End roving reaches the catalog rows instead of giving them a duplicate missing id and skipping them. * model picker: don't treat a partial base cache as downloaded A partially-cached base repo (a cancelled download that left only some weights) was counted as downloaded, so an On Device click routed to a fresh multi-GB re-download instead of the complete GGUF. The picker's endpoint (/api/models/ cached-models) did not carry a partial flag at all, so a frontend-only guard could not see it. Surface partial from that endpoint by reusing the hub inventory scan's snapshot-partial detector, plumb it through CachedModelRepo (backend + frontend types), and skip partial base repos when building the downloaded set. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * model picker + diffusion: drop partial/unloadable cached rows, skip defer-compile before a LoRA gen - On Device (cached non-GGUF) rows filtered partial-download snapshots back in: sortedCachedModels gated on passesTaskGate + a groupForRepoId key match but, unlike downloadedSet, never checked c.partial, so an incomplete unsloth snapshot showed as a loadable On Device row (click errors or silently re-fetches multi-GB). It also admitted repos that only match the catalog by group KEY (a base / uncurated-quant sibling like Qwen/Qwen-Image-2512) which have no loadable artifact and dead-end at the trust gate. Add !c.partial and gate on artifactForRepoId (what loadSpecFor resolves) instead of groupForRepoId, so a cached row shows only when the backend can load it. - Deferred speed-auto engaged the compile profile on the 3rd generation BEFORE _apply_loras. A compiled transformer rejects LoRA (supports_lora is False) and _apply_loras raises before its unchanged-selection no-op, so once compile engaged every LoRA generation on that load failed permanently. Skip the deferral when a LoRA is requested (compile and LoRA are mutually exclusive) and let it engage on a later LoRA-free generation. * Scope the cached-model partial probe to the listed snapshot dir list_cached_models builds each row from the largest/complete copy across HF cache roots, but _cached_repo_partial probed is_snapshot_partial with no repo_cache_dir, so the scan spanned every root: a stale .incomplete copy in one root would flag a complete copy in another as partial and hide the usable model from the picker (the click then routes to a re-download). Forward the winning snapshot's repo_path so all three partial signals are scoped to that copy, matching the sibling inventory paths (models/dataset cache_inventory, local_inventory). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Do not auto-route to gated repos, prefer complete cached copies, defer compile past attached LoRA, scope group expand keys Four fixes: - pickDefaultArtifact's not-downloaded ladder returned the gated BF16 FLUX.1-dev / Kontext-dev before the open GGUF on a large GPU, so a bare group click routed to a repo the user may lack license/token access to. Add a gated flag and skip gated artifacts in the not-downloaded ladder (an already-downloaded gated artifact is still returned). - list_cached_models picked the largest duplicate cache copy and computed partial only on it, so a larger partial copy shadowed a smaller complete one; since partial rows are dropped from the picker the usable model vanished. Prefer completeness, then size. - the deferred-speed compile engaged on a no-LoRA generation while an adapter from a prior generation was still attached, baking it into the compiled graph (the later unload is swallowed on a compiled pipe); also defer while adapters remain attached. - routeGroupClick's GGUF fallback toggled the context-free canonicalId while the chevron toggles the context-scoped expandKey, leaving the format list un-collapsible in one context, dead in the other, and risking cross-context expansion; thread expandKey through. * Guard video pipeline repos from deletion, drop the always-failing LTX FP8 artifact, prefer 720p Hunyuan Three round-6 fixes: - cached non-GGUF video repos now surface in the Video On-Device picker with the normal delete action, but /delete-cached only guarded chat + the Images engine, so a loaded/loading Wan / LTX / Hunyuan pipeline could have its HF snapshot removed from under it. Add a VideoBackend loading_repo_ids accessor and a video loaded/loading guard mirroring the Images one. - the catalog advertised Lightricks/LTX-2.3-fp8 as loadable, but the LTX-2.3 loader refuses the official scaled-FP8 single file (.weight_scale/.input_scale) and points to GGUF/BF16, so a pick routed to a ~76 GB download that always fails on load. Remove the FP8 artifact. - pickDefaultArtifact only sorts by format, so the HunyuanVideo group's 480p (listed first) beat the 720p even on GPUs where 720p fits the budget. List 720p first so the fit loop prefers it and falls back to 480p only on smaller cards. * diffusion: add compute int8/fp8_dynamic text-encoder quant, wire into video Add two torchao compute text-encoder quant modes to the diffusion precision engine, alongside the existing layerwise fp8 and weight-only nvfp4: - int8: per-token activation + per-channel weight (torch._int_mm), with per-layer keep-bf16 selection. int8 degrades on large encoders unless the most quant-sensitive decoder blocks stay bf16, so it engages only for families with a measured keep-bf16 schedule (qwen-image / qwen-image-edit keep first+last 6, flux.2-dev keeps first 3); a family without one falls back to fp8. - fp8_dynamic: per-row fp8 compute (torch._scaled_mm), keeping the matmul in fp8 on the tensor cores instead of upcasting each forward like the layerwise fp8. The selective int8 caster reuses the committed transformer-quant factory (_make_quant_config / make_filter_fn / exclude_tokens_for_scheme) plus a small structural first/last-N block skip, so it depends only on committed APIs. Wire text-encoder quant into the video backend, which previously loaded the companion encoder (Gemma3 / UMT5 / Qwen2.5-VL) dense bf16 while quantising only the DiT. text_encoder_quant is plumbed through the load request, validation, the load chain, the resolved record, and status, mirroring the image backend; it applies for every load kind (the encoder is dense regardless of how the DiT was sourced). Widen the image and video load request Literals and add the video status field. Tests: int8 family-schedule routing and fp8 fallback, fp8_dynamic routing, hardware gates (int8 sm_80+, fp8_dynamic sm_89+), the structural block selection, the real int8 filter closure (keeps the first blocks plus the vision tower / lm_head / T5 wo dense), and the video route threading and 422 validation. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * text-encoder quant: skip the torchao modes under offload (both backends) quantize_text_encoders applied int8-with-schedule / fp8_dynamic / nvfp4 (all torchao) to the text encoder regardless of the offload policy. An offload placement then moves the quantized encoder with Module.to(), which torchao tensor subclasses reject (aten._has_compatible_shallow_copy_type is unimplemented) -- a hard crash, the same one the DiT path already skips torchao quant under offload to avoid. Add offload_active to quantize_text_encoders and skip the torchao modes when set; layerwise fp8 is not torchao and still streams under offload. Both the video and image loaders pass offload_active = (offload policy != none). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * diffusion: skip non-bf16 linears for scaled_mm quant schemes The fp8 / mxfp8 / nvfp4 schemes run on torch._scaled_mm and the fp4 / mx GEMMs, which assert a bfloat16 input weight. On a mixed-precision DiT that keeps some linears in fp32 for numerical stability (the Wan and Hunyuan video transformers do this), quantize_ hits the first fp32 linear, raises, and the best-effort wrapper swallows it to None, so the whole transformer stays dense with no error and no speedup or memory saving. Add a require_bf16 gate to make_filter_fn and pass it for the scaled_mm schemes in quantize_transformer (and the fp8_dynamic text-encoder caster). The gate skips non-bf16 linears so the scheme engages on the bf16 ones. int8 uses torch._int_mm, which quantizes fp32/fp16 weights fine, so it leaves the gate off and keeps its current coverage. Verified on Wan2.2-TI2V-5B: fp8 and mxfp8 now quantize 303 linears via the committed quantize_transformer path where they previously engaged 0. * prequant builder: mirror the scaled-mm bf16 gate offline The runtime DiT quantizer skips non-bf16 Linears for the scaled_mm schemes (fp8, nvfp4, mxfp8) so the scheme engages on a mixed-precision transformer instead of aborting on the first fp32 Linear. The offline prequant builder reused make_filter_fn without that gate, so building an fp8/nvfp4/mxfp8 checkpoint for a mixed-precision DiT (Wan, Hunyuan keep _keep_in_fp32_modules in fp32 even under torch_dtype=bf16) would hit the same fp32 Linear and abort, breaking the builder's stated offline == runtime, LPIPS-0 invariant. Thread require_bf16 = scheme in _SCALED_MM_SCHEMES through the builder, record it in the checkpoint metadata, and verify it on load (mirrors the existing exclude_name_tokens guard) so a future _SCALED_MM_SCHEMES change cannot silently load a checkpoint built under the old filter. * Keep nvfp4 fp32 linears quantised (bf16 gate is fp8/mxfp8 only) Verified on torchao 0.17 / B200: fp8 per-row asserts 'PerRow quantization only works for bfloat16 precision input weight' and mxfp8 asserts 'Only supporting bf16 out dtype', but NVFP4's high-precision conversion quantises an fp32 weight fine (forward included). So the bf16 skip-gate must be fp8/mxfp8 only, not all scaled_mm schemes -- otherwise nvfp4 leaves large fp32 projections dense, losing the intended memory/speed gain. Rename _SCALED_MM_SCHEMES -> _REQUIRE_BF16_SCHEMES = (fp8, mxfp8) and thread it through the runtime filter, the offline builder, and the loader require_bf16 verification (offline == runtime preserved). * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |
||
|
Daniel Han
|
48628252bd |
Merge remote-tracking branch 'origin/image-generation' into diffusion-phase4-native
# Conflicts: # scripts/diffusion_bench.py # scripts/diffusion_quality.py # studio/backend/core/inference/diffusion.py # studio/backend/core/inference/diffusion_device.py # studio/backend/core/inference/diffusion_families.py # studio/backend/core/inference/diffusion_memory.py # studio/backend/core/inference/diffusion_precision.py # studio/backend/core/inference/diffusion_speed.py # studio/backend/models/inference.py # studio/backend/routes/inference.py # studio/backend/tests/test_diffusion_backend.py # studio/backend/tests/test_diffusion_device.py # studio/backend/tests/test_diffusion_memory.py # studio/backend/tests/test_diffusion_precision.py # studio/backend/tests/test_diffusion_speed.py |
||
|
oobabooga
|
8b69f22809 | Fix fp8 text-encoder quant crashing generation on T5 and tied-embedding encoders | ||
|
Daniel Han
|
6ae6fb1c45 |
Studio diffusion (Phase 2): memory planner, streamed offload, fp8 TE, speed layer, quality harness (#6675)
--------- Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com> |
||
|
pre-commit-ci[bot]
|
7ca5573498 |
[pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci |
||
|
Daniel Han
|
dbb0292561 |
Studio diffusion (Phase 2C): NVFP4 text-encoder quant (+ generalise fp8 knob)
Generalise the text-encoder precision knob from a fp8 bool to text_encoder_quant (fp8 | nvfp4). nvfp4 quantises the companion text encoder to 4-bit via torchao NVFP4 weight-only (two-level microscaling) on Blackwell's FP4 tensor cores; fp8 stays the broader-hardware path (cc>=8.9). Both are gated, best-effort, and run before placement; status reports the mode actually engaged. This is the lean realisation of GGUF-native text-encoder quant: 4-bit on the encoder without the 3045-line port. Verified on Z-Image (B200, balanced/group where the encoder stays resident), vs the bf16 encoder: nvfp4 cut generation peak VRAM 48% (10840 -> 5593 MB, the lowest TE option, below whole-model offload) at near-fp8 quality (16.4 vs 17.1 dB PSNR), and both quants ran faster than bf16. A memory-vs-quality tradeoff (off by default); size it per model with the Phase 5 quality harness. diffusion_bench gains --text-encoder-quant. 129 CPU tests pass. |
||
|
Daniel Han
|
817e14ad19 |
Studio diffusion (Phase 2B): opt-in fp8 text-encoder layerwise casting
Add a text_encoder_fp8 knob that casts the companion text encoder(s) to fp8 (e4m3) storage via diffusers apply_layerwise_casting, upcasting per layer to the bf16 compute dtype while normalisations and embeddings stay full precision. Applied before placement, gated to CUDA + bf16, best-effort (a failure leaves the encoder dense). status reports which encoders were cast. Verified on Z-Image (B200, balanced/group mode where the encoder stays resident): generation peak VRAM dropped 37% (10840 -> 6791 MB, below the lowest-VRAM offload) at near-resident speed. It is a memory-vs-quality tradeoff, not free -- ~20 dB PSNR vs the bf16 encoder, a larger shift than one transformer quant step -- so it is off by default and documented as such, with the Phase 5 harness to size the cost. 127 CPU tests pass. |