fp8 auto precision defaulted to precise accumulate on any non-consumer GPU,
which made fp8 2.05x slower than int8 on RTX 6000 Ada and slower than not
quantising at all. NVIDIA's professional whitepapers do publish equal FP8 rates
for both accumulate modes there, so the hardware premise held, but the cost is
in the cuBLAS path rather than the published rate. Default to fast accumulate:
measured on B200 the flag is a no-op (4096^3 _scaled_mm at 3023.8 vs 3041.8
TFLOP/s, bitwise-identical output, 1.213 s vs 1.230 s end to end), so it is a
large win where it bites and free where it does not. Precise accumulate stays
available via transformer_quant_fast_accum.
Z-Image's DiT is a Lumina2 derivative, so unsloth/Z-Image-GGUF and
unsloth/Z-Image-Turbo-GGUF both declare general.architecture = "lumina2" and the
whole line was tagged image-diffusion-unsupported and hidden from the Images "On
Device" list, though validate_load_request loads them. Resolve shared archs from
the repo/file name like bare "wan" already does, with a test asserting the picker
and the loader agree for every family.
The sage attention on-demand install ran an unpinned `pip install sageattention`,
but PyPI's newest wheel is 1.0.6 and diffusers refuses anything below 2.1.1: the
install always "succeeded", wrote an unusable version into the running venv, and
was rejected on the next line. Carry the dispatcher's floor so pip resolves
nothing instead.
The dense-quant disk gate sized the download from the bf16-RESIDENT table. The
fp32 families download twice that (Z-Image: 23,479 MiB against a 21,970 MiB
gate), leaving a window where the check passed and the download filled the disk;
Ideogram 4 ships fp8 and was overcharged the other way. Size the gate by
published bytes, verified against HF sibling metadata for all 12 families.
Patch installs went through unsloth_zoo, which refuses to import unless
UNSLOTH_IS_PRESENT is set, and that is set by unsloth itself. The server imports
unsloth at boot so it never showed there, but any other process ran silently
unpatched with every install returning False, which is 13 test failures on a
clean environment. Import unsloth and retry once, memoised per process.
Also: the GGUF+LoRA refusal pointed at the native engine without saying a GPU
host only selects it under UNSLOTH_DIFFUSION_ENGINE=sd_cpp, so the suggestion was
unreachable; the gallery recipe recorded loras from the generate request alone,
losing a load-time bake; load-progress claimed "40.07 GB downloaded" for a fully
cached load; and pickers.tsx imported three catalog-group helpers it never used.
Reported by oobabooga.
Comment-only pass over the Python this PR touches: drop what the code already
says, collapse multi-line explanations that still read on one line, and keep
the reasoning that is not recoverable from the code. No code, docstring
semantics or behaviour changes; verified with an AST comparison against the
previous revision, and the backend suite is unchanged (same 37 environment
failures as before: the API integration tests that need a live keyed server,
the flash-attn install hooks, and the GPU memory fields).
One family entry covers several published variants whose weights differ
(flux.1: schnell, dev, Krea-dev), but prequant resolution was keyed on
(family, scheme) alone, so only the default base could ever be served: the
loader's baked base_model_id validation correctly refused the schnell
checkpoint for dev and Krea-dev bases and every such load paid the dense
download plus on-the-fly quantise.
Add an optional prequant_variant_repos table on DiffusionFamily as
(base_repo, scheme, repo_id) triples and thread the resolved base repo
through resolve_prequant_source / usable_prequant_source and their three
call sites (load fast path, memory-plan probe, auto-policy candidate). A
base without its own entry keeps returning the family default, preserving
the existing refuse-then-dense behavior exactly.
Wire the flux.1 variants: the gate-validated unsloth/FLUX.1-dev-FP8
checkpoints (built in the earlier campaign but never reachable) and the
new unsloth/FLUX.1-Krea-dev-FP8.
- diffusion_cache: do not engage FBCache when the selected pipeline opens no cache_context.
A CacheMixin transformer is necessary but not sufficient -- Flux Kontext / img2img /
inpaint / controlnet reuse the CacheMixin FluxTransformer2DModel yet their __call__ never
opens a cache_context, so the First-Block-Cache hook raised 'No context is set' on the
first forward, crashing every default FLUX.1-Kontext edit (28 steps, above the FBCache
threshold). Detect it from the pipeline __call__ source, resolved off the instance so the
per-expert proxy view delegates to the real pipe.
- diffusion_attention: honor an explicit aiter backend on ROCm/AMD targets instead of
dropping it via the NVIDIA-only guard (aiter is the AMD ROCm kernel; it only works there).
- video: clear the CUDA cache on a failed load so a partially built pipeline's reserved VRAM
does not OOM the next load (mirrors the image backend), and re-check cancellation after the
export/mux so a clip cancelled during the blocking encode is discarded, not persisted.
- diffusion_auto_policy / diffusion_prequant: validate a request-supplied prequant path
override (present AND allowlisted) before budgeting the small prequant plan, so the loader
does not skip the dense shards and then rebuild dense after evicting the resident pipeline.
- diffusion_controlnet: family-gate a curated ControlNet addressed by its full repo id, not
only its short catalog id, so a cross-family repo id 400s up front instead of downloading
and loading through the wrong ControlNet class.
The two candidate-resolution tests exercised the real cache-disk gate, so a
small CI disk (< transient + 10 GiB free) dropped the candidate and the tests
failed non-deterministically across runners. Neutralize the disk probe in the
shared selector helper; the two disk-gate tests re-patch it to cover the gate.
An unset transformer_quant used to mean off (run the GGUF as-is), so the
hardware ladder only engaged when auto was explicitly chosen and the panel
showed Off as the default. Unset (or auto) now hands the decision to the
ladder: a dense-capable GPU gets at least int8, data-center silicon fp8,
falling back to the GGUF when the device, VRAM, family deny table or disk
cannot take it. An explicit none/off pins GGUF-as-is and is now
expressible in the API (previously only omission meant off, so pinned-off
and unset were indistinguishable); an explicit scheme pins that scheme.
The dense candidate also gains a free-disk gate: with auto as the default
the bf16 base download (up to ~40 GB) must never wedge a nearly-full
model-cache disk, so the candidate is dropped (GGUF build kept) when free
space cannot hold it plus a 10 GiB margin. Unprobeable disk passes.
Frontend: the Dtype select defaults to Auto (fastest for GPU), keeps Off
as an explicit choice, and sends none through instead of omitting it.
Suite: 622 diffusion tests green (default-load test rewritten to the new
contract, explicit-off short-circuit covered), CI-sim green.
resolve_dense_quant_candidate now passes fam.name to
select_transformer_quant_scheme so the policy's proposed scheme honors the
family deny table (qwen-image lands on int8 instead of proposing fp8 that
the execution path would refuse). Test stub updated for the new keyword.
The loader used to plan memory from the GGUF file size and only offer the dense
transformer-quant fast path when that plan was already resident, so on a card where
the GGUF forced offload the int8/fp8 build (roughly half the bf16 bytes, or exactly
the quantised size when a pre-quantized checkpoint exists) was never attempted.
diffusion_auto_policy.py is a pure decision layer: a bf16-resident component table
per family (transformer / text encoders / VAE, with base-repo overrides for the
multi-size families), per-scheme size factors with separate steady and transient
(build peak) numbers, and resolve_dense_quant_candidate which the loader now uses to
re-plan memory against the candidate artifact before settling for offload. The
engaged plan is adopted only when the dense build succeeds; the GGUF fallback keeps
its own plan.
Status now carries a resolved provenance record per Advanced control (value, source
auto or explicit, reason) so the UI can label backend decisions.
