resolve_dense_quant_candidate now passes fam.name to
select_transformer_quant_scheme so the policy's proposed scheme honors the
family deny table (qwen-image lands on int8 instead of proposing fp8 that
the execution path would refuse). Test stub updated for the new keyword.
A 28-pair accuracy gate on a B200 (same-seed vs the dense bf16 reference)
found per-row fp8 dynamic quant renders EVERY qwen-image frame black
(mean luma 0.0000, SSIM 0.016), reproduced identically with on-the-fly
quantize_ on the dense transformer, so it is the model's activation range,
not a checkpoint artifact. mxfp8 shows real semantic damage at 1024px
(CLIP delta mean 0.0146, worst cases 0.064/0.102) and nvfp4 measures
LPIPS mean 0.51. int8 dynamic (per-token scales) is excellent on Qwen:
LPIPS mean 0.069, SSIM 0.958.
The per-scheme smoke probe only proves the GEMM kernel runs, so it cannot
catch model-level breakage. Add _FAMILY_SCHEME_DENY consulted by
select_transformer_quant_scheme: auto skips denied schemes (Qwen lands on
int8) and an explicit denied request returns None, the same GGUF-fallback
contract as an unsupported scheme. Family is threaded from the three
diffusion.py call sites; existing behavior is unchanged for every other
family. 4 new tests; 529 diffusion tests green; CI-sim green.
train_precision_modes gates int8/fp8/mxfp8 on has_functional_torchao, and the
Backend CI runner does not install torchao, so the three capability-gating
tests collapsed to nf4/bf16/auto and failed. They exercise the CAPABILITY
gate, not torchao presence: stub the probe functional alongside the CUDA
capability patch. Validated with a torchao-blocked run (22 passed).
GET /api/inference/images/info returns each family's bf16 component sizes and
the estimated resident GB under bf16/int8/fp8/mxfp8/nvfp4, computed purely from
the auto-policy tables (no GPU probing, torch-free), so the panel can show the
Dtype tradeoff before anything is loaded.
DiffusionStatusResponse gains an additive resolved field: per-control
{value, source, reason} provenance the loader already records. The Advanced
panel renders a muted Auto: X pill next to Speed / Dtype / Attention / Memory /
Step cache / CPU offload when the backend decided that control (source auto),
with the reason as the tooltip; an explicit user choice renders no badge.
threads = None let sd.cpp default to the logical-core count. The diffusion CPU
path is compute-bound GGML matmuls, where oversubscribing hyperthreads adds
scheduling contention without extra throughput, so both the persistent server
and the one-shot sd-cli now pass cpu_count // 2 (min 1, fallback 8).
Attention: apply_attention_backend now best-effort installs the package an
explicitly requested optional backend needs (sage -> sageattention, flash ->
flash-attn, flash3/flash4 -> kernels, xformers), wheel-only via pip
--only-binary=:all: so a host without a CUDA toolchain never starts a source
build. Gated by UNSLOTH_DIFFUSION_ATTENTION_INSTALL (auto|0), mirroring the
sd.cpp prebuilt installer gate, and only reached after the arch gating in
select_attention_backend, so no install is attempted for a kernel this card
cannot run. Any failure keeps today's native fallback.
Step cache: transformer_cache gains a real auto state (unset or "auto"). At
load the policy engages FBCache when the model's default schedule reaches
FBCACHE_MIN_STEPS = 20 (dev-style 28-step models win ~1.4x; 4-9-step distilled
models never engage, a skipped step costs too much there). generate() then
re-checks the ACTUAL step count and toggles the cache idempotently across the
bar, so one resident load serves both a 28-step and a 4-step request with the
right cache state, and status/resolved provenance follow the toggle. An explicit
off or fbcache request is pinned and never toggled. Compile drops fullgraph when
an auto cache could still engage on a cache-capable transformer, since enabling
FBCache under a fullgraph-compiled transformer would crash.
Verified on GPU: flux.1-schnell load starts uncached (4-step default), engages
fbcache at 24 steps, disengages at 4, re-engages at 28, with images at each
step and the provenance record tracking each transition.
The A/B harness measured two regimes. bf16 loads (the Studio default on Ampere+)
are bit-identical with the flag on across all six families, 36/36 same-seed cases,
because the flag only changes fp16 GEMM accumulation. fp16 loads (the pre-Ampere
fallback dtype) show real same-seed drift on the families that genuinely run fp16
GEMMs: SDXL up to 0.050 mean abs diff, FLUX.1 0.028, FLUX.2-klein 0.045, all
finite, no new black frames. qwen-image renders black in fp16 with the flag off
too and z-image fp16 fails in attention, so both are dtype limitations, not
accumulation ones.
So the gate now takes the compute dtype and the speed tier: bf16 engages on any
active tier (provably output-neutral), fp16 engages only under max, the tier that
already trades exactness for measured speed. The deny-list stays empty by
measurement.
spec.forward imports Krea2Pipeline for prepare_position_ids, so the test needs a
real diffusers install; the backend CI matrix runs without one and failed on the
import. Same importorskip guard the sigmas gather test already uses.
fp16 accumulation (torch.backends.cuda.matmul.allow_fp16_accumulation) roughly
doubles fp16 GEMM throughput on consumer tensor cores by keeping the accumulator
in fp16. The flag only affects fp16 GEMMs: bf16-compute DiT families are untouched
by construction, while SDXL's fp16 UNet and any fp16 text encoder or VAE path get
the speedup.
Gate in apply_speed_optims: CUDA target, consumer GPU (datacenter parts keep fp32
accumulation), torch exposes the flag, family not in _FP16_ACCUM_DENY, and the
UNSLOTH_DISABLE_FP16_ACCUM kill switch is unset. The flag is captured in
snapshot_backend_flags and restored on unload like the other process-wide knobs.
_FP16_ACCUM_DENY starts empty: a same-seed A/B harness (off vs on per family at
512 and 1024 with long-prompt and high-guidance stress cases, non-finite, black
frame and drift checks) backs the empty list and populates it if a family ever
overflows.
The loader used to plan memory from the GGUF file size and only offer the dense
transformer-quant fast path when that plan was already resident, so on a card where
the GGUF forced offload the int8/fp8 build (roughly half the bf16 bytes, or exactly
the quantised size when a pre-quantized checkpoint exists) was never attempted.
diffusion_auto_policy.py is a pure decision layer: a bf16-resident component table
per family (transformer / text encoders / VAE, with base-repo overrides for the
multi-size families), per-scheme size factors with separate steady and transient
(build peak) numbers, and resolve_dense_quant_candidate which the loader now uses to
re-plan memory against the candidate artifact before settling for offload. The
engaged plan is adopted only when the dense build succeeds; the GGUF fallback keeps
its own plan.
Status now carries a resolved provenance record per Advanced control (value, source
auto or explicit, reason) so the UI can label backend decisions.
The always-visible description under the select is gone (the hint tooltip keeps the
full detail) and the no-model gallery placeholder now reads 'Select a diffusion model
to load'.
