Review follow ups on the more-families branch: the per channel scale now
broadcasts rank aware instead of assuming 2D (all shipped tensors are 2D,
verified across all three fp8 components, but a future non 2D quantized
tensor would have mis broadcast silently), the fused qkv split asserts the
expected 3x hidden row count so a GQA style export fails loudly, fp8
detection scans every shard header rather than the first, and the excluded
model match uses the segment aware token helper with a hunyuanimage-3
token so a future HunyuanImage 2.x is not blocked with a 3.0 reason.
The wheel-only pip install for an optional attention kernel ran inside
load_pipeline under _lock and _generate_lock, so a slow or hanging install
blocked unload and cancellation for up to the 600s timeout. Resolve and install
the kernel before taking the locks (only an explicit backend ever pulls a
package, and its resolution ignores the speed tier); the in-lock apply call is
then a fast no-op. Also decode and log pip's stderr on a failed install so the
fallback to native is diagnosable instead of showing only the exit code.
The raw speed_mode string was forwarded to _enable_fp16_accumulation, so a
case-variant like MAX failed the speed_mode != SPEED_MAX check and wrongly
disabled fp16 accumulation on float16 pipelines. Forward the normalized mode
and cover the case-insensitive path in the test.
The ideogram-ai/ideogram-4-fp8 repo stores its two DiTs and the Qwen3-VL text
encoder in a vendor float8 layout that diffusers 0.39.0 (and diffusers main)
cannot read, so a stock Ideogram4Pipeline.from_pretrained produced a pipeline
with randomly initialized attention weights left on the meta device: the load
then died at pipe.to(device) with "Cannot copy out of meta tensor", and any load
that got past that would have generated noise.
Two things broke:
- The DiT attention is stored FUSED as attention.qkv.weight ([3*hidden, hidden],
Q/K/V rows stacked) plus attention.o.weight, while the diffusers transformer has
split to_q/to_k/to_v/to_out.0. from_pretrained mapped neither name and left them
meta + random.
- Every quantized weight is float8_e4m3 with a per-output-channel weight_scale;
the real weight is fp8.float() * weight_scale[:, None]. diffusers dropped the
scales and loaded the raw fp8 values (range +-448) as the weights, so even the
weights that did map were wrong.
load_ideogram4_transformer now reads the shards, dequantizes every scaled weight,
splits the fused qkv into to_q/to_k/to_v and renames o to to_out.0, then loads the
result into a config-constructed model. It fails loudly if any key stays unmatched
so a partly random model can never ship. The dequantized fp8 projections match the
byte-identical -nf4 export (already in the diffusers split layout with a bnb
quantization_config) to cosine ~0.997, so the split order and scale axis are
confirmed. The conversion is gated on the fp8 marker (a *.weight_scale key) read
from the shard header only, so the -nf4 repos skip it and load through the stock
from_pretrained path without a wasteful full-shard read.
The fp8 text encoder needed the same float8 dequant (its keys already match the
transformers Qwen3-VL module, so no rename). load_ideogram4_text_encoder handles
the fp8 repo and delegates the bnb-4bit and dense repos to the shared krea shim.
One more incompatibility was in the diffusers pipeline itself: it calls
transformers create_causal_mask(inputs_embeds = ...) with no cache_position, but
on transformers 4.57.6 the parameter is spelled input_embeds and cache_position is
required. _patch_create_causal_mask installs a signature-aware wrapper that renames
the kwarg and supplies cache_position, and is self-disabling on a matching signature.
Adds unit tests for the fp8 dequant/split conversion and the causal-mask patch.
Verified live on a B200: ideogram-4-fp8 (both CFG paths), ideogram-4-nf4-diffusers,
and krea-2 with the retroanime LoRA all load and generate coherent images.
Measured on the fresh linux x64 prebuilt (z-image Q8_0, sd-cli, 192 CPU
threads, 512x512, 9 steps, steady state): sampling 56.1s vs 51.3s (about
9 percent faster), VAE decode unchanged, peak RSS identical. The sd.cpp
engine only serves the no-GPU tier, so the default profile now matches
max: --diffusion-fa plus --diffusion-conv-direct.
Ideogram 4 (diffusers 0.39 Ideogram4Pipeline) as a new image family. The vendor
publishes no bf16 checkpoint, so ideogram-ai/ideogram-4-fp8 (raw float8 DiTs,
upcast by from_pretrained) is the family base and ideogram-4-nf4-diffusers is
the bnb-4bit pipeline artifact (ideogram-4-nf4 is byte-identical and detects to
the same family). All three repos join the trusted non-GGUF allowlist and the
frontend safetensors catalog.
Family specifics handled:
- Dual-branch CFG runs through a SEPARATE unconditional_transformer, so the
auto-policy size table entry counts two ~9.3B DiTs (37.2 GB bf16), and the
pipeline-kind memory plan now takes max(cached bytes, family table) for the
family base repo: the fp8 repo's cached bytes undershoot the bf16-resident
footprint by ~2x, which would let auto planning pick a resident placement
that OOMs.
- The pipeline accepts EITHER guidance_scale OR a per-step guidance_schedule
(its default: the recommended 45x7.0 + 3x3.0 taper, valid only at 48 steps)
and raises when both are set. At the advertised defaults (48 steps, guidance
7) generate() drops the constant so the recommended taper engages; any other
request nulls the schedule so the constant broadcasts legally.
- Generation defaults per the model card: 48 steps, guidance 7 (both tables).
tencent/HunyuanImage-3.0 is deliberately excluded: it has no diffusers pipeline
(an 80B autoregressive MoE behind trust_remote_code). A structured exclusion
map now surfaces that reason verbatim from validate_load_request instead of
the generic unknown-family error.
The curated diffusion LoRA catalog gains the nine official krea/Krea-2-LoRA-*
style adapters (family-tagged krea-2, explicit weight filenames), so they show
up in the picker instead of requiring a typed repo id.
Tests: new test_diffusion_more_families.py (detection, trust, defaults, size
table, exclusion reason, curated catalog + family filter), two generate()
tests for the guidance_scale/guidance_schedule pairing, and the local-scan
LoRA test updated for a non-empty curated list. Backend suite + CI-sim
(block_diffusers/block_torchao) green; frontend builds.
An unset transformer_quant used to mean off (run the GGUF as-is), so the
hardware ladder only engaged when auto was explicitly chosen and the panel
showed Off as the default. Unset (or auto) now hands the decision to the
ladder: a dense-capable GPU gets at least int8, data-center silicon fp8,
falling back to the GGUF when the device, VRAM, family deny table or disk
cannot take it. An explicit none/off pins GGUF-as-is and is now
expressible in the API (previously only omission meant off, so pinned-off
and unset were indistinguishable); an explicit scheme pins that scheme.
The dense candidate also gains a free-disk gate: with auto as the default
the bf16 base download (up to ~40 GB) must never wedge a nearly-full
model-cache disk, so the candidate is dropped (GGUF build kept) when free
space cannot hold it plus a 10 GiB margin. Unprobeable disk passes.
Frontend: the Dtype select defaults to Auto (fastest for GPU), keeps Off
as an explicit choice, and sends none through instead of omitting it.
Suite: 622 diffusion tests green (default-load test rewritten to the new
contract, explicit-off short-circuit covered), CI-sim green.
resolve_dense_quant_candidate now passes fam.name to
select_transformer_quant_scheme so the policy's proposed scheme honors the
family deny table (qwen-image lands on int8 instead of proposing fp8 that
the execution path would refuse). Test stub updated for the new keyword.
A 28-pair accuracy gate on a B200 (same-seed vs the dense bf16 reference)
found per-row fp8 dynamic quant renders EVERY qwen-image frame black
(mean luma 0.0000, SSIM 0.016), reproduced identically with on-the-fly
quantize_ on the dense transformer, so it is the model's activation range,
not a checkpoint artifact. mxfp8 shows real semantic damage at 1024px
(CLIP delta mean 0.0146, worst cases 0.064/0.102) and nvfp4 measures
LPIPS mean 0.51. int8 dynamic (per-token scales) is excellent on Qwen:
LPIPS mean 0.069, SSIM 0.958.
The per-scheme smoke probe only proves the GEMM kernel runs, so it cannot
catch model-level breakage. Add _FAMILY_SCHEME_DENY consulted by
select_transformer_quant_scheme: auto skips denied schemes (Qwen lands on
int8) and an explicit denied request returns None, the same GGUF-fallback
contract as an unsupported scheme. Family is threaded from the three
diffusion.py call sites; existing behavior is unchanged for every other
family. 4 new tests; 529 diffusion tests green; CI-sim green.
GET /api/inference/images/info returns each family's bf16 component sizes and
the estimated resident GB under bf16/int8/fp8/mxfp8/nvfp4, computed purely from
the auto-policy tables (no GPU probing, torch-free), so the panel can show the
Dtype tradeoff before anything is loaded.
DiffusionStatusResponse gains an additive resolved field: per-control
{value, source, reason} provenance the loader already records. The Advanced
panel renders a muted Auto: X pill next to Speed / Dtype / Attention / Memory /
Step cache / CPU offload when the backend decided that control (source auto),
with the reason as the tooltip; an explicit user choice renders no badge.
threads = None let sd.cpp default to the logical-core count. The diffusion CPU
path is compute-bound GGML matmuls, where oversubscribing hyperthreads adds
scheduling contention without extra throughput, so both the persistent server
and the one-shot sd-cli now pass cpu_count // 2 (min 1, fallback 8).
Attention: apply_attention_backend now best-effort installs the package an
explicitly requested optional backend needs (sage -> sageattention, flash ->
flash-attn, flash3/flash4 -> kernels, xformers), wheel-only via pip
--only-binary=:all: so a host without a CUDA toolchain never starts a source
build. Gated by UNSLOTH_DIFFUSION_ATTENTION_INSTALL (auto|0), mirroring the
sd.cpp prebuilt installer gate, and only reached after the arch gating in
select_attention_backend, so no install is attempted for a kernel this card
cannot run. Any failure keeps today's native fallback.
Step cache: transformer_cache gains a real auto state (unset or "auto"). At
load the policy engages FBCache when the model's default schedule reaches
FBCACHE_MIN_STEPS = 20 (dev-style 28-step models win ~1.4x; 4-9-step distilled
models never engage, a skipped step costs too much there). generate() then
re-checks the ACTUAL step count and toggles the cache idempotently across the
bar, so one resident load serves both a 28-step and a 4-step request with the
right cache state, and status/resolved provenance follow the toggle. An explicit
off or fbcache request is pinned and never toggled. Compile drops fullgraph when
an auto cache could still engage on a cache-capable transformer, since enabling
FBCache under a fullgraph-compiled transformer would crash.
Verified on GPU: flux.1-schnell load starts uncached (4-step default), engages
fbcache at 24 steps, disengages at 4, re-engages at 28, with images at each
step and the provenance record tracking each transition.
The A/B harness measured two regimes. bf16 loads (the Studio default on Ampere+)
are bit-identical with the flag on across all six families, 36/36 same-seed cases,
because the flag only changes fp16 GEMM accumulation. fp16 loads (the pre-Ampere
fallback dtype) show real same-seed drift on the families that genuinely run fp16
GEMMs: SDXL up to 0.050 mean abs diff, FLUX.1 0.028, FLUX.2-klein 0.045, all
finite, no new black frames. qwen-image renders black in fp16 with the flag off
too and z-image fp16 fails in attention, so both are dtype limitations, not
accumulation ones.
So the gate now takes the compute dtype and the speed tier: bf16 engages on any
active tier (provably output-neutral), fp16 engages only under max, the tier that
already trades exactness for measured speed. The deny-list stays empty by
measurement.
fp16 accumulation (torch.backends.cuda.matmul.allow_fp16_accumulation) roughly
doubles fp16 GEMM throughput on consumer tensor cores by keeping the accumulator
in fp16. The flag only affects fp16 GEMMs: bf16-compute DiT families are untouched
by construction, while SDXL's fp16 UNet and any fp16 text encoder or VAE path get
the speedup.
Gate in apply_speed_optims: CUDA target, consumer GPU (datacenter parts keep fp32
accumulation), torch exposes the flag, family not in _FP16_ACCUM_DENY, and the
UNSLOTH_DISABLE_FP16_ACCUM kill switch is unset. The flag is captured in
snapshot_backend_flags and restored on unload like the other process-wide knobs.
_FP16_ACCUM_DENY starts empty: a same-seed A/B harness (off vs on per family at
512 and 1024 with long-prompt and high-guidance stress cases, non-finite, black
frame and drift checks) backs the empty list and populates it if a family ever
overflows.
