Comment-only pass over the Python this PR touches: drop what the code already
says, collapse multi-line explanations that still read on one line, and keep
the reasoning that is not recoverable from the code. No code, docstring
semantics or behaviour changes; verified with an AST comparison against the
previous revision, and the backend suite is unchanged (same 37 environment
failures as before: the API integration tests that need a live keyed server,
the flash-attn install hooks, and the GPU memory fields).
Batched generation: /images/generate takes a prompts list (one image per
prompt, txt2img only) or a seeds list (one prompt, one image per seed);
the legacy batch_size path derives per-image seeds base..base+n-1 like
the native engine. Every image gets its own torch.Generator so any batch
member replays alone from its gallery recipe; the whole list runs as one
forward by default with OOM backoff that halves a failed chunk, and an
explicit batch_size caps images per forward. Validated 10-22x over
serial engines on 32-image suites with LPIPS deltas within 0.002.
Conditioning cache on the inference path: UNSLOTH_DIFFUSION_COND_CACHE_DIR
(the inference sibling of the trainers' cond_cache_dir, same persistent
store) wraps encode_prompt so repeated prompts skip the text-encoder
forward entirely; verified bit-identical outputs. Bypassed while LoRA
adapters are attached; tensor-argument calls pass through uncached.
Compile cache: GGUF loads fingerprint their own bundles (quant=gguf, a
different compiled graph than the dense family) and batched calls
register every distinct (w, h, batch) chunk shape they ran, so the heavy
GGUF batched warmups (~159 s at batch 32 on 12B-class, ~655 s on 20B
CFG-batched) are paid once ever.
GGUF loader: strip the sd.cpp model.diffusion_model. container prefix in
the single-file converter; diffusers' FLUX.2 converter KeyErrors on it
and the Qwen-Image identity mapping strands the model on meta.
diffusion_compile_cache: auto mode now saves the Mega-cache bundle after the
first compiled generation (UNSLOTH_DIFFUSION_COMPILE_CACHE_SAVE=0 opts out), so
users get warm restarts without the distributor env; a bundle hit starts clean
(no pointless rewrite of the just-loaded artifacts) and explicit mode 1/on keeps
the distributor-style re-save. New register_shape + manifest shape coverage: a
STATIC compile produces new artifacts per (width, height, batch), so the
generate path registers each generation's shape and an uncovered shape
re-dirties the context, growing the bundle to cover every shape the session
used. Measured (B200, real backend): Qwen-Image deferred gen-3 hitch 29.1 ->
22.2 s warm with bit-identical output (7.9 MB bundle, ~0.5 s save); SDXL gen-3
115.7 -> 24.7 s and a mid-session 768px recompile 65.8 -> 12.6 s (bundle 63.6 ->
98.7 MB after the 768 re-save).
diffusion_speed: U-Net denoisers (UNet2DConditionModel; no _repeated_blocks, so
the regional compile never reached them) now get a whole-module STATIC
torch.compile on the default tier, plus fused QKV projections and a compiled VAE
decode. Measured on SDXL (30 steps / 7.0 / 1024px, 4 prompts, LPIPS vs the
bit-exact reference): 6.16 -> 3.14 s end to end (1.96x) at LPIPS 0.035, steady
state 0.70-0.88 s/image through the real backend. Rejected on measurement:
dynamic=True whole-module (366 s compile for 39.3 ms/step vs static's 73 s for
26.9), regional BasicTransformerBlock only (45.0 ms/step; ResNet convs stay
eager), max-autotune + inductor flags (25.9 ms/step for a 445 s warmup),
channels-last UNet alone (neutral). DiT tiers unchanged: fused QKV measured
exactly neutral under the regional compile (Qwen-Image 6.53 vs 6.52 s), so it
stays max-only there, and the DiT VAE decode stays eager (a few % of a DiT
generation). compiled_shapes_are_static tells the cache layer which loads are
per-shape (max tier, U-Net whole-module).
diffusion: register each generation's shape with the compile cache before the
save, pass pipe.unet to the cache fingerprint when the pipe has no transformer,
and correct the transformer_quant resolved reason on dense loads (it claimed a
GGUF transformer was loaded on every non-quantized pipeline load).
Tests: 333 passing across the related suites (speed 42, compile_cache 27, cache
40, precision 20, backend, base_precision, transformer_quant, memory); ruff
clean. Full measurement record: outputs/image_optim_round2_audit.md.
Adds the opt-in speed path for the GGUF diffusion transformer behind a
selectable speed mode (default off, so output is unchanged until a profile
is chosen):
- diffusion_eager_patches.py: shared eager fast-paths (channels_last,
attention/backend selection, fused norms and QKV) installed at load and
rolled back on unload or failed load.
- diffusion_compile_cache.py / diffusion_gguf_compile.py: a persistent
torch.compile cache and the GGUF-transformer compile wiring.
- diffusion_arch_patches.py: architecture-specific patches.
- diffusion_patch_backend.py: shared install/restore plumbing.
- diffusion_speed.py: speed-profile planning.
Tests for each module plus the benchmarking and probe scripts used to
measure speed, memory, and accuracy of the path.
