Collapse the multi-line comment blocks across the image, video, sd.cpp and diffusion-training code to one or two lines each, and drop comments that only restate the statement below them. Comments only, no code or behaviour changes.
Six tests read family_train_infos() (or a start preflight) without pinning the
host probes, so they only held on a machine with a bf16 accelerator: on a
GPU-less runner the DiT gate empties precision_modes, turns supports_compile
off, and replaces any other preflight message with the no-accelerator note, and
all six failed there. A conftest fixture pins both probes for exactly those
tests, so they assert the family metadata they are about on every host. The
gate's own CPU-only behaviour keeps its dedicated tests.
Comment-only pass over the Python this PR touches: drop what the code already
says, collapse multi-line explanations that still read on one line, and keep
the reasoning that is not recoverable from the code. No code, docstring
semantics or behaviour changes; verified with an AST comparison against the
previous revision, and the backend suite is unchanged (same 37 environment
failures as before: the API integration tests that need a live keyed server,
the flash-attn install hooks, and the GPU memory fields).
Five fixes from a review pass over the diffusion work.
delete-finetuned rmtree'd a model the Images or Video engine was holding: every
guard on that route is chat-only, and Images loads any local path, so deleting a
local diffusion model under the storage root pulled the weights (and the
companion VAE / text encoders sd.cpp re-reads each generation) out from under a
live pipeline. The cached-model route already refuses this; the trained/exported
one now does too, matching by path rather than repo id, and failing open on a
chat-only install so it cannot block ordinary deletes.
A staged download finishing while its page was hidden loaded the model and
evicted whatever the user was actually using: both diffusion pages stay mounted
behind the router and a load takes the GPU unconditionally. The pick is now held
until its page is on screen again, which is also what chat does.
A scoped download could report success having fetched nothing. With Hugging Face
metadata unavailable no manifest is written, so verification is a no-op, and
snapshot_download returns an existing snapshot folder without downloading when
its own repo_info call fails. A repo already on disk from a full snapshot job
(which ignores *.gguf) therefore completed with no weights and auto-loaded
against them. The requested file list needs no network, so it is checked against
the disk directly.
The XET to HTTP retry reclaimed the job slot without the scoped file list, and
that claim overwrites the stored record, so a later identical scoped start
compared an empty list against the real one and 409'd instead of adopting the
running download.
The DiT accelerator gate probed torch.mps.is_available(), which only exists from
torch 2.5 while the supported floor is 2.4. All three probes shared one
try/except, so on torch 2.4 the AttributeError read as 'no block' and a CPU-only
host still evicted the resident pipeline, downloaded the encoders and died in
the child. Each accelerator is probed on its own now, through
torch.backends.mps.
Six review findings, three of them evict-then-fail orderings:
- The chat load reclaimed the GPU without telling the arbiter it existed. A
chat load holds no llama-server process until its GGUF has downloaded,
which is minutes, so a competing Images/Video acquire in that window
found nothing to cancel, took the GPU, and the chat load then spawned
onto the same device. It now registers an in-flight marker through
acquire_for's register hook (under the arbiter lock, as the image and
video loads do), the evictor cancels a marked load, and the route undoes
itself if ownership moved while it loaded.
- The Hub-download conflict check ran after that handoff, so a GGUF the
download manager already owns destroyed the resident Images/Video
pipeline and then 409'd, having loaded nothing. It moves above the
handoff, together with the marker it handshakes with.
- The image load released the engine router's transition lock before
registering the load, so a second load choosing the other engine could
unload the still-idle engine this one captured; the load then landed on a
deactivated engine, where generate, status, unload and the arbiter's
evictor can no longer reach it. Registration now happens under that lock
and refuses if the engine changed.
- Training a DiT family on a host with no GPU was accepted: nf4 is not a
CPU fallback, its 4-bit load goes through bitsandbytes, which requires
CUDA, XPU or MPS. The start unloaded the working Images pipeline, pulled
the text encoders, and only then died in the child. Rejected before the
teardown now, and /info stops advertising a precision that always 400s.
SDXL keeps its documented fp32-on-CPU path.
- Both diffusion pages kept the routed-pick marker forever, so re-picking
the same checkpoint (after chat evicted it) neither loaded nor cleared
the query string. The marker is released once the query is gone. The
Images key also carried a stray NUL byte, which made the file read as
binary to grep and other tooling.
- diffusers dropped Python 3.9 in 0.38, so the unconditional >=0.39.0 pin
left pip no candidate at all on 3.9 and made every install that composes
the huggingface extras unresolvable there. The floor is conditional now.
Also fixes tests that were already red on the branch: two hand-built
request fakes had gone stale against fields this branch added, and the
handoff-ordering test only failed on a host with fewer than two GPUs.
torch.cuda.is_bf16_supported() defaults to counting pre-Ampere bf16 EMULATION as
supported, so on a T4/V100/RTX 20xx the DiT-training bf16 gates all passed even
though the trainer requires native Ampere-or-newer bf16: /diffusion/info advertised
the DiT precision modes, /diffusion/start's preflight let the run through and freed
resident GPU models, then the trainer child hit the real unsupported bf16 path. The
inference device resolver already fixed this (issue #6658) by gating NVIDIA on
capability major >= 8; the training path never got it. Add a shared
native_bf16_supported() helper (NVIDIA cap major >= 8; ROCm keeps the trustworthy
is_bf16_supported()) and use it in the three DiT bf16 sites -- train_precision_modes,
bf16_unsupported_reason, and the trainer guard -- so a pre-Ampere card is offered
nf4 only and never advertises/evicts-then-fails. Tests now exercise the emulation
case (is_bf16_supported True but capability < 8).
Several image/video/training preflights ran before the route acquires the GPU or
frees resident models, but let a doomed local pick through and only failed deep in
the background load, after the user's chat/Images/Video model was already evicted.
- Local base_repo / base_model: _is_trusted_diffusion_repo accepts any existing
local path, but the base loads via from_pretrained (needs model_index.json). A
local dir that is not a diffusers pipeline passed the trust gate, evicted the
resident model, then failed. Add a shared _assert_local_base_is_pipeline check
and call it in the image, video, and training preflights.
- Dataset images: discover_image_caption_pairs only checked filenames, so a
corrupt or zero-byte upload passed the start-route preflight, freed the GPU, then
crashed the spawned trainer in PIL. Add an opt-in verify_images decode probe
(cheap PIL header check) that the start route enables; the trainers leave it off
since they decode every image anyway.
- Local single-file safetensors: the On-Device scanner advertises a bare
.safetensors directory (no model_index.json) as a text-to-image model, but the
picker starts it as a pipeline with no filename, so every click 400s. Reinterpret
such a pick as a single_file load of the sole checkpoint (resolve_local_single_file)
so the advertised model is actually loadable.
Regression tests for each: local non-pipeline base (image/video/training), the
verify_images decode gate, and resolve_local_single_file.
The start route's precision preflight folded bf16/int8/fp8 into the CUDA
requirement but omitted mxfp8, so an mxfp8 request on a GPU-less host (or an
older CUDA GPU without Blackwell) passed the preflight, evicted resident image
and chat models, then raised only in the spawned trainer child. Mirror
_resolve_base_precision: require CUDA for mxfp8 and re-check the Blackwell
(sm100+) capability up front, so a doomed run is rejected before teardown.
The start-route preflight caught the bf16-GPU and int8-torchao requirements but not the dense
precisions' CUDA requirement: on a GPU-less host bf16_unsupported_reason exempts CPU-only, so a
bf16/fp8 (or int8-with-torchao) DiT request passed the preflight, evicted resident workloads, then
raised only in the trainer child. Add the dense-mode CUDA gate mirroring _resolve_base_precision so
the doomed run is rejected up front. Also pin bf16_unsupported_reason in the two positive-path
family-info tests so they are deterministic across GPU types (a non-bf16 CUDA box would otherwise
empty every DiT family's advertised modes).
The start route preflight only rejected non-bf16 GPUs; an explicit int8 request on
a host with a missing or stub torchao passed the preflight, evicted resident GPU
workloads, then died in the trainer child (its int8 base quantizer has no fallback).
Fold both gates into training_precision_preflight_error so int8-without-torchao fails
fast before eviction. Also empty the advertised DiT precision_modes (and surface the
reason in vram_note, drop compile) whenever the bf16 preflight would reject the family,
so /info never offers an nf4 DiT option the route always 400s.
- normalized() + family_train_infos() mirror the inference fp8 deny for
Qwen-Image (activation outliers exceed fp8's range and corrupt the trained
result); int8 stays allowed and the UI no longer advertises fp8 for it.
- _resolve_base_precision() gates an explicit int8 on a FUNCTIONAL torchao, the
same gate auto and /info already apply, so a missing/stub torchao fails fast
instead of silently loading dense with compile disabled.
- train_precision_modes() gates the dense modes (bf16/int8/fp8/auto) on
torch.cuda.is_bf16_supported(), so a non-bf16 CUDA GPU (T4/V100/RTX 20xx) is
offered only nf4 instead of a start that evicts resident models and then fails.
- start_diffusion_training preflights bf16 support for the DiT families BEFORE
_free_gpu_for_diffusion_training(), so any DiT start (nf4 included, since the
trainer requires bf16 unconditionally on CUDA) fails fast without eviction.
The Windows ROCm torchao import stub satisfies find_spec and even lets
from torchao.quantization import quantize_ succeed, but its quantize_ is a
no-op: auto would pick int8, leave the transformer dense, and disable
compile as if it were quantized. has_functional_torchao imports the exact
symbols the int8 path uses and rejects the stub via its sentinel; both the
auto picker and the /info advertised modes now use it
The SDXL trainer now regionally compiles its transformer blocks, so /info
advertises supports_compile for every family; the precision selector stays
DiT-only.
- base_precision="auto" only picks int8 when torchao is importable (the int8
quantize has no runtime fallback, unlike fp8); otherwise the middle band falls
back to nf4. Threaded as a parameter so the policy stays pure.
- The dense-mode validation (prequant base / bf16 compute) now applies only to
DiT families: sdxl ignores base_precision entirely, so a leftover value can no
longer fail an SDXL run. The mode-name validity check still runs everywhere.
New base_precision config for the DiT trainers: nf4 (unchanged default) |
bf16 | int8 | fp8 | auto, advertised per family + per machine through
/api/train/diffusion/info (precision_modes, recommended_precision,
supports_compile) so the UI can gate the selector.
- bf16: dense transformer + regional torch.compile (auto-armed). The
measured speed mode: 2.3x nf4 on FLUX (1.81 -> 4.12 steps/s), 2.6x on
Z-Image (2.5 -> 6.38 steps/s) on B200, at dense-weight VRAM
(FLUX 24.7 GB / Z-Image 13.6 GB peak vs 10.4 / 4.7 for nf4).
- int8: torchao weight-only int8 on the frozen base, quantized AFTER
add_adapter (quantizing first trips peft 0.18's TorchaoLoraLinear,
which is incompatible with the torchao 0.16 config API). Runs eager:
inductor rejects the int8 subclass training graph (aliased subclass
outputs), so compile is force-disabled for it.
- fp8: torchao convert_to_float8_training on the frozen linears
(filter skips lora_ modules, proj_out, non-divisible-by-16 dims,
pad_inner_dim), applied after add_adapter, compile auto-armed.
Works and round-trips, but measured SLOWER than compiled bf16 at
LoRA-training shapes (FLUX 3.15 vs 4.12 steps/s; Z-Image similar),
so it is an explicit opt-in and auto never picks it.
- auto: free VRAM (measured before load) + dense-size table -> bf16
when it fits with headroom, int8 in the middle band, else nf4.
Prequant bnb repos always resolve to nf4; dense modes on them are
rejected at validation with a pointer to the family's dense base.
Two crashes found and fixed along the way:
- The cuDNN SDPA backend's training graph fails on the FLUX attention
shapes (torch 2.10 + cu130, B200): mha_graph.execute errors, then the
context degrades into illegal memory accesses. The perf-flag guard now
pins flash/mem-efficient SDPA for the run (mathematically equivalent,
snapshot/restored). nf4 escaped it by routing attention differently.
- Regional compile now uses dynamic=True (the inference layer's proven
default): dynamic=False specialisation fused a gemm_and_bias epilogue
that failed with CUBLAS_STATUS_EXECUTION_FAILED on the FLUX training
graph; dynamic=True is also faster (Z-Image 3.84 -> 6.38 steps/s).
Verified: 98 backend tests green (new test_diffusion_base_precision.py:
validation, auto policy table, fp8 filter, compile gating, /info fields);
per-mode 40-step runs on FLUX + Z-Image with loss means inside the nf4
envelope and adapter round-trip generation through the normal LoRA path
for bf16-, fp8-, and int8-trained adapters.
