torch.cuda.is_bf16_supported() defaults to counting pre-Ampere bf16 EMULATION as
supported, so on a T4/V100/RTX 20xx the DiT-training bf16 gates all passed even
though the trainer requires native Ampere-or-newer bf16: /diffusion/info advertised
the DiT precision modes, /diffusion/start's preflight let the run through and freed
resident GPU models, then the trainer child hit the real unsupported bf16 path. The
inference device resolver already fixed this (issue #6658) by gating NVIDIA on
capability major >= 8; the training path never got it. Add a shared
native_bf16_supported() helper (NVIDIA cap major >= 8; ROCm keeps the trustworthy
is_bf16_supported()) and use it in the three DiT bf16 sites -- train_precision_modes,
bf16_unsupported_reason, and the trainer guard -- so a pre-Ampere card is offered
nf4 only and never advertises/evicts-then-fails. Tests now exercise the emulation
case (is_bf16_supported True but capability < 8).
Several image/video/training preflights ran before the route acquires the GPU or
frees resident models, but let a doomed local pick through and only failed deep in
the background load, after the user's chat/Images/Video model was already evicted.
- Local base_repo / base_model: _is_trusted_diffusion_repo accepts any existing
local path, but the base loads via from_pretrained (needs model_index.json). A
local dir that is not a diffusers pipeline passed the trust gate, evicted the
resident model, then failed. Add a shared _assert_local_base_is_pipeline check
and call it in the image, video, and training preflights.
- Dataset images: discover_image_caption_pairs only checked filenames, so a
corrupt or zero-byte upload passed the start-route preflight, freed the GPU, then
crashed the spawned trainer in PIL. Add an opt-in verify_images decode probe
(cheap PIL header check) that the start route enables; the trainers leave it off
since they decode every image anyway.
- Local single-file safetensors: the On-Device scanner advertises a bare
.safetensors directory (no model_index.json) as a text-to-image model, but the
picker starts it as a pipeline with no filename, so every click 400s. Reinterpret
such a pick as a single_file load of the sole checkpoint (resolve_local_single_file)
so the advertised model is actually loadable.
Regression tests for each: local non-pipeline base (image/video/training), the
verify_images decode gate, and resolve_local_single_file.
The start route's precision preflight folded bf16/int8/fp8 into the CUDA
requirement but omitted mxfp8, so an mxfp8 request on a GPU-less host (or an
older CUDA GPU without Blackwell) passed the preflight, evicted resident image
and chat models, then raised only in the spawned trainer child. Mirror
_resolve_base_precision: require CUDA for mxfp8 and re-check the Blackwell
(sm100+) capability up front, so a doomed run is rejected before teardown.
The start-route preflight caught the bf16-GPU and int8-torchao requirements but not the dense
precisions' CUDA requirement: on a GPU-less host bf16_unsupported_reason exempts CPU-only, so a
bf16/fp8 (or int8-with-torchao) DiT request passed the preflight, evicted resident workloads, then
raised only in the trainer child. Add the dense-mode CUDA gate mirroring _resolve_base_precision so
the doomed run is rejected up front. Also pin bf16_unsupported_reason in the two positive-path
family-info tests so they are deterministic across GPU types (a non-bf16 CUDA box would otherwise
empty every DiT family's advertised modes).
The start route preflight only rejected non-bf16 GPUs; an explicit int8 request on
a host with a missing or stub torchao passed the preflight, evicted resident GPU
workloads, then died in the trainer child (its int8 base quantizer has no fallback).
Fold both gates into training_precision_preflight_error so int8-without-torchao fails
fast before eviction. Also empty the advertised DiT precision_modes (and surface the
reason in vram_note, drop compile) whenever the bf16 preflight would reject the family,
so /info never offers an nf4 DiT option the route always 400s.
- normalized() + family_train_infos() mirror the inference fp8 deny for
Qwen-Image (activation outliers exceed fp8's range and corrupt the trained
result); int8 stays allowed and the UI no longer advertises fp8 for it.
- _resolve_base_precision() gates an explicit int8 on a FUNCTIONAL torchao, the
same gate auto and /info already apply, so a missing/stub torchao fails fast
instead of silently loading dense with compile disabled.
- train_precision_modes() gates the dense modes (bf16/int8/fp8/auto) on
torch.cuda.is_bf16_supported(), so a non-bf16 CUDA GPU (T4/V100/RTX 20xx) is
offered only nf4 instead of a start that evicts resident models and then fails.
- start_diffusion_training preflights bf16 support for the DiT families BEFORE
_free_gpu_for_diffusion_training(), so any DiT start (nf4 included, since the
trainer requires bf16 unconditionally on CUDA) fails fast without eviction.
The Windows ROCm torchao import stub satisfies find_spec and even lets
from torchao.quantization import quantize_ succeed, but its quantize_ is a
no-op: auto would pick int8, leave the transformer dense, and disable
compile as if it were quantized. has_functional_torchao imports the exact
symbols the int8 path uses and rejects the stub via its sentinel; both the
auto picker and the /info advertised modes now use it
The SDXL trainer now regionally compiles its transformer blocks, so /info
advertises supports_compile for every family; the precision selector stays
DiT-only.
- base_precision="auto" only picks int8 when torchao is importable (the int8
quantize has no runtime fallback, unlike fp8); otherwise the middle band falls
back to nf4. Threaded as a parameter so the policy stays pure.
- The dense-mode validation (prequant base / bf16 compute) now applies only to
DiT families: sdxl ignores base_precision entirely, so a leftover value can no
longer fail an SDXL run. The mode-name validity check still runs everywhere.
New base_precision config for the DiT trainers: nf4 (unchanged default) |
bf16 | int8 | fp8 | auto, advertised per family + per machine through
/api/train/diffusion/info (precision_modes, recommended_precision,
supports_compile) so the UI can gate the selector.
- bf16: dense transformer + regional torch.compile (auto-armed). The
measured speed mode: 2.3x nf4 on FLUX (1.81 -> 4.12 steps/s), 2.6x on
Z-Image (2.5 -> 6.38 steps/s) on B200, at dense-weight VRAM
(FLUX 24.7 GB / Z-Image 13.6 GB peak vs 10.4 / 4.7 for nf4).
- int8: torchao weight-only int8 on the frozen base, quantized AFTER
add_adapter (quantizing first trips peft 0.18's TorchaoLoraLinear,
which is incompatible with the torchao 0.16 config API). Runs eager:
inductor rejects the int8 subclass training graph (aliased subclass
outputs), so compile is force-disabled for it.
- fp8: torchao convert_to_float8_training on the frozen linears
(filter skips lora_ modules, proj_out, non-divisible-by-16 dims,
pad_inner_dim), applied after add_adapter, compile auto-armed.
Works and round-trips, but measured SLOWER than compiled bf16 at
LoRA-training shapes (FLUX 3.15 vs 4.12 steps/s; Z-Image similar),
so it is an explicit opt-in and auto never picks it.
- auto: free VRAM (measured before load) + dense-size table -> bf16
when it fits with headroom, int8 in the middle band, else nf4.
Prequant bnb repos always resolve to nf4; dense modes on them are
rejected at validation with a pointer to the family's dense base.
Two crashes found and fixed along the way:
- The cuDNN SDPA backend's training graph fails on the FLUX attention
shapes (torch 2.10 + cu130, B200): mha_graph.execute errors, then the
context degrades into illegal memory accesses. The perf-flag guard now
pins flash/mem-efficient SDPA for the run (mathematically equivalent,
snapshot/restored). nf4 escaped it by routing attention differently.
- Regional compile now uses dynamic=True (the inference layer's proven
default): dynamic=False specialisation fused a gemm_and_bias epilogue
that failed with CUBLAS_STATUS_EXECUTION_FAILED on the FLUX training
graph; dynamic=True is also faster (Z-Image 3.84 -> 6.38 steps/s).
Verified: 98 backend tests green (new test_diffusion_base_precision.py:
validation, auto policy table, fp8 filter, compile gating, /info fields);
per-mode 40-step runs on FLUX + Z-Image with loss means inside the nf4
envelope and adapter round-trip generation through the normal LoRA path
for bf16-, fp8-, and int8-trained adapters.
