The start-route preflight caught the bf16-GPU and int8-torchao requirements but not the dense
precisions' CUDA requirement: on a GPU-less host bf16_unsupported_reason exempts CPU-only, so a
bf16/fp8 (or int8-with-torchao) DiT request passed the preflight, evicted resident workloads, then
raised only in the trainer child. Add the dense-mode CUDA gate mirroring _resolve_base_precision so
the doomed run is rejected up front. Also pin bf16_unsupported_reason in the two positive-path
family-info tests so they are deterministic across GPU types (a non-bf16 CUDA box would otherwise
empty every DiT family's advertised modes).
The auto FBCache policy keyed on the full step count whenever strength was omitted, but the
loader only passes the strength kwarg when it is set, so an img2img/inpaint pipe then runs its
OWN signature default (< 1, e.g. FluxImg2ImgPipeline's 0.6). FBCache would engage on the full
28 steps while the pipe actually denoises ~16, degrading the image on exactly the short
trajectory the policy exists to keep uncached. Thread the pipe's signature default into the
policy via a new effective_request_strength helper (unit-tested), so the effective denoise count
matches what the pipe runs.
_dense_quant_prefetch_needed widened the prefetch to pull the base repo's transformer/
shards whenever a dense-quant candidate resolved, but balanced/low_vram (and the legacy
cpu_offload flag) force load_pipeline onto offload unconditionally in plan_diffusion_memory,
so its re-plan never flips to OFFLOAD_NONE and the dense build never runs. The offloaded GGUF
path then never touches those shards, so the widened prefetch only wastes a multi-GB download,
and a disk-full on that begin_load pull has no GGUF fallback (unlike the in-load_pipeline dense
failure). Mirror plan_diffusion_memory's definite-offload gates so the prefetch stays scoped.
Krea-2-Raw is in _TRUSTED_NON_GGUF_REPOS, so it is inference-loadable, but the generic
"krea" generation-defaults key matched it too and applied Turbo's distilled 8-step / no-CFG
recipe, producing degraded output on the undistilled base. Add a more specific krea-2-raw key
(52 steps, guidance 3.5 per the model card) ahead of the generic one, in both the backend
table and the frontend MODEL_DEFAULTS so the Studio UI and the OpenAI images route agree.
The start route preflight only rejected non-bf16 GPUs; an explicit int8 request on
a host with a missing or stub torchao passed the preflight, evicted resident GPU
workloads, then died in the trainer child (its int8 base quantizer has no fallback).
Fold both gates into training_precision_preflight_error so int8-without-torchao fails
fast before eviction. Also empty the advertised DiT precision_modes (and surface the
reason in vram_note, drop compile) whenever the bf16 preflight would reject the family,
so /info never offers an nf4 DiT option the route always 400s.
- _dense_quant_prefetch_needed widened the transformer/ prefetch to pull the base
repo's full dense bf16 shards even when a prequant checkpoint is configured
(candidate.prequant), contradicting its own docstring. That both defeats the
prequant download savings and can hard-fail begin_load on a disk-full (no GGUF
fallback there). Only widen for a real dense build (candidate is not None and
not candidate.prequant).
- DiffusionStatusResponse declared no 'resolved' field, so Pydantic's default
extra='ignore' silently dropped the per-control auto-policy provenance the
backend records (build_resolved_record / state.resolved) -- the plumbing never
reached any client. Declare the field so it round-trips.
For a narrow (fp8) Ideogram-4 base the planner reserves the family's known bf16
component total, but the reservation was gated on model_dense_mib being non-None.
On a first-time load an empty blob cache (or a best-effort download probe that
swallowed a transient HF error) leaves model_dense_mib None, so the guard skipped
the reservation exactly when it was needed: the planner then read 'size unknown ->
stay resident' and the ~54 GB pipeline OOMed a card that offload would have fit.
family_bf16_components_gb is a network-free constant, so reserve it whenever the
cache signal is absent (use it directly when None, else take the max).
DiffusionCharts deliberately renders only Training Loss + Gradient Norm (the LR
curve is the deterministic schedule the user picked), but the settings copy and two
comments still promised a live LR chart. Reword them so the UI no longer references a
chart that was intentionally dropped.
- DiffusionFamily gains deploy_base_repo (krea/Krea-2-Turbo): deploying a LoRA
trained on Raw now previews it on Turbo, not the non-distilled Raw checkpoint.
Scoped to a same-precision override so it never turns an nf4 train base into a
larger bf16 deploy load; exposed through family_train_infos -> the Train UI's
onDeployClick / historical-run deploy resolve the deploy base.
- _GENERATION_DEFAULTS gains a Krea entry (8 steps, 0 CFG) so the OpenAI
/v1/images/generations route matches the Create UI's documented distilled recipe
instead of falling through to the generic (9, 0.0).
- normalized() + family_train_infos() mirror the inference fp8 deny for
Qwen-Image (activation outliers exceed fp8's range and corrupt the trained
result); int8 stays allowed and the UI no longer advertises fp8 for it.
- _resolve_base_precision() gates an explicit int8 on a FUNCTIONAL torchao, the
same gate auto and /info already apply, so a missing/stub torchao fails fast
instead of silently loading dense with compile disabled.
- train_precision_modes() gates the dense modes (bf16/int8/fp8/auto) on
torch.cuda.is_bf16_supported(), so a non-bf16 CUDA GPU (T4/V100/RTX 20xx) is
offered only nf4 instead of a start that evicts resident models and then fails.
- start_diffusion_training preflights bf16 support for the DiT families BEFORE
_free_gpu_for_diffusion_training(), so any DiT start (nf4 included, since the
trainer requires bf16 unconditionally on CUDA) fails fast without eviction.
- _dense_quant_prefetch_needed widened the prefetch to pull the base repo's bf16 transformer/ shards
whenever a dense-quant scheme could resolve, with no disk check. On the offload path that can fill
the cache volume mid-download and hard-fail the load in a spot unload/cancel cannot preempt, instead
of the disk guard falling back to running the GGUF as-is (the Dtype hint's documented disk fallback).
Defer to resolve_dense_quant_candidate, the same disk-aware resolver load_pipeline re-plans against,
so the prefetch widens only when the dense build would really run.
- An explicit Speed=off (bit-exact) load with an unset dtype was promoted to auto-quant by the Dtype
default, silently engaging int8/fp8 + compile and breaking the bit-exact request (an auto DEFAULT
overriding an EXPLICIT control). Suppress the auto-dtype default when speed is explicitly off, in both
load_pipeline and the prefetch.
- The torchao 0.17 MX training path swaps a matched frozen Linear's weight for a wrapper tensor
whose linear override computes input @ weight_t and drops the bias, so mxfp8'ing a biased frozen
linear silently loses its bias and corrupts the base output the LoRA regresses against (verified
on Blackwell: the bias term is fully dropped). Skip biased linears in _mx_module_filter.
- _resolve_base_precision re-checked explicit dense modes against the live device but only rejected
CPU, so an explicit mxfp8 request on a non-Blackwell CUDA GPU passed and then crashed at the first
MX GEMM after a full dense-transformer load. /info only advertises mxfp8 on sm100+; mirror that
gate here and fail fast for a stale or direct client below Blackwell.
UNSLOTH_DISABLE_FP16_ACCUM is the documented safety escape hatch for fp16-accumulation
numerical drift, but it was matched as .strip() in (1, true, yes) with no lowercasing, so
UNSLOTH_DISABLE_FP16_ACCUM=TRUE (or YES / On) was silently ignored and fp16 accumulation
stayed on. Lowercase before matching (the family-name check on the next line already does)
and accept on. Existing 1/true/yes still match.
- The FP8 Qwen3-VL text encoder was constructed at the process fp32 default before the
dequantized bf16 weights are copied in. That ~8B-param fp32 scaffold peaks ~2x on host RAM
(loading FIRST, before the DiTs), so a 64 GB host can OOM. Build it at the target dtype under
set_default_dtype, mirroring the DiT loader; rotary inv_freq is still computed in explicit fp32.
- The auto-policy memory table listed the text encoder at 8.8 GB, its FP8 on-disk size, while the
DiTs were doubled to their bf16-resident sizes. The loader dequantizes the encoder to bf16 too
(~16.3 GB), so the entry understated the resident footprint by ~7.5 GB and could let the planner
pick a resident placement that OOMs. Size it as bf16-resident.
- Speed (regional compile, QKV fuse) and the attention backend only touched pipe.transformer, so
ideogram-4's second denoiser (unconditional_transformer, run every step for dual-branch CFG)
stayed eager/native while status reported the optimization as engaged. Iterate every denoiser DiT
(mirroring the offload path) so both experts are optimized. Guarded on attr presence, so single-DiT
families are unchanged.
effective_denoise_steps computed ceil(steps * strength) (steps - int(steps - steps*strength)),
but diffusers get_timesteps denoises init_timestep = min(int(num_inference_steps * strength),
num_inference_steps), i.e. the floored product. The two differ by one whenever the product is
fractional, and that flips the auto FBCache decision in the (19, 20) band: a strength-0.7
28-step img2img denoises int(19.6) = 19 real steps (below FBCACHE_MIN_STEPS = 20) but the old
formula returned 20 and engaged FBCache on that short trajectory, exactly the quality hit the
auto policy exists to avoid. Return min(int(steps * strength), steps) to match diffusers, and
fix the two tests that replayed the old formula.
Also honor _default_threads' documented fallback: (os.cpu_count() or 8) // 2 yields 4 when the
count is unknown, contradicting the docstring's 'falls back to 8'. Return 8 in that case.
The dense-quant re-plan passes transformer_resident_override_mib (the bf16 build
peak) AND computes companions via _companion_cache_bytes(base), which sums every
flat blob in the HF cache. Because the dense path prefetches the base transformer/
shards into that same cache before load_pipeline runs, the transformer is counted
twice, inflating the footprint (~44 GB instead of ~20 GB in the reproduction) and
wrongly forcing offload for models that fit resident -- the case this path exists
to enable. Add companion_override_mib and pass the auto-policy's own text-encoder
plus VAE estimate on the re-plan so the cache (with its prefetched transformer) is
not read for this artifact.
- Step cache 'Off' is preserved: the frontend defaulted to 'off' and mapped it
to an omitted transformer_cache, which the backend now reads as 'auto', so
leaving the control at Off silently enabled FBCache on 20+ step families.
Default the control to Auto, add an explicit Auto option, and send
auto -> omitted so Off maps to an explicit cache-off.
- Kernel auto-install adds --no-deps: 'pip install --only-binary :all: xformers'
resolves xformers' pinned torch and replaces the running torch/triton. --no-deps
installs only the best-effort kernel wheel; an ABI mismatch just fails to import
and falls back to native, never clobbering core deps.
- Do not retry a failed kernel install under the load lock: the pre-install runs
outside the locks, then the in-lock resolve re-attempts pip (up to 600s) while
holding _generate_lock/_lock and blocking unload/cancel/new loads. Record the
attempt in a process-level set so the in-lock call short-circuits to native.
- Cache auto-toggle keys on effective denoise steps: an image-conditioned run with
strength < 1 (upscale default 0.35) denoises a fraction of the requested steps,
so a 28-step request runs ~10 steps. Compute the effective count the way diffusers
get_timesteps does and gate FBCache on it, only when strength is actually applied.
The fp8 loader builds Ideogram4Transformer2DModel via from_config, which
materializes the full ~9B-parameter module at the process default dtype (fp32)
before the dequantized bf16 weights are copied in and cast at the end. That
fp32 scaffold is ~2x the bf16 model (~37 GB vs ~18 GB) on host RAM, and the
second (unconditional) DiT builds while the first DiT and the text encoder are
already resident, so it can OOM smaller hosts. Wrap from_config in
set_default_dtype(dtype) so the module is built at the target dtype directly.
rotary_emb.inv_freq (the only __init__ state absent from the checkpoint) is
computed in explicit fp32, so a bf16 default leaves it correct.
Ideogram 4 assembles two DiTs per-component (a conditional transformer plus a
separate unconditional_transformer), so there is no transformer-only single-file
or GGUF artifact that could supply both. Add a pipeline_only family flag and
reject the gguf/single_file kinds in validate_load_request, before a load evicts
the current model, instead of assembling a pipeline missing its second DiT.
Extend the fp8 bf16-resident size override to a LOCAL directory mirror of the
ideogram-4-fp8 base: such a path never string-matches base_repo, so detect the
fp8 layout from the transformer shard headers (a *.weight_scale marker) and
reserve the bf16 footprint, matching the remote-base behaviour. A local nf4
mirror has no fp8 scales and correctly stays planned against its compressed bytes.
A successful on-demand wheel install writes into site-packages after the
import system already cached that directory's listing, so the very next
find_spec / import in the same process can miss the freshly installed
package when the install lands within the directory mtime's resolution.
That silently falls set_attention_backend back to native on first use.
Call importlib.invalidate_caches() after a successful install so the
kernel is picked up immediately, and cover both the success and failure
paths in the attention installer tests.
