For a narrow (fp8) Ideogram-4 base the planner reserves the family's known bf16
component total, but the reservation was gated on model_dense_mib being non-None.
On a first-time load an empty blob cache (or a best-effort download probe that
swallowed a transient HF error) leaves model_dense_mib None, so the guard skipped
the reservation exactly when it was needed: the planner then read 'size unknown ->
stay resident' and the ~54 GB pipeline OOMed a card that offload would have fit.
family_bf16_components_gb is a network-free constant, so reserve it whenever the
cache signal is absent (use it directly when None, else take the max).
- The FP8 Qwen3-VL text encoder was constructed at the process fp32 default before the
dequantized bf16 weights are copied in. That ~8B-param fp32 scaffold peaks ~2x on host RAM
(loading FIRST, before the DiTs), so a 64 GB host can OOM. Build it at the target dtype under
set_default_dtype, mirroring the DiT loader; rotary inv_freq is still computed in explicit fp32.
- The auto-policy memory table listed the text encoder at 8.8 GB, its FP8 on-disk size, while the
DiTs were doubled to their bf16-resident sizes. The loader dequantizes the encoder to bf16 too
(~16.3 GB), so the entry understated the resident footprint by ~7.5 GB and could let the planner
pick a resident placement that OOMs. Size it as bf16-resident.
- Speed (regional compile, QKV fuse) and the attention backend only touched pipe.transformer, so
ideogram-4's second denoiser (unconditional_transformer, run every step for dual-branch CFG)
stayed eager/native while status reported the optimization as engaged. Iterate every denoiser DiT
(mirroring the offload path) so both experts are optimized. Guarded on attr presence, so single-DiT
families are unchanged.
The fp8 loader builds Ideogram4Transformer2DModel via from_config, which
materializes the full ~9B-parameter module at the process default dtype (fp32)
before the dequantized bf16 weights are copied in and cast at the end. That
fp32 scaffold is ~2x the bf16 model (~37 GB vs ~18 GB) on host RAM, and the
second (unconditional) DiT builds while the first DiT and the text encoder are
already resident, so it can OOM smaller hosts. Wrap from_config in
set_default_dtype(dtype) so the module is built at the target dtype directly.
rotary_emb.inv_freq (the only __init__ state absent from the checkpoint) is
computed in explicit fp32, so a bf16 default leaves it correct.
Ideogram 4 assembles two DiTs per-component (a conditional transformer plus a
separate unconditional_transformer), so there is no transformer-only single-file
or GGUF artifact that could supply both. Add a pipeline_only family flag and
reject the gguf/single_file kinds in validate_load_request, before a load evicts
the current model, instead of assembling a pipeline missing its second DiT.
Extend the fp8 bf16-resident size override to a LOCAL directory mirror of the
ideogram-4-fp8 base: such a path never string-matches base_repo, so detect the
fp8 layout from the transformer shard headers (a *.weight_scale marker) and
reserve the bf16 footprint, matching the remote-base behaviour. A local nf4
mirror has no fp8 scales and correctly stays planned against its compressed bytes.
A successful on-demand wheel install writes into site-packages after the
import system already cached that directory's listing, so the very next
find_spec / import in the same process can miss the freshly installed
package when the install lands within the directory mtime's resolution.
That silently falls set_attention_backend back to native on first use.
Call importlib.invalidate_caches() after a successful install so the
kernel is picked up immediately, and cover both the success and failure
paths in the attention installer tests.
Review follow-ups on the image-generation PR:
- ControlNet: resolve_controlnet accepts a bare owner/name repo without the
non-GGUF base trust gate, and _controlnet_pipe hands it straight to
from_pretrained. A malicious pickle .bin would deserialize on load, so run
the same Hugging Face malware preflight (evaluate_file_security) the chat and
export loaders use before any remote ControlNet load; local dirs are exempt.
- Dataset thumbnails: key the cache on the full filename instead of the stem so
sample.png and sample.jpg no longer collide on one .thumbs file (which could
serve or delete the wrong image); the delete cleanup globs the same key.
- Diffusion training start: mirror start_training's API-key guard so an API
client cannot start training (which frees VRAM by unloading chat) while an
inference request is streaming; it now returns 409 before any GPU is freed.
- Model picker: include the curated safetensors row keys in the recommended
roving key list so arrow-key navigation reaches those rows instead of hitting
the duplicate option-missing id.
Tests: ControlNet malware gate (remote blocked before from_pretrained, local
skipped), thumbnail same-stem cache separation, API-key diffusion-start 409
before GPU free. Full diffusion suites green.
The latent cache holds two fp32 posterior tensors per crop/flip variant per
image, pinned on CUDA hosts, so datasets with thousands of images can exhaust
host or pinned memory with no fallback. Estimate the cache size from the first
real encoded latent and fall back to per-step VAE encoding when it exceeds a
4 GiB budget. UNSLOTH_DIFFUSION_FORCE_LATENT_CACHE bypasses the gate; the
existing UNSLOTH_DIFFUSION_NO_LATENT_CACHE opt-out is unchanged.
Replace the with-replacement per-batch index draw in the SDXL and DiT LoRA
trainers with a shared PermutationBatchSampler that visits every image once per
cycle before repeating, so short runs cover the whole dataset. The sampler
reshuffles from the run's rng so the index stream stays seed-deterministic.
Guard the diffusion run detail route against a valid-JSON non-object record,
which previously raised TypeError and returned a 500; it now 404s like the list
path's shape check.
Add regression tests for both.
