Collapse the remaining multi-line comment blocks in the attention, cache, LoRA, prequant, precision and compile-cache modules, the sd.cpp arg builder and engine, the video routes, the Ideogram 4 assembly, the model picker, and the diffusion test suites. Comments only, no code or behaviour changes.
Comment-only pass over the Python this PR touches: drop what the code already
says, collapse multi-line explanations that still read on one line, and keep
the reasoning that is not recoverable from the code. No code, docstring
semantics or behaviour changes; verified with an AST comparison against the
previous revision, and the backend suite is unchanged (same 37 environment
failures as before: the API integration tests that need a live keyed server,
the flash-attn install hooks, and the GPU memory fields).
- The FP8 Qwen3-VL text encoder was constructed at the process fp32 default before the
dequantized bf16 weights are copied in. That ~8B-param fp32 scaffold peaks ~2x on host RAM
(loading FIRST, before the DiTs), so a 64 GB host can OOM. Build it at the target dtype under
set_default_dtype, mirroring the DiT loader; rotary inv_freq is still computed in explicit fp32.
- The auto-policy memory table listed the text encoder at 8.8 GB, its FP8 on-disk size, while the
DiTs were doubled to their bf16-resident sizes. The loader dequantizes the encoder to bf16 too
(~16.3 GB), so the entry understated the resident footprint by ~7.5 GB and could let the planner
pick a resident placement that OOMs. Size it as bf16-resident.
- Speed (regional compile, QKV fuse) and the attention backend only touched pipe.transformer, so
ideogram-4's second denoiser (unconditional_transformer, run every step for dual-branch CFG)
stayed eager/native while status reported the optimization as engaged. Iterate every denoiser DiT
(mirroring the offload path) so both experts are optimized. Guarded on attr presence, so single-DiT
families are unchanged.
The fp8 loader builds Ideogram4Transformer2DModel via from_config, which
materializes the full ~9B-parameter module at the process default dtype (fp32)
before the dequantized bf16 weights are copied in and cast at the end. That
fp32 scaffold is ~2x the bf16 model (~37 GB vs ~18 GB) on host RAM, and the
second (unconditional) DiT builds while the first DiT and the text encoder are
already resident, so it can OOM smaller hosts. Wrap from_config in
set_default_dtype(dtype) so the module is built at the target dtype directly.
rotary_emb.inv_freq (the only __init__ state absent from the checkpoint) is
computed in explicit fp32, so a bf16 default leaves it correct.
Ideogram 4 assembles two DiTs per-component (a conditional transformer plus a
separate unconditional_transformer), so there is no transformer-only single-file
or GGUF artifact that could supply both. Add a pipeline_only family flag and
reject the gguf/single_file kinds in validate_load_request, before a load evicts
the current model, instead of assembling a pipeline missing its second DiT.
Extend the fp8 bf16-resident size override to a LOCAL directory mirror of the
ideogram-4-fp8 base: such a path never string-matches base_repo, so detect the
fp8 layout from the transformer shard headers (a *.weight_scale marker) and
reserve the bf16 footprint, matching the remote-base behaviour. A local nf4
mirror has no fp8 scales and correctly stays planned against its compressed bytes.
Review follow ups on the more-families branch: the per channel scale now
broadcasts rank aware instead of assuming 2D (all shipped tensors are 2D,
verified across all three fp8 components, but a future non 2D quantized
tensor would have mis broadcast silently), the fused qkv split asserts the
expected 3x hidden row count so a GQA style export fails loudly, fp8
detection scans every shard header rather than the first, and the excluded
model match uses the segment aware token helper with a hunyuanimage-3
token so a future HunyuanImage 2.x is not blocked with a 3.0 reason.
The ideogram-ai/ideogram-4-fp8 repo stores its two DiTs and the Qwen3-VL text
encoder in a vendor float8 layout that diffusers 0.39.0 (and diffusers main)
cannot read, so a stock Ideogram4Pipeline.from_pretrained produced a pipeline
with randomly initialized attention weights left on the meta device: the load
then died at pipe.to(device) with "Cannot copy out of meta tensor", and any load
that got past that would have generated noise.
Two things broke:
- The DiT attention is stored FUSED as attention.qkv.weight ([3*hidden, hidden],
Q/K/V rows stacked) plus attention.o.weight, while the diffusers transformer has
split to_q/to_k/to_v/to_out.0. from_pretrained mapped neither name and left them
meta + random.
- Every quantized weight is float8_e4m3 with a per-output-channel weight_scale;
the real weight is fp8.float() * weight_scale[:, None]. diffusers dropped the
scales and loaded the raw fp8 values (range +-448) as the weights, so even the
weights that did map were wrong.
load_ideogram4_transformer now reads the shards, dequantizes every scaled weight,
splits the fused qkv into to_q/to_k/to_v and renames o to to_out.0, then loads the
result into a config-constructed model. It fails loudly if any key stays unmatched
so a partly random model can never ship. The dequantized fp8 projections match the
byte-identical -nf4 export (already in the diffusers split layout with a bnb
quantization_config) to cosine ~0.997, so the split order and scale axis are
confirmed. The conversion is gated on the fp8 marker (a *.weight_scale key) read
from the shard header only, so the -nf4 repos skip it and load through the stock
from_pretrained path without a wasteful full-shard read.
The fp8 text encoder needed the same float8 dequant (its keys already match the
transformers Qwen3-VL module, so no rename). load_ideogram4_text_encoder handles
the fp8 repo and delegates the bnb-4bit and dense repos to the shared krea shim.
One more incompatibility was in the diffusers pipeline itself: it calls
transformers create_causal_mask(inputs_embeds = ...) with no cache_position, but
on transformers 4.57.6 the parameter is spelled input_embeds and cache_position is
required. _patch_create_causal_mask installs a signature-aware wrapper that renames
the kwarg and supplies cache_position, and is self-disabling on a matching signature.
Adds unit tests for the fp8 dequant/split conversion and the causal-mask patch.
Verified live on a B200: ideogram-4-fp8 (both CFG paths), ideogram-4-nf4-diffusers,
and krea-2 with the retroanime LoRA all load and generate coherent images.
