Review follow ups on the more-families branch: the per channel scale now
broadcasts rank aware instead of assuming 2D (all shipped tensors are 2D,
verified across all three fp8 components, but a future non 2D quantized
tensor would have mis broadcast silently), the fused qkv split asserts the
expected 3x hidden row count so a GQA style export fails loudly, fp8
detection scans every shard header rather than the first, and the excluded
model match uses the segment aware token helper with a hunyuanimage-3
token so a future HunyuanImage 2.x is not blocked with a 3.0 reason.
logs/ (a 1.3 MB ComfyUI object_info dump plus stale PID files), temp/ (PR body
and commit message scratch), and async_task_outputs/ (agent task transcripts)
are environment specific runtime artifacts that were committed by accident and
carry stale local state into every checkout. Remove them and gitignore the
directories so they cannot be re-added.
The generic Studio config dict path can deliver these flags as strings, and a
non-empty string like "false" is truthy, so an opt-out silently no-ops (the
latent cache still builds, TF32 stays on). Coerce them the same way
gradient_checkpointing already is.
The wheel-only pip install for an optional attention kernel ran inside
load_pipeline under _lock and _generate_lock, so a slow or hanging install
blocked unload and cancellation for up to the 600s timeout. Resolve and install
the kernel before taking the locks (only an explicit backend ever pulls a
package, and its resolution ignores the speed tier); the in-lock apply call is
then a fast no-op. Also decode and log pip's stderr on a failed install so the
fallback to native is diagnosable instead of showing only the exit code.
The raw speed_mode string was forwarded to _enable_fp16_accumulation, so a
case-variant like MAX failed the speed_mode != SPEED_MAX check and wrongly
disabled fp16 accumulation on float16 pipelines. Forward the normalized mode
and cover the case-insensitive path in the test.
The ideogram-ai/ideogram-4-fp8 repo stores its two DiTs and the Qwen3-VL text
encoder in a vendor float8 layout that diffusers 0.39.0 (and diffusers main)
cannot read, so a stock Ideogram4Pipeline.from_pretrained produced a pipeline
with randomly initialized attention weights left on the meta device: the load
then died at pipe.to(device) with "Cannot copy out of meta tensor", and any load
that got past that would have generated noise.
Two things broke:
- The DiT attention is stored FUSED as attention.qkv.weight ([3*hidden, hidden],
Q/K/V rows stacked) plus attention.o.weight, while the diffusers transformer has
split to_q/to_k/to_v/to_out.0. from_pretrained mapped neither name and left them
meta + random.
- Every quantized weight is float8_e4m3 with a per-output-channel weight_scale;
the real weight is fp8.float() * weight_scale[:, None]. diffusers dropped the
scales and loaded the raw fp8 values (range +-448) as the weights, so even the
weights that did map were wrong.
load_ideogram4_transformer now reads the shards, dequantizes every scaled weight,
splits the fused qkv into to_q/to_k/to_v and renames o to to_out.0, then loads the
result into a config-constructed model. It fails loudly if any key stays unmatched
so a partly random model can never ship. The dequantized fp8 projections match the
byte-identical -nf4 export (already in the diffusers split layout with a bnb
quantization_config) to cosine ~0.997, so the split order and scale axis are
confirmed. The conversion is gated on the fp8 marker (a *.weight_scale key) read
from the shard header only, so the -nf4 repos skip it and load through the stock
from_pretrained path without a wasteful full-shard read.
The fp8 text encoder needed the same float8 dequant (its keys already match the
transformers Qwen3-VL module, so no rename). load_ideogram4_text_encoder handles
the fp8 repo and delegates the bnb-4bit and dense repos to the shared krea shim.
One more incompatibility was in the diffusers pipeline itself: it calls
transformers create_causal_mask(inputs_embeds = ...) with no cache_position, but
on transformers 4.57.6 the parameter is spelled input_embeds and cache_position is
required. _patch_create_causal_mask installs a signature-aware wrapper that renames
the kwarg and supplies cache_position, and is self-disabling on a matching signature.
Adds unit tests for the fp8 dequant/split conversion and the causal-mask patch.
Verified live on a B200: ideogram-4-fp8 (both CFG paths), ideogram-4-nf4-diffusers,
and krea-2 with the retroanime LoRA all load and generate coherent images.
Measured on the fresh linux x64 prebuilt (z-image Q8_0, sd-cli, 192 CPU
threads, 512x512, 9 steps, steady state): sampling 56.1s vs 51.3s (about
9 percent faster), VAE decode unchanged, peak RSS identical. The sd.cpp
engine only serves the no-GPU tier, so the default profile now matches
max: --diffusion-fa plus --diffusion-conv-direct.
Ideogram 4 (diffusers 0.39 Ideogram4Pipeline) as a new image family. The vendor
publishes no bf16 checkpoint, so ideogram-ai/ideogram-4-fp8 (raw float8 DiTs,
upcast by from_pretrained) is the family base and ideogram-4-nf4-diffusers is
the bnb-4bit pipeline artifact (ideogram-4-nf4 is byte-identical and detects to
the same family). All three repos join the trusted non-GGUF allowlist and the
frontend safetensors catalog.
Family specifics handled:
- Dual-branch CFG runs through a SEPARATE unconditional_transformer, so the
auto-policy size table entry counts two ~9.3B DiTs (37.2 GB bf16), and the
pipeline-kind memory plan now takes max(cached bytes, family table) for the
family base repo: the fp8 repo's cached bytes undershoot the bf16-resident
footprint by ~2x, which would let auto planning pick a resident placement
that OOMs.
- The pipeline accepts EITHER guidance_scale OR a per-step guidance_schedule
(its default: the recommended 45x7.0 + 3x3.0 taper, valid only at 48 steps)
and raises when both are set. At the advertised defaults (48 steps, guidance
7) generate() drops the constant so the recommended taper engages; any other
request nulls the schedule so the constant broadcasts legally.
- Generation defaults per the model card: 48 steps, guidance 7 (both tables).
tencent/HunyuanImage-3.0 is deliberately excluded: it has no diffusers pipeline
(an 80B autoregressive MoE behind trust_remote_code). A structured exclusion
map now surfaces that reason verbatim from validate_load_request instead of
the generic unknown-family error.
The curated diffusion LoRA catalog gains the nine official krea/Krea-2-LoRA-*
style adapters (family-tagged krea-2, explicit weight filenames), so they show
up in the picker instead of requiring a typed repo id.
Tests: new test_diffusion_more_families.py (detection, trust, defaults, size
table, exclusion reason, curated catalog + family filter), two generate()
tests for the guidance_scale/guidance_schedule pairing, and the local-scan
LoRA test updated for a non-empty curated list. Backend suite + CI-sim
(block_diffusers/block_torchao) green; frontend builds.
An unset transformer_quant used to mean off (run the GGUF as-is), so the
hardware ladder only engaged when auto was explicitly chosen and the panel
showed Off as the default. Unset (or auto) now hands the decision to the
ladder: a dense-capable GPU gets at least int8, data-center silicon fp8,
falling back to the GGUF when the device, VRAM, family deny table or disk
cannot take it. An explicit none/off pins GGUF-as-is and is now
expressible in the API (previously only omission meant off, so pinned-off
and unset were indistinguishable); an explicit scheme pins that scheme.
The dense candidate also gains a free-disk gate: with auto as the default
the bf16 base download (up to ~40 GB) must never wedge a nearly-full
model-cache disk, so the candidate is dropped (GGUF build kept) when free
space cannot hold it plus a 10 GiB margin. Unprobeable disk passes.
Frontend: the Dtype select defaults to Auto (fastest for GPU), keeps Off
as an explicit choice, and sends none through instead of omitting it.
Suite: 622 diffusion tests green (default-load test rewritten to the new
contract, explicit-off short-circuit covered), CI-sim green.
resolve_dense_quant_candidate now passes fam.name to
select_transformer_quant_scheme so the policy's proposed scheme honors the
family deny table (qwen-image lands on int8 instead of proposing fp8 that
the execution path would refuse). Test stub updated for the new keyword.
