Studio diffusion (Phase 8): opt-in fast transformer (torchao int8/fp8/fp4 on a dense source)

Add an opt-in transformer_quant mode that loads the dense bf16 transformer and
torchao-quantises it onto the low-precision tensor cores, instead of the GGUF
transformer (which dequantises to bf16 per matmul and so runs at bf16 rate). On a
B200 (Z-Image-Turbo, 1024px/8 steps): auto picks fp8 at 0.614s vs GGUF+compile's
0.823s (1.34x), int8 0.626s (1.32x), both at lower LPIPS than GGUF's own 4-bit floor.

GGUF+compile stays the low-memory default and the fallback. The mode is gated on
CUDA + bf16 + resident VRAM headroom (the dense load peaks ~21GB vs GGUF's 13GB);
any unsupported arch/scheme, OOM, or quant failure falls back to GGUF with a logged
reason. auto picks the best scheme per GPU via a real quantise+matmul smoke probe
(Blackwell nvfp4/fp8/mxfp8, Ada/Hopper fp8, Ampere int8); a min-features filter skips
the tiny projections that crash int8's torch._int_mm. New module mirrors
diffusion_precision.py; quant runs before compile before placement.

184 -> tests pass; new test_diffusion_transformer_quant.py plus backend/route
coverage. scripts/diffusion_bench.py gains --transformer-quant; scripts/quant_probe.py
is the standalone torchao lever probe.
This commit is contained in:
Daniel Han 2026-06-26 06:15:18 +00:00
commit 983f2c14a0
9 changed files with 1040 additions and 28 deletions

View file

@ -216,6 +216,7 @@ def _run(args: argparse.Namespace) -> dict[str, Any]:
memory_mode = args.memory_mode,
speed_mode = args.speed_mode,
text_encoder_quant = args.text_encoder_quant,
transformer_quant = args.transformer_quant,
)
_wait_for_load(backend)
_cuda_sync()
@ -297,6 +298,7 @@ def _run(args: argparse.Namespace) -> dict[str, Any]:
"speed_mode": args.speed_mode,
"cpu_offload": args.cpu_offload,
"text_encoder_quant": args.text_encoder_quant,
"transformer_quant": args.transformer_quant,
},
}
@ -456,6 +458,14 @@ def _build_parser() -> argparse.ArgumentParser:
choices = ["fp8", "nvfp4"],
help = "quantise the companion text encoder (fp8 or nvfp4)",
)
p.add_argument(
"--transformer-quant",
default = None,
choices = ["auto", "int8", "fp8", "nvfp4", "mxfp8"],
help = "opt-in fast transformer: load the DENSE bf16 transformer and torchao-"
"quantise it onto the low-precision tensor cores (faster than GGUF, higher "
"VRAM). auto picks per GPU; falls back to GGUF if unsupported / no VRAM",
)
p.add_argument(
"--cpu-offload", action = "store_true", help = "legacy: force whole-module CPU offload"
)