Studio diffusion (Phase 8): opt-in fast transformer (torchao int8/fp8/fp4 on a dense source)
Add an opt-in transformer_quant mode that loads the dense bf16 transformer and torchao-quantises it onto the low-precision tensor cores, instead of the GGUF transformer (which dequantises to bf16 per matmul and so runs at bf16 rate). On a B200 (Z-Image-Turbo, 1024px/8 steps): auto picks fp8 at 0.614s vs GGUF+compile's 0.823s (1.34x), int8 0.626s (1.32x), both at lower LPIPS than GGUF's own 4-bit floor. GGUF+compile stays the low-memory default and the fallback. The mode is gated on CUDA + bf16 + resident VRAM headroom (the dense load peaks ~21GB vs GGUF's 13GB); any unsupported arch/scheme, OOM, or quant failure falls back to GGUF with a logged reason. auto picks the best scheme per GPU via a real quantise+matmul smoke probe (Blackwell nvfp4/fp8/mxfp8, Ada/Hopper fp8, Ampere int8); a min-features filter skips the tiny projections that crash int8's torch._int_mm. New module mirrors diffusion_precision.py; quant runs before compile before placement. 184 -> tests pass; new test_diffusion_transformer_quant.py plus backend/route coverage. scripts/diffusion_bench.py gains --transformer-quant; scripts/quant_probe.py is the standalone torchao lever probe.
This commit is contained in:
parent
ede94176f6
commit
983f2c14a0
9 changed files with 1040 additions and 28 deletions
|
|
@ -216,6 +216,7 @@ def _run(args: argparse.Namespace) -> dict[str, Any]:
|
|||
memory_mode = args.memory_mode,
|
||||
speed_mode = args.speed_mode,
|
||||
text_encoder_quant = args.text_encoder_quant,
|
||||
transformer_quant = args.transformer_quant,
|
||||
)
|
||||
_wait_for_load(backend)
|
||||
_cuda_sync()
|
||||
|
|
@ -297,6 +298,7 @@ def _run(args: argparse.Namespace) -> dict[str, Any]:
|
|||
"speed_mode": args.speed_mode,
|
||||
"cpu_offload": args.cpu_offload,
|
||||
"text_encoder_quant": args.text_encoder_quant,
|
||||
"transformer_quant": args.transformer_quant,
|
||||
},
|
||||
}
|
||||
|
||||
|
|
@ -456,6 +458,14 @@ def _build_parser() -> argparse.ArgumentParser:
|
|||
choices = ["fp8", "nvfp4"],
|
||||
help = "quantise the companion text encoder (fp8 or nvfp4)",
|
||||
)
|
||||
p.add_argument(
|
||||
"--transformer-quant",
|
||||
default = None,
|
||||
choices = ["auto", "int8", "fp8", "nvfp4", "mxfp8"],
|
||||
help = "opt-in fast transformer: load the DENSE bf16 transformer and torchao-"
|
||||
"quantise it onto the low-precision tensor cores (faster than GGUF, higher "
|
||||
"VRAM). auto picks per GPU; falls back to GGUF if unsupported / no VRAM",
|
||||
)
|
||||
p.add_argument(
|
||||
"--cpu-offload", action = "store_true", help = "legacy: force whole-module CPU offload"
|
||||
)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue