ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Studio diffusion (Phase 8): detect consumer vs data-center GPU for fp8 accumulate, with user override

Browse source 
Consumer/workstation GPUs (GDDR) halve fp8 FP32-accumulate throughput, so they want
fast (FP16) accumulate; data-center HBM parts (B200/H100/A100/L40) are not nerfed and
prefer the higher-precision FP32 accumulate. Add _is_consumer_gpu() (token-exact match
on the device name per NVIDIA's GPU list, so workstation A4000 != data-center A40;
GeForce/TITAN and unknown default to consumer) and gate the fp8 use_fast_accum on it.

Measured: fast accumulate is ~2x on consumer Blackwell and ~8% on B200 (0.608 vs 0.665s),
no overflow, quality below the quant noise floor. So the default leans to accuracy on
data-center; a new request field transformer_quant_fast_accum (null=auto, true/false=force)
lets the operator override per load (scripts/diffusion_bench.py --fp8-fast-accum auto|on|off).

187 diffusion tests pass (+ consumer detection, _resolve_fast_accum, and the override
threading).
This commit is contained in:
Daniel Han 2026-06-26 08:01:04 +00:00
commit 10db6a4777
7 changed files with 170 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

11
scripts/diffusion_bench.py
View file

@ -217,6 +217,9 @@ def _run(args: argparse.Namespace) -> dict[str, Any]:
speed_mode = args.speed_mode,
text_encoder_quant = args.text_encoder_quant,
transformer_quant = args.transformer_quant,
transformer_quant_fast_accum = {"auto": None, "on": True, "off": False}[
args.fp8_fast_accum
],
)
_wait_for_load(backend)
_cuda_sync()
@ -299,6 +302,7 @@ def _run(args: argparse.Namespace) -> dict[str, Any]:
"cpu_offload": args.cpu_offload,
"text_encoder_quant": args.text_encoder_quant,
"transformer_quant": args.transformer_quant,
"fp8_fast_accum": args.fp8_fast_accum,
},
}
@ -466,6 +470,13 @@ def _build_parser() -> argparse.ArgumentParser:
"quantise it onto the low-precision tensor cores (faster than GGUF, higher "
"VRAM). auto picks per GPU; falls back to GGUF if unsupported / no VRAM",
)
p.add_argument(
"--fp8-fast-accum",
default = "auto",
choices = ["auto", "on", "off"],
help = "fp8 accumulate: auto picks by GPU class (fast on consumer, precise on "
"data-center); on/off force it",
)
p.add_argument(
"--cpu-offload", action = "store_true", help = "legacy: force whole-module CPU offload"
)