Probe fp8_dynamic per conv dimensionality and apply bench levers before placement

The fp8_dynamic conv smoke probe only exercised Conv2d, so a torchao build
whose Conv3d kernel path is missing or broken would pass the probe for a
video VAE (HunyuanVideo-1.5), report it quantized, and crash at the first
decode. The probe now runs per (device, conv ndim) and an explicit request
must pass it for every conv dimensionality the target VAE contains; the
auto ladder gate inspects the VAE the same way when one is provided.

The video benchmark moved the fully dense pipeline to CUDA before applying
the configured quant/optimisation levers, the reverse of the production
loader (video.py quantizes before apply_memory_plan). Dense-oversized
configs could OOM where the shipped quantized path loads fine, and
load_peak_gb recorded the dense placement. The bench now builds on CPU,
applies the levers, then places on CUDA and captures the load peak.
This commit is contained in:
Daniel Han 2026-07-10 08:01:24 +00:00
commit 6f887afb2b
3 changed files with 125 additions and 34 deletions

View file

@ -298,7 +298,10 @@ def _build_pipe(repo: str, force_fp32_vae: bool):
if force_fp32_vae:
torch_dtype = {"vae": torch.float32, "default": torch.bfloat16}
pipe = diffusers.DiffusionPipeline.from_pretrained(repo, torch_dtype = torch_dtype)
pipe = pipe.to("cuda")
# Stays on CPU: the caller applies the configured levers FIRST and only then places on
# CUDA, mirroring the loader (video.py quantizes before apply_memory_plan). Placing the
# dense pipeline first would OOM configs whose quantized form fits but whose dense form
# does not, and would record a dense load peak for a quantized row.
if force_fp32_vae and getattr(pipe, "vae", None) is not None:
pipe.vae.to(torch.float32) # belt-and-suspenders; a no-op on the primary path above
return pipe
@ -613,8 +616,10 @@ def _run_config(
torch._dynamo.reset()
except Exception:
pass
# Loader order (video.py): build on CPU, apply quant/optim levers, THEN place on CUDA.
# Placing dense first would OOM configs whose quantized form fits but dense does not, and
# load_peak_gb would record the dense placement instead of the measured config.
pipe = _build_pipe(repo, force_fp32)
load_peak = _peak_gb()
engaged = _apply_levers(
pipe,
cfg,
@ -624,6 +629,9 @@ def _run_config(
default_steps = default_steps,
logger = logger,
)
pipe = pipe.to("cuda")
_sync()
load_peak = _peak_gb()
_empty()
weights_gb = _alloc_gb()
dit_active = engaged["dit"] is not None