Reject partial dual-DiT quantization in video_speedmem_bench (the loader fails
that load all-or-none; a mixed quantized/dense row is unloadable), toggle the
generation-time FBCache recheck on every expert view like the loader's per-view
iteration, and rescore lpips_vs_reference in a post-pass so a --configs order
that lists reference late no longer publishes null.
In quant_speedmem_bench, track per-encoder engagement via a weight-storage
fingerprint so a partial multi-encoder cast cannot certify a still-dense
encoder with a ~1.0 cosine, and load vae_force_fp32 families (Wan) at fp32
with a matching latent dtype so the dense VAE row measures what production
runs.
Gate the attention-trim tests with pytest.importorskip so a no-torch
environment keeps the backend test suite collectable.
Speed=off contract: the companion (text encoder / VAE) suppression under
an explicit Speed=off only matched an UNSET request, but auto is
backend-owned like transformer_quant, so an explicit
text_encoder_quant/vae_quant=auto would still engage fp8/int8 and break
the bit-exact request. Match 'auto' as well in both the image and video
loaders (a concrete scheme still forces quant). Covered by new
explicit-auto suppression tests.
Benchmark accuracy:
- quant_speedmem_bench teacc: when quantize_text_encoders returns None
(scheme skipped) the encoder is still dense, so scoring it against the
dense reference falsely certified a scheme that never ran. Record it
NOT engaged instead of collecting accuracy metrics.
- quant_speedmem_bench e2e: report the actual engaged te/vae scheme,
falling back to dense (not the requested auto) when the caster stayed
bf16, so a no-op default is not mislabelled as an auto-quantised run.
- video_speedmem_bench: HunyuanVideo ignores callback_on_step_end, so
step_ts stayed empty and per_step_ms was published as 0.0 for every
row. Time the denoise via a scheduler.step wrapper for that path.
video_speedmem_bench.py drives the real video-loader lever functions
(quantize_transformer / quantize_text_encoders / quantize_vae / apply_speed_optims
/ apply_attention_backend / apply_step_cache) with the loader's own defaults, so each
measured config reflects a real load. It decomposes the video speed/memory stack
(compile, cuDNN attention, First-Block-Cache, DiT/TE/VAE quant) with per-step latency,
peak resident GB, and per-frame LPIPS vs a bit-exact reference. This is the harness that
surfaced and validated the Wan fp8 black-frame fix.
quant_speedmem_bench.py gains the DiT-quant mode (dense vs fp8/int8/mxfp8 speed, peak
memory, and LPIPS vs the dense render) plus the shared LPIPS(AlexNet) helper.
A B200 speed/memory sweep (new scripts/quant_speedmem_bench.py) shows the VAE quant
win is a video story. Image AutoencoderKLs are ~0.15-0.26 GB, so fp8 saves ~0.1 GB
and only slows their tiny decode (+6-16%); the video Conv3d VAEs are ~2.5 GB and
halve to ~1.2 GB at ~2% decode cost. So VAE auto now only engages above a ~1 GB size
floor: small image VAEs stay dense (faster decode, no quant quality risk), video VAEs
still quantize. An explicit fp8 / fp8_dynamic request skips the gate (opted in).
The same sweep confirmed the text-encoder default is already right: fp8_dynamic is
E2E-neutral (denoise per-step unchanged; +2% one-time encode) and, by hidden-state
cosine vs bf16, marginally more accurate than layerwise fp8 -- so that default is left
as is. Tests cover the gate (small skipped, large quantized, explicit bypasses).
