video/image: honor explicit Speed=off for companions + trim, probe explicit TE kernels, bench fidelity
Address the Codex review round on the video/quant work: - Companion auto-quant now honors an explicit Speed=off. Both loaders already pin the DiT dense under an explicit off (bit-exact reference), but the unset text-encoder / VAE quant still promoted to auto and silently fp8/int8'd the companions, breaking the bit-exact request. An UNSET speed still auto-quantises; an explicit companion scheme still forces it. - The HunyuanVideo joint-attention trim is a speed lever (it swaps to the fused SDPA kernel), so gate it on a non-off speed tier exactly like the adjacent attention-backend selection -- the off path keeps the stock dense-mask attention. - Explicit torchao text-encoder modes (int8 / fp8_dynamic / nvfp4) now run the same kernel smoke test the auto ladder uses. They could clear the capability gate yet fail the real GEMM on a build where quantize_ wraps the encoder but the kernel is broken; the caster's try/except only covers the cast, not the first forward, so the load would report engaged then crash at generation. Now it falls back to dense. Layerwise fp8 has no torchao GEMM, so the probe is a no-op for it. - The trim pre-hook's fallback restores the caller's original kwargs (it may have emptied the image stream / trimmed a text stream before failing), so the stock dense-mask path runs on exactly what it expects, matching the empty-prompt guard. - video_speedmem_bench mirrors the loader: installs the Hunyuan trim before the backend set (gated on an active tier) and skips the auto int8 quant when it is the fp8-denied memory fallback and dense fits resident, so the shipped/auto rows measure what the loader actually runs. Tests: TE explicit-mode kernel probe (+ layerwise-fp8 bypass), trim mid-trim restore, and loader-level speed=off companion suppression + trim skip for both backends. 262 backend tests pass; ruff clean.
This commit is contained in:
parent
5eb64bdcac
commit
be04ba00f4
9 changed files with 201 additions and 15 deletions
|
|
@ -342,9 +342,16 @@ def _apply_levers(
|
|||
via _SecondExpertView, exactly like the loader, so A14B latency + accuracy are real."""
|
||||
from core.inference.diffusion_precision import quantize_text_encoders
|
||||
from core.inference.diffusion_vae_quant import quantize_vae
|
||||
from core.inference.diffusion_transformer_quant import quantize_transformer
|
||||
from core.inference.diffusion_transformer_quant import (
|
||||
quantize_transformer,
|
||||
is_int8_memory_fallback,
|
||||
)
|
||||
from core.inference.diffusion_speed import apply_speed_optims, snapshot_backend_flags
|
||||
from core.inference.diffusion_attention import select_attention_backend, apply_attention_backend
|
||||
from core.inference.diffusion_attention import (
|
||||
select_attention_backend,
|
||||
apply_attention_backend,
|
||||
install_hunyuan_attention_trim,
|
||||
)
|
||||
from core.inference.diffusion_cache import (
|
||||
apply_step_cache,
|
||||
TC_FBCACHE,
|
||||
|
|
@ -367,8 +374,15 @@ def _apply_levers(
|
|||
if getattr(pipe, "transformer_2", None) is not None:
|
||||
views.append(_SecondExpertView(pipe))
|
||||
|
||||
# DiT quant (pipeline kind, resident): mutates each expert's transformer in place.
|
||||
if cfg["dit"] not in ("none", "off"):
|
||||
# DiT quant (pipeline kind, resident): mutates each expert's transformer in place. Mirror the
|
||||
# loader's dense-fit skip: for an AUTO request on an int8-fallback family (HunyuanVideo-1.5, where
|
||||
# fp8 is black-framed so auto lands on int8, a memory-only lever ~7% slower AND less accurate than
|
||||
# dense+compile), run the dense DiT instead when it fits resident -- and the benchmark always
|
||||
# loads resident (no offload). Explicit int8/fp8 configs are honored (the whole point of the
|
||||
# sweep). Without this the "shipped"/"ditquant" auto rows would measure int8 where the loader runs
|
||||
# dense, overstating the shipped cost on Hunyuan.
|
||||
dense_fit_skip = cfg["dit"] == "auto" and is_int8_memory_fallback(tgt, fam_name)
|
||||
if cfg["dit"] not in ("none", "off") and not dense_fit_skip:
|
||||
schemes = [
|
||||
quantize_transformer(v, tgt, mode = cfg["dit"], family = fam_name, logger = logger)
|
||||
for v in views
|
||||
|
|
@ -423,6 +437,16 @@ def _apply_levers(
|
|||
)
|
||||
cache_active = engaged["cache"] not in (None, "off")
|
||||
|
||||
# HunyuanVideo-1.5 joint-attention trim (per expert), BEFORE the backend set so the requested
|
||||
# kernel pins onto the new processors -- exactly the loader's order. Drops the ~99% zero-padded
|
||||
# text tokens so the fused SDPA kernel runs (~18x/DiT-forward, cosine ~1.0). A speed lever, so
|
||||
# gated on an active tier like the loader; no-op for every non-Hunyuan family.
|
||||
trim_engaged = False
|
||||
if speed_active:
|
||||
for v in views:
|
||||
trim_engaged = install_hunyuan_attention_trim(v, fam_obj, logger = logger) or trim_engaged
|
||||
engaged["attn_trim"] = trim_engaged
|
||||
|
||||
# Attention (per expert).
|
||||
backend = select_attention_backend(tgt, cfg["attn"], speed_active = speed_active)
|
||||
for v in views:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue