The prewarm registered its cancel event in _active_generate_cancel, but a
begin_generate arriving mid-warmup overwrote that slot with its own event
and then queued its worker behind the full warmup on _generate_lock. From
that point unload and cancel_generate signalled the wrong event, so the
warmup could no longer be aborted and the first real request waited out
the 9-54s the prewarm exists to hide.
Track the prewarm's event in a dedicated _prewarm_cancel slot (cleared
identity-checked alongside _active_generate_cancel) and signal it from
begin_generate before registering the real job's event, and from direct
generate() calls that skip begin_generate. The warmup then aborts at its
next step boundary and the real job takes the lock, while unload/cancel
keep working against whichever run is actually active.
vLLM and SGLang finish every compilation at server startup (dummy batches
through each compiled shape) so no request ever pays a compile mid-serving.
The video backend's compiled tier instead paid a first-generation extra after
every restart: ~54 s cold and ~11.3 s even with a warm Mega-cache bundle (the
residual is dynamo tracing plus cudnn.benchmark autotune, which the bundle
cannot carry).
After a compiled DEFAULT-tier resident load commits, a daemon thread now runs
one tiny throwaway generation (192x128 snapped, 4k+1-lattice 9 frames, 2
steps) under the generate lock. The default tier compiles dynamic=True, so
the small trace serves every later resolution. Measured through the real
backend (HunyuanVideo-1.5-480p, B200, 480x288/17f/30 steps):
warm bundle: first-generation extra 11.3 s -> 2.1 s (9.6 s background warmup)
cold start: the full compile moves off the user's first request entirely
(14.5 s background; first generation extra 2.1 s), and the
warmup persists the Mega-cache bundle itself
steady state: unchanged (2.4-2.5 s per 30-step clip in every phase)
Exactly lossless by construction: the warmup only changes when compilation
work happens. It resets its step-cache residuals, the real generation seeds
its own generator, and no process-wide flag is touched.
The warmup registers itself as the active cancellable job, so unload, a new
load, or cancel_generate abort it at a step boundary (verified: unload 2 s
into a running prewarm returns in 6.7 s with the warmup cancelled). It yields
untouched when a real request arrived first and is token-scoped against
superseded loads. Gated per family (supports_compile_prewarm), skipped for
speed=max (static per-shape graphs a warmup shape cannot serve), offload
(every warmup forward would stream the DiT over PCIe), and CFG parallel (its
planner owns compile-sensitive runs); the UNSLOTH_DIFFUSION_COMPILE_PREWARM
kill switch disables it. The decision and reason land in the resolved record.
Tests: +4 hermetic (decision gates, engaged-load spawn with snapped tiny
shape, skip-without-compile, yield to generations / stale tokens); related
backend set 551 passed; ruff clean.
Every current release of the kernels package (0.13 through 0.16) builds its
dependency tables with huggingface_hub >= 1.0's strict-dataclass API, and with
an older hub the breakage is not contained to the requested backend: import
kernels raises at module scope, and diffusers imports kernels whenever it is
installed, so a single on-demand install (an explicit flash3/flash4 attention
request on a stack pinned to hub < 1.0) permanently breaks every later
diffusers pipeline import on the box until the package is removed. Reproduced
against hub 0.36.2 with kernels 0.13.0 and 0.16.0: the HunyuanVideo-1.5
pipeline import dies in hub's strict-dataclass validator both times.
_ensure_attention_backend_installed now checks the resident hub version before
installing kernels (_kernels_hub_compatible) and refuses on < 1.0, logging why
and falling back to the native default, which is the best-effort contract the
installer already promises for an uninstallable wheel. The refusal is a policy
decision, not a failed attempt, so it is not memoised and a later request on a
fixed environment can still install. An undeterminable hub version keeps the
previous permissive behaviour, and the gate applies only to the kernels
package: sage/flash-attn/xformers wheels do not import hub at module scope.
Tests: the refusal (nothing memoised), the hub >= 1.0 allow, the
package-scoping, and the version-parse fallback.
The pre-warmed torch.compile cache (diffusion_compile_cache.py: fingerprinted
bundles over torch.compiler.save/load_cache_artifacts plus a persistent per-key
TORCHINDUCTOR_CACHE_DIR) was wired into the image backend only, so every video
load re-paid the full first-generation compile after every process restart (the
stock inductor dir lives in /tmp).
video.py now mirrors the image backend exactly: compile_cache.begin runs after
the attention-backend set (the fingerprint keys on the engaged kernel) and
before apply_speed_optims on a compile-eligible default/max tier, keyed on the
same fullgraph decision as the compile itself (an engaged or still-toggleable
step cache and a planned offload both drop it); the context is committed to
_VideoLoadState and compile_cache.save persists the bundle after the first
successful generation (env-gated distributor / first-run-warm mode);
_teardown_state restores the inductor dir, and a token-scoped
_rollback_precommit_compile_cache covers loads that die before the state
commit, mirroring the globals and CFG-parallel rollbacks.
Measured on HunyuanVideo-1.5-480p through the real VideoBackend (B200,
480x288/17f/30 steps): the first-generation extra drops from 107.5 s cold to
13.8 s when the 12.8 MB bundle loads into a fresh inductor dir (0.10 s load)
and to 11.7 s from the persistent per-key dir alone; through the wired
production path a restart lands at 10.5-10.8 s (bundle-only included) vs
86.5 s cold. Steady state is unchanged (2.4-2.6 s), and the loaded artifacts
are the same bits a local compile would produce, so numerics are untouched.
Tests: begin/save/restore lifecycle with the fullgraph keying, the Speed=off
and compile-ineligible skips, and the token-scoped pre-commit rollback.
Speed=off is the reference contract: the loaders pin every auto speed
lever (transformer/TE/VAE quant tri-states) to off, but the cfg_parallel
auto path never consulted speed_active, so a resident two GPU
HunyuanVideo-1.5 load with Speed=off could still reserve a second GPU
and install the CFG-parallel proxy. Auto now returns off when
speed_active is false; an explicit cfg_parallel=on stays honored as a
deliberate override (the install-failure test now exercises exactly
that override path).
Mirror production's explicit-MagCache re-engagement in the video bench: a
magcache row installed at the family default step count now re-interpolates
its curve, retention window, and skip budget when --steps differs, instead of
timing a stale schedule users never run.
Report the generation-time cache state: the row's cache field is derived from
the post-toggle transformer marker rather than the load-time engagement, so an
auto cache toggled off below the step threshold (or a re-sized explicit
magcache) is published as it actually ran; the load-time value stays available
as cache_at_load.
Restore the process-wide backend flags (cudnn.benchmark, TF32 and fp16
accumulation, emulate_precision_casts) after each config, like the production
unload path, so a speed-enabled row cannot poison a later reference or off row
in the same --configs run.
Only settle the CFG-parallel dispatch key after a run that actually routed
the replica: a guidance-near-1 generation disables the overlap without
warming the replica, so its completed key must not unlock thread dispatch
for the next CFG-enabled run at the same shape (that first compile has to
stay serialized).
Restore the process-global thread-safe cuDNN attention patch when the
CFG-parallel install fails after the patch landed: no proxy is committed on
that path, so teardown would never reach it and later single-device
generations would keep running the direct aten replacement.
Tear down a CFG-parallel proxy installed by a load that is cancelled or
fails before the _VideoLoadState commit: the proxy owns a daemon worker,
the DiT replica's VRAM, and possibly the cuDNN patch. The load stashes the
proxy pre-commit and _run_load's error handler rolls it back, token-scoped
exactly like the speed-globals rollback.
Re-engage an EXPLICIT magcache choice when the actual step count differs
from the configured one, so the ratio curve, retention window, and skip
budget are re-interpolated over the real schedule (the on/off choice never
changes); auto already re-engaged via maybe_toggle_step_cache. The step
marker comparison uses endswith so #s5 cannot match inside #s50.
Bench fidelity: the e2e auto row quantizes companions before CUDA placement
(mirroring the loader, so load_peak_gb records the measured configuration),
the video bench clears step-cache residuals before every generation exactly
like VideoBackend.generate, and the image-interface dit/e2e modes reject
video families with a pointer to video_speedmem_bench.py.
Extends the HunyuanVideo-1.5 round-1/2 optimization levers to wan2.2-ti2v-5b,
wan2.2-t2v-a14b (dual-expert MoE) and ltx-2, shipping only what beats the
incumbent on the measured accuracy-speed frontier (B200, LPIPS(AlexNet)
pairwise vs the same uncached compiled stack at identical seed/settings).
- Wan2.2-TI2V-5B auto step cache switches FBCache to calibrated MagCache:
balanced (0.12, 3, 0.2) measures 1.65x at pairwise LPIPS 0.034 vs the
incumbent FBCache 0.08 at 1.49x/0.031, and 1.73x/0.044 vs 1.71x/0.083 at
the fast points (FBCache error grows unboundedly past its threshold while
MagCache's budget caps it). A 50-step calibrated curve ships; cond/uncond
branches agree within 0.0008 so one curve serves both CFG contexts.
- Per-expert MagCache plumbing for dual-expert MoEs: the experts split the
schedule at the boundary timestep (Wan2.2-A14B: 16 + 34 of 50) and the hook
counts each expert's own forwards from 0, so a shared full-schedule curve
would be misaligned for both. apply_step_cache / maybe_toggle_step_cache /
the loader now thread an expert name; a second expert resolves
family::transformer_2 curves and sub-curves scale their configured step
count by steps/50. Single-DiT behaviour unchanged.
- Wan2.2-A14B keeps FBCache: with per-expert curves, FBCache 0.12 at
2.88x/0.128 dominates balanced MagCache (1.80x/0.145) and FBCache 0.08 sits
at 1.28x/0.098; the 16-step high-noise expert starves MagCache's skip
budget. No calibrated curve ships, so an explicit magcache request runs
uncached with a warning instead of engaging a measured-worse mode.
- Wan2.2-A14B TE auto quant resolves dense: TE fp8_dynamic alone costs
pairwise LPIPS 0.1195 for a 1.03x once-per-generation encode (146.7 to
142.7 s e2e). Wan2.2-TI2V-5B shares the UMT5 encoder but stays quantized
(0.0396 pairwise at a real 1.09x on its much faster DiT).
- LTX-2 TE fp8_dynamic family-denied: torchao per-row compute fp8 on the
Gemma3-27B encoder black-frames the whole clip (mean luma 137.9 to 0.0,
LPIPS 0.78; reproduced compiled and eager), while layerwise fp8 is
near-lossless (pairwise 0.0043) at the same shrink, so auto falls through
to it and explicit fp8_dynamic requests are refused.
- LTX-2 step caching deliberately stays unregistered, now documented on
_EXTRA_BLOCK_METADATA: the block returns a joint (video, audio) stream pair
and both cache hook families would substitute text embeddings into the
audio slot on every skipped step; a dual-stream cache is required, and the
distilled checkpoints run below FBCACHE_MIN_STEPS anyway.
- Compile parity (emulate_precision_casts) verified family-neutral and kept
global: wan5b 1.75x/0.0029 on vs 1.54x/0.0082 off; ltx2 1.308x/0.0013 vs
1.307x/0.0025; a14b 2711 vs 2717 ms/step. Cache-hook compile arming
verified to generalize (wan5b fb@0.04 armed 1.216x vs raw 1.048x). Dual-GPU
CFG stays HunyuanVideo-1.5-only: LTX-2 runs batch-CFG in one forward and
the Wan pipelines consume each branch inline with no guider combine hook.
- video_speedmem_bench gains epc_off (compile-parity isolation) and
fbcache_explicit / magcache_explicit configs plus expert-aware cache
application mirroring the loader.
Measured via scripts/video_speedmem_bench.py and the round-3 single-load
probes; full data and per-family decision table in
outputs/video_families_optim_round3.md (workspace). Tests: 235 passing across
the five video inference suite files (9 new: per-expert curve resolution and
step scaling, uncalibrated-expert refusal, toggle expert threading, wan5b
magcache auto load/toggle, ltx2 deny auto+explicit, a14b TE auto-dense);
ruff clean.
Require the model_index.json to sit at the snapshot ROOT before flagging a
cached repo as pipeline-loadable: CachedFileInfo.file_name is the basename,
so the previous name match also claimed nested copies (subdir/model_index.json)
and the picker then sent a from_pretrained load that fails only after the GPU
handoff. Scope by file_path against the revision's snapshot_path.
Validate the maximum derived seed before the multi-run image loop: an explicit
seed near 2**53-1 plus the per-run offset (base + i*batchSize) exceeded the
backend cap and 422'd a later run after earlier images had already generated.
Route local single-file .safetensors picks on the Images and Video pages
through the single_file load path (parent dir + basename), matching the local
GGUF branch: the pipeline route rejects a bare file with no model_index.json,
and only after evicting the resident model.
Applies the video round-2 accuracy findings to the image diffusion stack and fixes
two real image-path bugs found while measuring. All numbers B200, production
settings (family default steps/guidance, 1024px, seed 42, 4 fixed prompts), LPIPS
(AlexNet) via the new scripts/image_speedmem_bench.py, which drives the production
lever functions in the loader's own order.
- inductor precision parity: emulate_precision_casts=True on the regional-compile
path (fused pointwise kernels keep fp32 intermediates where eager rounds to bf16
between ops). Pairwise LPIPS of the compiled tier vs the same-stack eager tier:
Qwen-Image 0.019 to 0.006 at identical speed (72.4 vs 72.5 ms/step), FLUX.1-dev
0.046 to 0.029 at +2% step time (69.8 vs 68.3, reproduced), FLUX.2-klein-4B
0.018 to 0.017 at identical speed. Snapshot/restored with the other process-wide
backend flags so an off load never inherits it.
- cache x compile composition: re-point each cache hook's fn_ref.original_forward
at a torch.compile'd wrapper of the same bound method (armed only where the
speed layer compiled the block; restored before every disable_cache and before
the partial-hook cleanup). Qwen-Image FBCache computed steps 91.8 to 71.2 ms
(back at the uncached compiled rate), 1.21x end to end (7.36 to 6.06 s per 4
images); FLUX.1-dev already traced through its FBCache hook and is measured
neutral (same-process armed vs unarmed latents bit-identical). Skip counts
within noise (13 vs 11 of 76; pairwise LPIPS 0.005).
- FBCache mid-session toggle crash: diffusers 0.39 caches the HookRegistry child
list on first cache_context use, so an uncached generation followed by a
20+-step generation (the auto toggle path) enabled hooks the context never
reached and crashed with "No context is set" (reproduced live on FLUX.1-dev).
Invalidate the stale child cache after every enable_cache.
- TE fp8_dynamic zero-row guard: torchao per-row fp8 derives a per-output-channel
scale from the row amax, so an all-zero weight row is 0/0 = NaN. SDXL's
text_encoder_2 (OpenCLIP bigG) ships exactly such a row, and every explicit
fp8_dynamic SDXL render came out black; keep zero-row Linears dense (LPIPS
0.976 black to 0.096 working). Other families' encoders have no such rows and
are byte-identical.
- No AUTO TE quant exists on the image branch (text_encoder_quant defaults dense,
explicit-only), so the video round's auto-dense retune has no image analogue;
the explicit lever's cost is now measured (TE fp8_dynamic alone, LPIPS vs
bit-exact: Qwen-Image 0.038, FLUX.1-dev 0.084, SDXL 0.096; no speed win, VRAM
-6.5 GB on Qwen-Image) for the docs.
Tests: 96 passing across the cache/speed/precision suites (11 new arming, 2
child-registry, 2 zero-row, 4 inductor-flag); ruff clean.
Cuts the shipped default's LPIPS vs the bit-exact reference from 0.224 to 0.139
while going faster (24.9 s to 21.2 s at 720p/33f/30 steps, 22.7x vs reference),
and makes the remaining speed/accuracy trade a user knob.
- inductor precision parity: set emulate_precision_casts=True for the regional
compile (fused pointwise kernels kept fp32 intermediates where eager rounds to
bf16 between ops); full-clip LPIPS vs bit-exact 0.221 to 0.052 at zero speed
cost. Snapshot/restored with the other process-wide backend flags.
- cache x compile composition fix: diffusers cache hooks are
torch.compiler.disable'd, so every COMPUTED step ran eager (1.69 vs 1.09
s/step) under MagCache/FBCache in both enable orders. Re-point each hook's
fn_ref.original_forward at a torch.compile'd wrapper of the same bound method
(armed only where the speed layer compiled the block; restored before every
disable_cache so the uncached path stays pristine). Balanced MagCache at 50
steps: 1.48x to 2.17x, identical skip counts, bit-identical uncached rerun
after enable/disable cycles.
- transformer_cache_quality knob (quality|balanced|fast; API + UI + bench)
mapping to (threshold, max_skip_steps, retention_ratio). Auto resolves to the
near-lossless quality preset (0.06, 2, 0.3; 1.63-1.64x at pairwise LPIPS
0.05-0.09) for the HunyuanVideo-1.5 families and to balanced (the pre-knob
values, byte-identical behaviour) everywhere else.
- TE auto-quant resolves dense for HunyuanVideo-1.5: TE fp8_dynamic alone moves
the clip to LPIPS 0.236 vs bit-exact for zero speed win (the quantised encoder
perturbs the conditioning and the trajectory amplifies it chaotically); VAE
fp8 stays in auto (0.053, at the compile floor). Explicit schemes honored.
- dual-GPU CFG branch parallelism (new diffusion_cfg_parallel.py): transformer
proxy + DiT replica on the most-free second CUDA device + worker thread,
branch-routed off the pipeline's own cache_context names. Auto engages only
where measured bit-identical (eager tier: max abs diff 0.0, 1.66x); the
compiled stack is explicit cfg_parallel=on (1.52x over the sequential
default; per-device compiled artifacts differ by 1 bf16 ulp/step, documented
in the resolved record). Fail-soft gates: family allowlist, guider CFG,
pipeline kind, dense DiT, no offload, free-VRAM check; single-GPU loads are
untouched and the memory plan stays single-device.
- video API: the transformer_cache literal now accepts auto/magcache (an
explicit magcache request was rejected at the pydantic layer); the mxfp8
family deny records the round-2 measurement (block-32 MX scaling fixes the
zero-row collapse, no black frames, but is latency-neutral at LPIPS 0.37:
fails both ship bars).
Measured on B200 via the production lever path (video_speedmem_bench.py, which
gained a --cache-quality lever and companion-quant isolation configs). Tests:
441 passing across the video inference suite (32 new for cfg-parallel, 20 for
presets/arming, 3 for the inductor flag, 2 for TE auto-dense); ruff clean.
HunyuanVideo-1.5 loads previously logged 'fbcache unavailable (Model class
HunyuanVideo15TransformerBlock not registered)': diffusers 0.39 ships FBCache
block metadata for HunyuanVideo 1.0 but not 1.5, although the 1.5 DiT is fully
cache-shaped (CacheMixin, homogeneous residual-additive dual-stream blocks,
cache_context per guidance branch). Register the missing metadata at engage
time (deferring to a native registration when a future diffusers ships one).
Measured on a B200 (720p t2v, 1280x720, 33 frames, seed 42), FBCache is fast
but not shippable for this family: 1.44x at 30 steps / 2.41x at 50 steps, at
LPIPS 0.43-0.54 vs the same uncached stack with a +5..8 luma drift (no skip
cap or error budget, so the trajectory derails into a different clip). MagCache
(same registry metadata, also dispatched via enable_cache) is bounded by
design and lands the win: 1.49x end-to-end at 50 steps at LPIPS 0.147 with the
same composition, 1.21x at LPIPS 0.071 on the 480p model. Ship magcache as the
per-family AUTO cache mode for hunyuanvideo-1.5 / -720p with 50-step
calibrated per-family mag_ratios (cond/uncond curves agree within 0.014,
30 vs 50-step calibration within 0.027 after interpolation); every other
family keeps fbcache, and explicit fbcache/magcache requests are honored.
The auto toggle re-engages magcache on a step-count change so the ratio curve
is re-interpolated over the actual schedule.
Two production bugs fixed along the way:
- diffusers' HookRegistry caches its child-registry list, so enabling a cache
AFTER any uncached generation (the auto off-to-on toggle) left the new block
hooks without a context ('No context is set' on the first cached forward).
Invalidate the stale cache after every enable_cache.
- An explicit int8 DiT request crashed under the padded-text trim: torchao's
int8 dynamic path returns a zero-token (M=0) input unprojected (t2v byt5 /
image streams -> cond-type add shape crash) and torch._int_mm requires
M > 16 (an empty negative prompt trims to ~6 tokens -> TokenRefiner crash).
Add per-family int8 excludes for the text-stream linears (context_embedder*,
image_embedder, add_q/k/v_proj, to_add_out, ff_context); they run at tens of
tokens vs the ~32k video stream, so the exclusion costs nothing measurable.
Bench: trim lever key (trim_off / eager_trim isolation configs), int8_cudnn +
shipped_nocache rows, --cache-threshold, warmup timing, per-config frame
persistence for offline LPIPS rescoring, and the loader's per-family auto
cache mode mirrored. Full 720p matrix recorded: reference 481.6s ->
trim+cudnn+compile 35.4s -> shipped default with TE/VAE quant + magcache
24.9s (19.4x, peak VRAM 89.4 -> 81.8 GB), int8 latency-neutral (dense auto
policy confirmed), compile 1.56x per step, trim 13.5x per step at production
shapes.
Validated end to end through the real VideoBackend: load resolves
transformer_cache=magcache with trim + compile + cudnn, generation
re-interpolates 50 -> 30 steps, auto-disengages below 20 steps, re-engages
after an uncached generation, unload restores globals. Hermetic tests cover
the registration, the child-cache invalidation, magcache engage/threshold/
no-curve/no-steps paths, auto-mode routing, toggle re-interpolation, and the
family int8 excludes.
Step 0 now reads "Preparing (text encoding + warmup)..." on the video and
images pages: text encoding and warmup run before the first scheduler tick,
so the bar otherwise sits on "step 0/N" for up to a minute at 720p.
Generation progress polls are also wired to a visibilitychange listener for
their lifetime: background tabs clamp setInterval to one second and can
suspend it entirely after a few minutes, so returning to the tab now fires
one immediate poll (overlap-guarded) instead of showing a stale label until
the next throttled tick. The listener is removed with the interval on the
terminal phase, generation end, and unmount.
POST /video/generate previously held the response open for the whole
generation (multi-minute for 720p), so in --secure mode the Cloudflare
quick tunnel's ~100s origin-response cap returned a 524 while the server
kept generating, and the frontend treated the run as failed.
Generation now follows the same return-at-once pattern as /video/load:
begin_generate validates synchronously (409 on no model or on a second
concurrent generate via a new busy sentinel) and runs the existing
generate + gallery-persist pipeline, with the route's exact error
mapping, on a daemon thread. GET /video/generate-progress gains optional
terminal fields: phase completed carries the saved gallery record, phase
failed a client-safe error; active only drops together with a terminal
phase. The cancel event is registered before the worker starts so
/video/generate/cancel keeps working across the whole job.
VideoGenerateResponse becomes an accepted acknowledgement (status
started, video kept as an always-null compat field). The video page
fires the POST, then drives completion off the progress poll it already
runs (completed prepends the clip, failed surfaces the error, the
cancelled sentinel stays toast-free). The API-key training-start guards
now also probe the video backend for an in-flight background clip, since
it is no longer visible as an in-flight HTTP request to the keep-warm
counter.
Route tests keep the fake backend for load/generate/status but inherit
the real job machinery, covering immediate accept, concurrent 409, the
terminal completed record, sanitized/ValueError/cancelled failures, and
cancel of a running job.
Tag cached diffusion repos that ship no model_index.json with single_file in the
cached-models listing, and keep them out of the task-scoped On Device pickers
unless the curated catalog carries their artifact: the selection fall-through
loads uncataloged rows as a full pipeline and from_pretrained fails on a
single-file checkpoint repo after the GPU handoff.
Refresh diffusion status after a successful generation run on the Images page.
Speed Auto compiles the transformer on the third LoRA-free generation and flips
supports_lora to false; without the refresh the LoRA picker stayed enabled and
the next LoRA generation failed on the backend.
Match upload passthrough exact paths with trailing slashes normalized: the
trailing-slash variant of /api/train/diffusion/dataset reaches MaxBodyMiddleware
before the router's redirect_slashes 307, so it fell through to the default
/api/train body cap and 413ed large uploads. JSON sub-routes keep extra path
components after normalization and stay on the small cap.
The fp8_dynamic conv smoke probe only exercised Conv2d, so a torchao build
whose Conv3d kernel path is missing or broken would pass the probe for a
video VAE (HunyuanVideo-1.5), report it quantized, and crash at the first
decode. The probe now runs per (device, conv ndim) and an explicit request
must pass it for every conv dimensionality the target VAE contains; the
auto ladder gate inspects the VAE the same way when one is provided.
The video benchmark moved the fully dense pipeline to CUDA before applying
the configured quant/optimisation levers, the reverse of the production
loader (video.py quantizes before apply_memory_plan). Dense-oversized
configs could OOM where the shipped quantized path loads fine, and
load_peak_gb recorded the dense placement. The bench now builds on CPU,
applies the levers, then places on CUDA and captures the load peak.
Reject partial dual-DiT quantization in video_speedmem_bench (the loader fails
that load all-or-none; a mixed quantized/dense row is unloadable), toggle the
generation-time FBCache recheck on every expert view like the loader's per-view
iteration, and rescore lpips_vs_reference in a post-pass so a --configs order
that lists reference late no longer publishes null.
In quant_speedmem_bench, track per-encoder engagement via a weight-storage
fingerprint so a partial multi-encoder cast cannot certify a still-dense
encoder with a ~1.0 cosine, and load vae_force_fp32 families (Wan) at fp32
with a matching latent dtype so the dense VAE row measures what production
runs.
Gate the attention-trim tests with pytest.importorskip so a no-torch
environment keeps the backend test suite collectable.
Run backend.start_training off the event loop with asyncio.to_thread so the
synchronous diffusion/video unload calls (which wait on engine generation
locks) cannot freeze concurrent requests; guard against overlapping starts
with a _start_in_progress compare-and-set under the service lock.
Resolve bare diffusion dataset names directly under datasets_root() before
falling back to the generic resolver, so an unrelated LLM upload file or
recipe folder sharing the name cannot shadow the image dataset.
Reject exact duplicate filenames within one multipart upload batch: two
parts staged to the same destination would let the later tmp.replace
silently discard the earlier file. Case variants stay exempt per the
existing stem-guard contract.
Require an instance prompt in the train panel when only some images have
captions, since backend discovery silently skips uncaptioned images.
Review round follow-ups:
- Drop the machine-specific HF_HOME defaults from the four bench /
reproduction scripts (fp8_layer_ablation, hunyuan_int8_profile,
quant_accuracy_sweep, video_speedmem_bench); they pointed at a private
workspace cache and broke the scripts on any other machine. The
standard HF_HOME env override still applies.
- Correct the vae_quant 'auto' descriptions (image + video request
fields, select_vae_quant_scheme docstring, loader comment) to match
the shipped ladder: auto engages layerwise fp8 only; fp8_dynamic is an
explicit opt-in and is never picked automatically.
- Enforce _TE_FAMILY_SCHEME_DENY on the explicit text-encoder path too,
gating the final concrete mode (so an int8 -> fp8 fallback is
re-checked), matching the table's documented contract and the VAE
module's behavior. Covered by a new test.
Also merges origin/image-generation (single-GPU fit-budget fix) to keep
the stacked head self-consistent.
The catalog fit budget used gpu.memoryTotalGb, which sums VRAM across
every GPU. That sum is right for the chat/llama.cpp path (tensor-split
shards across cards) but wrong for the diffusion/video catalog: those
backends place the whole pipeline on a single device (pipe.to or cpu
offload, never device_map), so on a multi-GPU host the fit toggle and
bare-group-click routing credited VRAM no single card has. On a 4x24 GB
plus 128 GB RAM host the 114 GB Wan A14B bf16 group passed the toggle
(0.7*96 + 0.7*128 budget) and a click would OOM, the exact load the
toggle exists to prevent. Expose maxDeviceMemoryGb (largest single
device) from use-gpu-info and use it for deviceBudget; the chat path
keeps the sum. Single-GPU hosts are unchanged.
The catalog fit budget used gpu.memoryTotalGb, which sums VRAM across
every GPU. That sum is right for the chat/llama.cpp path (tensor-split
shards across cards) but wrong for the diffusion/video catalog: those
backends place the whole pipeline on a single device (pipe.to or cpu
offload, never device_map), so on a multi-GPU host the fit toggle and
bare-group-click routing credited VRAM no single card has. On a 4x24 GB
plus 128 GB RAM host the 114 GB Wan A14B bf16 group passed the toggle
(0.7*96 + 0.7*128 budget) and a click would OOM, the exact load the
toggle exists to prevent. Expose maxDeviceMemoryGb (largest single
device) from use-gpu-info and use it for deviceBudget; the chat path
keeps the sum. Single-GPU hosts are unchanged.
* Clarify in README that -H 0.0.0.0 starts a public Cloudflare tunnel
* Hedge tunnel URL wording and restore trusted-network caution
* Tighten the 0.0.0.0 tunnel note
* Drop trust-the-network caution from tunnel note
* Restore trusted-network note on the raw-bind sentence
* Use Cloudflare's quick tunnel terminology and consolidate the trust warning
Speed=off contract: the companion (text encoder / VAE) suppression under
an explicit Speed=off only matched an UNSET request, but auto is
backend-owned like transformer_quant, so an explicit
text_encoder_quant/vae_quant=auto would still engage fp8/int8 and break
the bit-exact request. Match 'auto' as well in both the image and video
loaders (a concrete scheme still forces quant). Covered by new
explicit-auto suppression tests.
Benchmark accuracy:
- quant_speedmem_bench teacc: when quantize_text_encoders returns None
(scheme skipped) the encoder is still dense, so scoring it against the
dense reference falsely certified a scheme that never ran. Record it
NOT engaged instead of collecting accuracy metrics.
- quant_speedmem_bench e2e: report the actual engaged te/vae scheme,
falling back to dense (not the requested auto) when the caster stayed
bf16, so a no-op default is not mislabelled as an auto-quantised run.
- video_speedmem_bench: HunyuanVideo ignores callback_on_step_end, so
step_ts stayed empty and per_step_ms was published as 0.0 for every
row. Time the denoise via a scheduler.step wrapper for that path.
* Studio: resolve the repo-root MTP drafter after the MTP/ GGUF rename
The Gemma 4 QAT GGUF repos renamed the higher-precision MTP/ subdir
copies from gemma-4-...-<quant>-MTP.gguf to mtp-gemma-4-...-<quant>.gguf,
so their basenames now start with the same mtp- prefix as the small
repo-root drafter (mtp-gemma-4-E4B-it.gguf).
The drafter selectors filtered candidates by a mtp- basename prefix and
took the first in sort order. With the new names the MTP/ copies also
match, and because MTP/ (uppercase) sorts before the lowercase root file,
selection flipped to the large BF16 copy under MTP/ instead of the root
drafter both functions document they should pick.
Restrict both selectors, and the companion byte estimate, to root-level
mtp-*.gguf so the MTP/ copies stay explicit-selection only:
- core/inference/llama_cpp.py _pick_mtp (loader auto-download)
- hub/utils/gguf_plan.py preferred_mtp_sibling (Hub variant plans)
- routes/inference.py _remote_gguf_companion_bytes (VRAM headroom)
Also reuse a drafter already in the local cache before downloading, so a
device that already holds a copy on disk does not re-fetch it.
Old-scheme names keep working (they have no root-level mtp- sibling to
mis-select). Adds regression tests for the new naming, both selection
paths, and the on-disk reuse.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate MTP drafter cache reuse to offline mode
Reuse the cached drafter only when HF is offline. Online, route back
through _download_companion_gguf/hf_hub_download so the current revision
is checked (etag) and a changed drafter is refetched, matching the
offline-only cross-snapshot reuse already used for the main GGUF. This
avoids pairing freshly downloaded weights with a stale cached draft.
Make the reuse tests offline and add an online-skips-reuse test.
* Studio: prefer a root MTP drafter across all cached snapshots
Offline reuse scanned snapshots one at a time and returned the first
snapshot that held any drafter, only preferring root within it. A newer
partial snapshot with just the MTP/ copy could shadow the small root
drafter in an older snapshot. Collect drafters across all snapshots and
prefer any repo-root file before an MTP/ copy.
* Studio: keep newest-first snapshot order when reusing cached drafters
Collecting root candidates and sorting by absolute snapshot path could
pick a drafter from an older snapshot. _iter_hf_cache_snapshots yields
newest first and the main GGUF is resolved in that order, so preserve it
(root still preferred over MTP/ copies) to avoid pairing a fresh main
weight with a stale drafter revision.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Restore dropped FP8 weight_scale_inv tensors on load
Some block-scale FP8 checkpoints (for example Qwen3.6-27B-FP8, issue #6200) load
with transformers leaving an mlp.gate_proj as a plain bf16 Linear instead of an
fp8 module. Its raw quantized values are read into the bf16 weight and the
weight_scale_inv is dropped as an unexpected key, so the weight is used un-scaled
and the base model is garbage (perplexity around 2 million).
After load, for every checkpoint weight_scale_inv whose live weight is not fp8,
dequantize the orphaned weight in place using the block scale from the checkpoint
index. Modules that were converted correctly keep an fp8 weight and are skipped,
so healthy checkpoints and single-file checkpoints are a no-op.
Verified on Qwen3.6-27B-FP8: 64 gate_proj scales restored, perplexity 2028902 to
8.9. No-op on Qwen3-8B-FP8 (all scales already live).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden FP8 weight_scale_inv restore from review
- Skip restore when the model has no fp8 weights, so an intentionally
dequantized load (load_in_16bit) is never re-scaled and corrupted.
- Thread revision, subfolder and cache_dir through the index and shard
downloads so scales come from the same snapshot as the weights.
- Cover unsharded single-file model.safetensors checkpoints (no index).
- Handle transposed block-scale layouts and skip on a true grid mismatch
instead of applying a wrong scale.
- Match text-only VLM loads where the language_model prefix was stripped.
- Restore on the FastLanguageModel text path too, not only vision.
- Handle a scalar weight_block_size; per-tensor error handling so one bad
tensor cannot abort the rest or hide a partial mutation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address second review round on FP8 scale restore
- Bound peak memory: dequantize block views in place with the fp32 scale
broadcast instead of materializing a full expanded scale and fp32 copy,
so a near-VRAM-limit load is not pushed into OOM by the repair.
- Restore on the sequence-classification load path too.
- Cover more VLM key remappings (language_model.model.* to
model.language_model.*) when matching modules.
- Skip the restore for variant loads (variant=...) rather than risk
applying default-checkpoint scales to variant weights.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Align FP8 scale restore revision with the loaded weights and warn on disk-offloaded layers
In llama.py the CausalLM/SequenceClassification weight loads resolve model_name on its
default branch (revision is not forwarded there), so read the dropped weight_scale_inv
tensors from the same default branch instead of the requested revision, avoiding rescaling
default-branch weights with scales from another revision.
In loader_utils.py a disk-offloaded layer keeps its weight on the meta device until the
offload hook materializes it, so the scale cannot be applied in place. Skip such layers
explicitly and print a warning rather than silently leaving them unscaled.
* Tighten comments in the FP8 scale restore path
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The explicit text_encoder_quant=nvfp4 path gated on the transformer
smoke probe, which builds the dynamic-activation NVFP4 config, while the
TE caster _cast_nvfp4 applies weight-only NVFP4WeightOnlyConfig. On a
Blackwell build that carries the weight-only FP4 path but not the
dynamic FP4 GEMM, the probe would fail and the encoder would silently
stay dense even though the caster would run. Add a dedicated weight-only
NVFP4 smoke probe (mirroring _cast_nvfp4's config) and route TE nvfp4
through it; int8 / fp8_dynamic keep the dynamic transformer probe since
their TE casters are also dynamic-activation.
The video route forwards vae_quant to validate_load_request, but the
_FakeBackend stub in test_video_routes.py had not been updated to accept
it, so all 11 route tests raised TypeError: unexpected keyword argument
'vae_quant'. Add the keyword to mirror the real backend signature.
parse_direct_linux_release_bundle and direct_linux_release_plan are no
longer reached by any live code path. Fork Linux installs resolve through
_fork_manifest_release_plans -> _linux_published_attempts, and the upstream
(ggml-org) path uses direct_upstream_release_plan. The dead parser also
called _resolve_linux_bundle_profile, which no longer exists, so its CUDA
branch would raise NameError if ever executed.
Drop both functions and the obsolete TestDirectLinuxNvidiaCpuGate; its live
equivalent TestLinuxPublishedAttemptsNvidiaCpuGate already covers the
NVIDIA no-silent-CPU behaviour.
* scripts: refresh scan_packages allowlist baseline
Regenerate scripts/scan_packages_baseline.json against the current
resolved dependency set so the blocking pip scan-packages gate matches
what the scanner now finds. Refreshes evidence hashes for benign
findings whose code shifted lines (unsloth-zoo mlx loader, gguf/mlx
test /tmp fixtures) and adds two mainstream-library entries that were
newly surfaced (torch inductor codecache base64+subprocess compile
cache, torch testing common_utils socket import). Stale entries whose
matching code changed and no longer triggers are dropped.
All entries remain CRITICAL/HIGH findings manually judged benign;
matched on (package, file, check, evidence_hash).
* ci(security-audit): re-run scan when the allowlist baseline changes
The security-audit pull_request trigger listed the scanners but not
their allowlist baselines, so a baseline-only edit never re-ran the
scan that consumes it. A refreshed baseline could therefore merge
without CI confirming its evidence hashes match what the scanner finds.
Add scan_packages_baseline.json and scan_npm_packages_baseline.json to
the paths filter so baseline changes are validated on their own PR.
* Keep native RoPE scaling when extending context; carry rope_theta for linear
When max_seq_length exceeds a model's native window, the loader overwrote the
model's rope_scaling with linear scaling. For models that already ship a scaled
RoPE (llama3/yarn/longrope) that is far worse for long context, and on
transformers v5 the linear dict omitted rope_theta (v5 keeps it under
rope_parameters), so the rotary base fell back to 10000 and broke past ~8K tokens.
Keep the native scaling and just widen the window; only synthesize linear for
plain-RoPE models, and carry rope_theta so v5 keeps the real base.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Only preserve native llama3 when extending context; keep linear fallback otherwise
The patched attention constructor (patch_llama_rope_scaling) rebuilds only linear,
llama3 and longrope and its longrope branch reads a top-level
original_max_position_embeddings, so preserving yarn or a nested-only longrope config
would raise during construction on transformers <= 4.47.1. Keep only llama3 native;
yarn/longrope/other types fall back to the linear override, still carrying rope_theta.
* Correct long-context extension comment to match llama3-only preservation
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix fast_gemv crash on compressed-tensors FP8 models
Loading a compressed-tensors FP8 checkpoint (for example
unsloth/Llama-3.2-1B-Instruct-FP8-Block) with fast_inference=False and
running a forward crashed with 'Parameter object has no attribute absmax'
inside fast_gemv.
A compressed-tensors CompressedLinear exposes an already dequantized bf16
weight at forward time while keeping a weight_scale Parameter. The quant
state resolution in get_lora_parameters/get_lora_parameters_bias fell back
to that weight_scale, so a bf16 weight was routed into the bitsandbytes
fast_gemv/fast_dequantize path, which expects a bitsandbytes QuantState
with an absmax attribute.
Only fall back to weight_scale_inv/weight_scale when the weight is still
fp8. A decompressed bf16 weight then resolves to no quant state and flows
through the normal bf16 path, which already handles bias and the LoRA
backward. Real fp8 and bitsandbytes 4bit weights are unchanged.
* Skip the fast_gemv dispatch test before importing unsloth when bitsandbytes is absent
