A cold FLUX.2-dev int8 load on an idle 183 GB B200 planned offload=model
(companions exceed budget) and silently served the GGUF as-is; the identical
retry went resident and engaged the hosted prequant. The plan arithmetic was
byte-identical across both loads (required 90,228 MiB, resident needs free of
about 124 GB); the only divergent input was torch.cuda.mem_get_info, which is
device-wide and instantaneous: a transient foreign CUDA context briefly held
about 100 GB at the first snapshot, and the planner trusted that single read.
Three changes:
- settled_snapshot_device_memory: on cuda, synchronize + empty_cache
(best-effort) and take the MAX free over up to 3 spaced reads. A transient
can only shrink free, so the max rejects transient undercounts while a
persistent tenant still caps every read. _plan_memory now uses it.
- plan_fits_total_capacity + one replan retry: when the dense/prequant
candidate fits TOTAL device capacity under the standard reserve and the 0.85
resident margin, an offload verdict can only stem from the free reading, so
the loader re-snapshots and replans once before declining the fast path.
Explicit balanced/low_vram modes skip the retry (they offload by mode).
- diffusion.transformer_quant_declined log line with required/budget/free and
the plan reasons, so the next decline is diagnosable from the server log
(previously silent).
Verified: cold FLUX.2-dev int8 first load in a fresh server now engages the
hosted prequant resident (offload=none).
cf1910c92d