Wire hosted pre-quantized DiT checkpoints into the image families

Point prequant_repos for flux.1, flux.2-klein, flux.2-dev, qwen-image
(int8 only there; fp8 is family-denied), z-image and krea-2 at the
unsloth/<Model>-FP8 Hub repos carrying gate-validated int8 and fp8
transformer checkpoints, so the fast quant path loads the small
pre-quantized file instead of materialising the dense bf16 transformer
and quantising on device. Measured on FLUX.2-dev int8: build peak drops
from 60.7 GB (dense + quantize) to 30.7 GB (hosted prequant), identical
30.7 GB resident after either path since loading a checkpoint is
bit-identical to on-the-fly quantisation.

The hosted repos name files <Model>-<SCHEME>.pt, so resolve_prequant_source
now derives that model-name filename from the repo id (scheme suffix
stripped case-insensitively) and carries the legacy transformer_<scheme>.pt
as a fallback the resolver tries when the primary 404s, keeping older
repos loadable.

Wiring a repo also exposed a fallback hazard: with a prequant source
present, the dense-fit preflight used to be skipped entirely, so a failed
prequant download would fall through to the dense bf16 load the memory
plan never budgeted, OOMing after eviction. The preflight now always runs
and gates an allow_dense_fallback flag through _load_dense_quant_pipeline:
a dense misfit still skips the fast path when no prequant exists, but with
one it proceeds and a prequant failure raises to the GGUF build instead of
loading dense. The same flag is set when the auto-policy replans an
offloaded GGUF against a prequant-sized transient.

Tests updated to the new filename convention plus new coverage for the
derivation and the legacy-name fallback; the prequant-skips-refit test now
asserts the re-check runs and forbids the dense fallback. Verified end to
end on GPU: z-image int8 resolves the hosted repo, downloads the
model-name file and renders (6.8s load, 5.9 GB peak).
This commit is contained in:
Daniel Han 2026-07-17 05:47:45 +00:00
commit fbcf070fce
5 changed files with 168 additions and 28 deletions

View file

@ -90,18 +90,32 @@ def local_prequant_path_ready(path: str) -> bool:
@dataclass(frozen = True)
class PrequantSource:
"""Where a pre-quantized checkpoint lives. ``kind`` is "path" (a local file) or "repo"
(Hub repo id in ``location`` + ``filename``)."""
(Hub repo id in ``location`` + ``filename``; ``fallback_filename`` is tried when the
primary name is absent, covering repos still on the legacy transformer_<scheme>.pt)."""
kind: str
location: str
filename: Optional[str] = None
fallback_filename: Optional[str] = None
def prequant_filename(scheme: str) -> str:
"""The conventional checkpoint filename for ``scheme`` inside a Hub repo."""
"""The legacy checkpoint filename for ``scheme`` inside a Hub repo."""
return f"transformer_{scheme}.pt"
def prequant_repo_filename(repo_id: str, scheme: str) -> str:
"""The model-name checkpoint filename for ``scheme`` in ``repo_id``: the hosted repos are
named <Model>-FP8 (or -INT8 / -quantized) and carry <Model>-<SCHEME>.pt files, e.g.
unsloth/Z-Image-Turbo-FP8 -> Z-Image-Turbo-INT8.pt / Z-Image-Turbo-FP8.pt."""
model = repo_id.rsplit("/", 1)[-1]
for suffix in ("-fp8", "-int8", "-quantized"):
if model.lower().endswith(suffix):
model = model[: -len(suffix)]
break
return f"{model}-{scheme.upper()}.pt"
def resolve_prequant_source(
fam: Any,
scheme: str,
@ -122,7 +136,12 @@ def resolve_prequant_source(
except Exception: # noqa: BLE001 — a bad family object must not break the load
repo_id = None
if repo_id:
return PrequantSource(kind = "repo", location = repo_id, filename = prequant_filename(scheme))
return PrequantSource(
kind = "repo",
location = repo_id,
filename = prequant_repo_filename(repo_id, scheme),
fallback_filename = prequant_filename(scheme),
)
return None
@ -243,7 +262,19 @@ def _resolve_checkpoint_path(source: PrequantSource, hf_token: Optional[str]) ->
return expanded if os.path.isfile(expanded) else None
if source.kind == "repo":
from huggingface_hub import hf_hub_download
return hf_hub_download(repo_id = source.location, filename = source.filename, token = hf_token)
try:
from huggingface_hub.errors import EntryNotFoundError
except Exception: # noqa: BLE001 — older hub layouts; fall back to a private marker
class EntryNotFoundError(Exception): # type: ignore[no-redef]
pass
try:
return hf_hub_download(repo_id = source.location, filename = source.filename, token = hf_token)
except EntryNotFoundError:
if not source.fallback_filename or source.fallback_filename == source.filename:
raise
return hf_hub_download(
repo_id = source.location, filename = source.fallback_filename, token = hf_token
)
return None