Serialize the GPU handoffs, gate DiT training on a GPU, and keep 3.9 installable
Six review findings, three of them evict-then-fail orderings: - The chat load reclaimed the GPU without telling the arbiter it existed. A chat load holds no llama-server process until its GGUF has downloaded, which is minutes, so a competing Images/Video acquire in that window found nothing to cancel, took the GPU, and the chat load then spawned onto the same device. It now registers an in-flight marker through acquire_for's register hook (under the arbiter lock, as the image and video loads do), the evictor cancels a marked load, and the route undoes itself if ownership moved while it loaded. - The Hub-download conflict check ran after that handoff, so a GGUF the download manager already owns destroyed the resident Images/Video pipeline and then 409'd, having loaded nothing. It moves above the handoff, together with the marker it handshakes with. - The image load released the engine router's transition lock before registering the load, so a second load choosing the other engine could unload the still-idle engine this one captured; the load then landed on a deactivated engine, where generate, status, unload and the arbiter's evictor can no longer reach it. Registration now happens under that lock and refuses if the engine changed. - Training a DiT family on a host with no GPU was accepted: nf4 is not a CPU fallback, its 4-bit load goes through bitsandbytes, which requires CUDA, XPU or MPS. The start unloaded the working Images pipeline, pulled the text encoders, and only then died in the child. Rejected before the teardown now, and /info stops advertising a precision that always 400s. SDXL keeps its documented fp32-on-CPU path. - Both diffusion pages kept the routed-pick marker forever, so re-picking the same checkpoint (after chat evicted it) neither loaded nor cleared the query string. The marker is released once the query is gone. The Images key also carried a stray NUL byte, which made the file read as binary to grep and other tooling. - diffusers dropped Python 3.9 in 0.38, so the unconditional >=0.39.0 pin left pip no candidate at all on 3.9 and made every install that composes the huggingface extras unresolvable there. The floor is conditional now. Also fixes tests that were already red on the branch: two hand-built request fakes had gone stale against fields this branch added, and the handoff-ordering test only failed on a host with fewer than two GPUs.
This commit is contained in:
parent
764adf95d5
commit
bc00a8e797
14 changed files with 520 additions and 69 deletions
|
|
@ -219,17 +219,28 @@ def test_training_precision_preflight_error(monkeypatch):
|
|||
assert training_precision_preflight_error("sdxl", "int8") is None
|
||||
assert training_precision_preflight_error("", "int8") is None
|
||||
|
||||
# On a CUDA-ABSENT host, bf16_unsupported_reason exempts CPU-only, but the DiT trainer's dense
|
||||
# precisions still require CUDA (mirroring _resolve_base_precision), so bf16/int8/fp8/mxfp8 for
|
||||
# a DiT family are rejected UP FRONT rather than after eviction. nf4/auto (and SDXL) still pass.
|
||||
# On a host with NO accelerator every DiT precision is rejected up front rather than after
|
||||
# eviction -- nf4 included, since its 4-bit load goes through bitsandbytes, which needs a GPU.
|
||||
# SDXL (its own fp32-on-CPU path) still passes.
|
||||
monkeypatch.setattr(common, "has_functional_torchao", lambda: True)
|
||||
monkeypatch.setattr(torch.cuda, "is_available", lambda: False)
|
||||
monkeypatch.setattr(torch.xpu, "is_available", lambda: False)
|
||||
monkeypatch.setattr(torch.mps, "is_available", lambda: False)
|
||||
for dense in ("bf16", "int8", "fp8", "mxfp8"):
|
||||
reason = training_precision_preflight_error("flux.1", dense)
|
||||
assert reason is not None and "CUDA" in reason
|
||||
for mode in ("nf4", "auto"):
|
||||
reason = training_precision_preflight_error("flux.1", mode)
|
||||
assert reason is not None and "GPU" in reason
|
||||
assert training_precision_preflight_error("sdxl", "bf16") is None
|
||||
|
||||
# An accelerator that is not CUDA (XPU here) satisfies the 4-bit load, so nf4/auto pass again
|
||||
# while the dense CUDA-only precisions stay rejected.
|
||||
monkeypatch.setattr(torch.xpu, "is_available", lambda: True)
|
||||
assert training_precision_preflight_error("flux.1", "nf4") is None
|
||||
assert training_precision_preflight_error("flux.1", "auto") is None
|
||||
assert training_precision_preflight_error("sdxl", "bf16") is None
|
||||
assert training_precision_preflight_error("flux.1", "bf16") is not None
|
||||
monkeypatch.setattr(torch.xpu, "is_available", lambda: False)
|
||||
|
||||
# mxfp8 needs a Blackwell (sm100+) GPU: its MX GEMM has no kernel below sm100 and would raise at
|
||||
# the first training step, AFTER a full dense-transformer load. On a CUDA GPU that is older than
|
||||
|
|
@ -609,3 +620,36 @@ def test_assert_trusted_base_model_rejects_local_non_pipeline(tmp_path):
|
|||
# An untrusted remote base is still rejected by the trust gate.
|
||||
with pytest.raises(ValueError, match = "untrusted"):
|
||||
common._assert_trusted_base_model("evil/base")
|
||||
|
||||
def test_dit_accelerator_missing_reason_and_info_hide_train_without_a_gpu(monkeypatch):
|
||||
# Clicking Start on a GPU-less host evicted the resident Images pipeline, downloaded the text
|
||||
# encoders, and only then died in the child: diffusers' bitsandbytes quantizer refuses 4-bit
|
||||
# without CUDA/XPU/MPS. Reject it up front, and stop /info advertising a mode that always 400s.
|
||||
import torch
|
||||
|
||||
from core.training.diffusion_train_common import (
|
||||
_DIT_TRAIN_FAMILIES,
|
||||
dit_accelerator_missing_reason,
|
||||
family_train_infos,
|
||||
)
|
||||
|
||||
monkeypatch.setattr(torch.cuda, "is_available", lambda: False)
|
||||
monkeypatch.setattr(torch.xpu, "is_available", lambda: False)
|
||||
monkeypatch.setattr(torch.mps, "is_available", lambda: False)
|
||||
assert "GPU" in (dit_accelerator_missing_reason("flux.1") or "")
|
||||
# SDXL and unknown families keep their own paths.
|
||||
assert dit_accelerator_missing_reason("sdxl") is None
|
||||
assert dit_accelerator_missing_reason("") is None
|
||||
|
||||
infos = {info["name"]: info for info in family_train_infos()}
|
||||
for name, info in infos.items():
|
||||
if name in _DIT_TRAIN_FAMILIES:
|
||||
assert info["precision_modes"] == []
|
||||
assert "GPU" in info["vram_note"]
|
||||
assert info["supports_compile"] is False
|
||||
else:
|
||||
assert info["precision_modes"] != [] or name not in _DIT_TRAIN_FAMILIES
|
||||
|
||||
# Any accelerator clears it (MPS here, which bitsandbytes accepts).
|
||||
monkeypatch.setattr(torch.mps, "is_available", lambda: True)
|
||||
assert dit_accelerator_missing_reason("flux.1") is None
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue