Comment-only pass over the Python this PR touches: drop what the code already
says, collapse multi-line explanations that still read on one line, and keep
the reasoning that is not recoverable from the code. No code, docstring
semantics or behaviour changes; verified with an AST comparison against the
previous revision, and the backend suite is unchanged (same 37 environment
failures as before: the API integration tests that need a live keyed server,
the flash-attn install hooks, and the GPU memory fields).
Six review findings, three of them evict-then-fail orderings:
- The chat load reclaimed the GPU without telling the arbiter it existed. A
chat load holds no llama-server process until its GGUF has downloaded,
which is minutes, so a competing Images/Video acquire in that window
found nothing to cancel, took the GPU, and the chat load then spawned
onto the same device. It now registers an in-flight marker through
acquire_for's register hook (under the arbiter lock, as the image and
video loads do), the evictor cancels a marked load, and the route undoes
itself if ownership moved while it loaded.
- The Hub-download conflict check ran after that handoff, so a GGUF the
download manager already owns destroyed the resident Images/Video
pipeline and then 409'd, having loaded nothing. It moves above the
handoff, together with the marker it handshakes with.
- The image load released the engine router's transition lock before
registering the load, so a second load choosing the other engine could
unload the still-idle engine this one captured; the load then landed on a
deactivated engine, where generate, status, unload and the arbiter's
evictor can no longer reach it. Registration now happens under that lock
and refuses if the engine changed.
- Training a DiT family on a host with no GPU was accepted: nf4 is not a
CPU fallback, its 4-bit load goes through bitsandbytes, which requires
CUDA, XPU or MPS. The start unloaded the working Images pipeline, pulled
the text encoders, and only then died in the child. Rejected before the
teardown now, and /info stops advertising a precision that always 400s.
SDXL keeps its documented fp32-on-CPU path.
- Both diffusion pages kept the routed-pick marker forever, so re-picking
the same checkpoint (after chat evicted it) neither loaded nor cleared
the query string. The marker is released once the query is gone. The
Images key also carried a stray NUL byte, which made the file read as
binary to grep and other tooling.
- diffusers dropped Python 3.9 in 0.38, so the unconditional >=0.39.0 pin
left pip no candidate at all on 3.9 and made every install that composes
the huggingface extras unresolvable there. The floor is conditional now.
Also fixes tests that were already red on the branch: two hand-built
request fakes had gone stale against fields this branch added, and the
handoff-ordering test only failed on a host with fewer than two GPUs.
* studio: fix Backend CI red on main from an ambiguous ordering anchor
test_load_marker_precedes_hub_guard_and_unload fails on main, so every
open PR against the repo inherits the failure.
Root cause. #7239 (a7761e174) reworked the GGUF GPU-pool validation in
_load_model_impl from "if config.is_gguf and effective_gpu_ids is not
None:" to a bare "if config.is_gguf:", placed earlier in the function
than the GGUF load branch. The test anchors on
source.index("if config.is_gguf:"), a first-match search, so it silently
re-anchored onto the GPU-pool statement. #7251 (95f42bcce) then restored
the assertion "= _resolve_inherited_extra_args(" before
"if config.is_gguf:" against a tree where that anchor already pointed at
the wrong statement, and main went red. Checking out 95f42bcce and
running the suite reproduces the same single failure.
The code is correct. _resolve_inherited_extra_args still runs before the
GGUF load branch and before the hub-download guard that consumes
extra_llama_args for require_mmproj, so the guarantee #7251 protects is
intact; only the assertion is wrong.
Fix. Assert that guarantee behaviourally instead of by source offsets.
The new test drives _load_model_impl over a vision GGUF with a stored
--no-mmproj from a previous same-model load and captures the
require_mmproj the hub guard is called with: inherited --no-mmproj gives
False, nothing to inherit gives True, and an explicit request list wins
over the stored one both ways. Moving the resolution call after the
guard makes the inherited case report True and the test fails, so it
detects the reorder the old assertion was meant to catch, without
depending on how many "if config.is_gguf:" statements the endpoint has.
The surviving marker-before-guard-before-unload assertion had the same
ambiguous anchor for its slice start, silently widening the slice past
the GPU-pool block. It now slices from the "if config.is_gguf:" nearest
above the in-flight marker, which pins the load branch.
The structlog test stub gains a get_logger factory so routes/inference.py
is importable when structlog is absent.
34 pass in tests/test_gguf_load_cache_reuse.py (was 32 pass, 1 fail);
350 pass across it plus test_llama_cpp_mmproj_fallback.py and
test_llama_cpp_mtp_detection.py. A full backend run before and after is
identical apart from this test going from fail to pass.
* studio/tests: repair a pre-existing bare structlog stub before importing routes
* studio/tests: tighten the comments on the new load-ordering coverage
* Tighten comments on the load-ordering coverage for PR #7442
_load_model_impl contains more than one `if config.is_gguf:`, so
source.index() returned the earlier one, which belongs to a different check
than the branch the assertion is reasoning about. The inheritance call sits at
line 4543, the earlier branch at 4508 and the branch holding the load marker at
4567, so the comparison read 186995 < 185014 and failed on main.
The branch is now located from the load marker itself, which is the landmark
the rest of the test already relies on, so the assertion compares the
inheritance call against the branch that actually guards it. The slice used by
the following assertions is anchored the same way, which also tightens them:
they previously searched from the earlier branch to end of file.
The invariant is unchanged and still has teeth: moving the inheritance call
after the branch makes the assertion fail.
Co-authored-by: danielhanchen <unslothai@gmail.com>
The gguf order fix that landed on main dropped the only assertion
covering the prerequisite that llama_extra_args inheritance runs before
the GGUF branch: the inherited value (a carried --no-mmproj) shapes the
hub guard's require_mmproj, so a future reorder could reject a load
over an mmproj download the inherited arguments would disable. The
comment also misattributed the inheritance site to
_guard_chat_load_against_training.
The assertion is restored anchored on the call form
"= _resolve_inherited_extra_args(", which pins the endpoint's call site
(the bare name would match the function definition, which always
precedes the endpoint, making the check vacuous), and the comment now
names the real inheritance site. 32 tests pass.
Adds a configurable Hugging Face model download cache location to Unsloth Studio, selectable from Settings, with per-cache download manifests, scoped deletion, and read-only inventory of previously selected caches.
#6414 moved the llama_extra_args inheritance out of the GGUF branch in
_load_model_impl into _guard_chat_load_against_training, which runs before the
branch, so 'if request.llama_extra_args is None' is no longer inside the
gguf_branch slice that test_load_marker_precedes_hub_guard_and_unload checks.
The assertion failed on that now-missing landmark even though the guarantee it
protects (the gguf_load_in_flight marker is entered before the hub-download
guard and the unload) is intact. Drop the relocated landmark from the ordering
so the test matches the current structure.
Co-authored-by: danielhanchen <unslothshared@gmail.com>
