ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Studio: fix the "No model loaded" error, and optionally auto-download a model named in an API request (#7454)

Browse source 
* Studio: say which model is missing instead of "No model loaded"

A /v1 request naming a model that is not downloaded returned the generic
"No model loaded. Call POST /inference/load first.", which cannot fix it.
Return 404 model_not_found naming the model and listing what can serve,
and make the API usage examples name a model the server actually has.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: page the API monitor, show model load/unload, pin the example quant

The monitor rendered all 50 retained entries in one scroller: page it 5 at a
time, freezing history while paged back so live traffic cannot reorder it.
Add model load/unload rows so the feed shows what the server is doing, and
stop the header rendering the loaded model as a raw host path. Advertise each
model's GGUF quant on /v1/models so the example pins repo:QUANT, and move the
auto-switch section above the monitor with shorter copy.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: optionally download a model named in an OpenAI API request

Auto-switch only ever loaded models already on disk, so naming one this
server does not have either 404s or, when something else is loaded, gets
quietly answered by the resident model.

Add openai_api_auto_download_model (off by default, gated on auto-switch).
When on, a /v1 request naming a GGUF repo that is not downloaded starts a
background fetch and returns 503 with Retry-After and a typed
model_downloading code. The resident model keeps serving in the meantime,
and the retry after the download completes is served by the new model
through the existing auto-switch path.

The download reuses the Hub manager's service layer, which already does
repo-id validation, casing, claim bookkeeping, disk preflight, resume and
cancel. The in-loader download is deliberately not used: it silently falls
back to a smaller quant under low disk, which is wrong when the caller
named an exact one.

Admission is narrow, since a request only needs an API key:

- namespace/name only, so gpt-4 and other foreign ids fall through to the
  resident model exactly as before
- GGUF only, decided from the remote file list rather than the repo name
- anything declaring auto_map is refused, so trust_remote_code stays a
  deliberate opt-in in the UI and can never be granted over the API
- a single download at a time, plus a free-disk reserve
- one model_info call answers existence, gating and the quant list, so a
  missing repo, a gated repo and a wrong quant each get their own error

With the setting off every one of these paths is byte-identical to before.

Also:

- monitor rows for downloads, with a live percentage
- public_model_id resolves an HF cache snapshot to its repo id, so a
  cache-loaded model is no longer labelled with a commit sha; this drops
  the duplicate helper added for the monitor and fixes the same leak in
  the inference status response
- the unedited sk-unsloth-YOUR_KEY from the copyable examples now says so
  instead of "Invalid or expired API key"; every other bad key keeps the
  generic message

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: add an Unload button to the API monitor

The monitor names the loaded model but offered no way to free it. Idle
auto-unload is the only existing release path, and it needs a TTL and a
wait.

The button sits next to Refresh, appears only while a model is loaded and
is disabled mid-unload. /unload matches on the internal identifier, which
this response deliberately omits because it would be a host path, so the
click reads it from /api/inference/status the same way the chat runtime
does rather than widening the monitor payload.

Also stamp the manual unload row with the quant, read before the teardown
clears it, so it reads repo:QUANT like the load row it pairs with.

* Studio: keep the API monitor Unload button visible when idle

It only rendered while a model was loaded, which hid the one manual
release path at exactly the moment someone goes looking for it. Render it
always, disabled with a "No model is loaded" tooltip when there is nothing
to free.

* Studio: never answer a named model with a different one

Asking for a model this server is not serving returned 200 from whatever
was resident. Requesting gemma-4-E2B-it-GGUF:UD-Q6_K_XL while UD-Q4_K_XL
was loaded got a confident answer from the wrong quant, with nothing in
the response saying so.

A name carrying a namespace (org/model, optionally :QUANT) is a concrete
reference, so 404 instead, with the reason:

- wrong quant  -> names the quants that are actually downloaded
- not on disk  -> lists what is available
- on disk but auto-switch off -> says to turn it on

Ids without a namespace (gpt-4, claude-3, default) are foreign labels
rather than references, so they still fall through to the resident model
and drop-in clients are unaffected. A bare org/model is still satisfied by
any loaded quant of that repo; only an explicit :QUANT must match.

The check runs whatever the auto-switch and auto-download toggles are,
since serving the wrong weights is wrong in every configuration. It is
skipped when nothing is loaded, where the existing no-model-loaded error
already says the right thing, and when the model is on disk with
auto-switch on, where a failed swap should still fall back.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: use a simpler prompt in the API usage examples

"What is Unsloth Studio?" rather than "Can Unsloth Studio do API calling?".
One constant feeds all nine snippet tabs.

* Studio: only refuse a model reference meant for this server

A namespace alone was treated as a concrete model reference, so a /v1
request naming anthropic/claude-3.5-sonnet, openai/gpt-4o or any other
LiteLLM or OpenRouter style vendor/model id started returning 404 instead
of being answered by the resident model. Refuse only on evidence the
caller meant this server: an explicit GGUF quant label, or a repo that is
actually on disk here. gpt-4 and vendor/model alike fall through again,
while the wrong-quant and wrong-repo cases this PR exists for still
refuse.

Also from review:

- Release the single download slot by object identity, not repo id. A
  stale watcher could clear a newer download of the same repo and let a
  second multi-GB fetch start alongside it.
- Catch BaseException around admission: CancelledError is not an
  Exception, so a cancelled request stranded the slot for the process
  lifetime.
- Honour the download service's accepted=False, which it returns without
  raising for a cross-variant conflict, instead of promising a download
  that was never dispatched.
- Treat a failed status probe as unknown rather than idle, so a transient
  read cannot fail the monitor row and free the slot under a live worker.
- Check gated repos with auth_check. The Hub serves metadata for a gated
  repo without granting its files, so the licence gate was being reported
  as the unrelated custom-code refusal.
- Size the disk reserve from the download plan, which includes the mmproj
  and MTP companions the worker fetches with every quant.
- Never fetch under the server's own HF token. The repo is named by
  whoever holds an API key, so the ambient token let that key pull the
  owner's private repos.
- Refuse an explicit quant on a backend with no quant identity, gated on
  the suffix really being a quant so Ollama style :latest tags still match.
- Raise instead of falling through when the diagnosis fails: the mismatch
  is already established by then, only the wording is uncertain.
- Report a failed switch as 503 model_switch_failed rather than answering
  as the resident model.
- Fail an open monitor row under the same lock as the check, so a finish
  landing in between cannot stamp an error onto a completed row.
- Usage examples never emit a hardcoded model id: the catalog is tri-state
  and the panel asks for a model to be loaded instead of printing one the
  server cannot serve. It also refreshes when the loaded model changes.
- Keep the monitor pager reachable while frozen entries expire.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: scope the auto-download 404 cache to the caller's credentials

The Hub answers 404 for a private repo the caller cannot see, so caching
that verdict per repo alone let one anonymous request mark a private repo
unservable for everyone for the whole TTL. A later caller sending a valid
X-Unsloth-HF-Token skipped the probe and fell through to the resident
model instead of downloading what it asked for. Keyed on the repo id plus
a digest of the token now, so the token itself is never held.

Two more from the same review:

- Clear the chat runtime checkpoint after unloading from the API monitor,
  as the chat eject flow already does. The store went on treating the
  freed checkpoint as loaded and the usage examples kept naming it.
- Point gated and not-found callers at the X-Unsloth-HF-Token header.
  Automatic download deliberately ignores the server's own Hugging Face
  identity, so telling the user to add a token in Studio sent them round
  the same 403 forever.

* Studio: tighten the comments added by this branch

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: keep API auto-download off the server's Hugging Face identity

Passing None for the caller's token was not anonymous. spawn_worker
substitutes the backend's HF_TOKEN for a falsy one, and HfApi(token=None)
falls back to a cached login, so a repo named by an API-key holder could
still be fetched under the owner's Hub identity and land in the shared
catalog. The metadata probe and auth_check now pass an explicit False,
and dispatch threads allow_ambient_token=False so the worker stays
anonymous too. The flag defaults to True, so the UI download path keeps
the ambient fallback that private repos rely on.

Three more from the same review:

- Require an exact hf_variant match only when the suffix is really a
  quant. The llama.cpp branch still compared Ollama style :latest and :8b
  against the loaded quant and refused the resident model, which is the
  opposite of what looks_like_quant classifies them as.
- Decode an HF cache repo id only when the models-- component is followed
  by snapshots. An ordinary directory whose name merely starts with
  models-- was being read as an encoded repo id.
- Return the probing response before consulting the job registry when an
  adopted claim has no variant yet. A stale error on the whole-repo key
  could otherwise release the slot the first request's probe still holds,
  letting a second large download start beside it.

* Studio: stop treating a namespace as what decides model intent

The rule refused a reference only when it carried a namespace, which was
wrong in both directions. vendor/model is how LiteLLM and OpenRouter name
every provider, and a standalone or custom-folder GGUF is advertised
without one, so asking for a path-free local id such as model-Q4_K_M was
answered by whatever else happened to be resident. The slashless early
return is gone and the same evidence test now applies to every id: an
explicit quant, or a model that actually resolves here. gpt-4 and default
still fall through because they are not local, not because of their shape.

Also:

- Recognise bits-per-weight quant labels. _extract_quant_label emits
  IQ4_XS-3.53bpw and the resolver and downloader both accept it, but
  _GGUF_KNOWN_QUANT_RE has no bpw group, so looks_like_quant rejected a
  reference the rest of the machinery understands.
- Upper-case the synthetic names handed to _pick_best_gguf. Its preference
  tokens are upper case and matched case-sensitively, so a repo with
  lower-case filenames skipped the preference and took the first entry,
  which can be F16.
- Only offer a downloaded but unloaded model as a runnable example when
  auto-switch is on. It is off by default, so the copied snippet hit the
  no-model-loaded error, which is the failure this branch exists to fix.

The tool-passthrough cancel test stubbed asyncio.to_thread module-wide, so
it cancelled at the first thread hop rather than the generation hop it
means to test. Model resolution runs off the loop before the monitor row
opens, so that stub now passes the resolver through.

* Studio: tighten the comments added since the last pass

* Studio: match a resident model through its resolver alias

A manual load stores the model by its on-disk path while the resolver and
/v1/models advertise it as publisher/model, so _loaded_satisfies could not
recognise the alias. Reducing the resolution to a boolean then threw away
the load path that would have proved the match, and the request was
refused with 404 for a model the server was serving at that moment. Common
for LM Studio models and custom-folder aliases. The resolved path is
compared against the resident backend before anything is refused.

Also:

- Size disk admission on what is left to fetch. expected_bytes is the whole
  plan, so a resumed quant or a companion already pulled in by another
  quant was charged for twice and could 507 a download that fits. Cached
  blobs are subtracted through existing_blob_bytes, the same accounting the
  worker's own preflight does, and it falls open to the full size when no
  blob hashes are available.
- Report a cancelled download as cancelled. The catch-all sent every state
  other than complete or idle through fail_open, so a deliberate cancel
  rendered as a download failure rather than the monitor's cancelled state.
- Keep polling the servable ids while nothing is loaded. The poll settled
  as soon as auto-switch was on, so turning it back off left the examples
  naming an unloaded model until something else remounted the panel.

* Studio: shorten the comments added in the last pass

* Studio: keep the FLA fast-path tests hermetic across transformers versions

_discover_fla_model_types scans the *installed* transformers for modeling
files importing `from fla.`, so `models/qwen3_5/` only exists from
transformers 5.x. The backend supports transformers>=4.51, and on a 4.x
install the Qwen3.5 gate returns False, so 14 tests in
test_training_worker_flash_attn.py silently exercised a no-op instead of the
install path and failed their call-count assertions.

Pin the discovered model_type set in those 14 tests, the same way
test_hook_does_not_install_tilelang_for_model_outside_allowlist already pins
it against newly added FLA model_types. Test-only change: the production
gate and the _discover_fla_model_types unit tests are untouched.

* Studio: keep the /v1 admission check off the model-scanning path

The admission check added here runs on every /v1 request, including with
auto-switch off, where the route used to return straight away. It called
resolve_local_gguf, whose index is cached for 5s and otherwise rebuilt by
walking ./models and every HF cache root, under a lock the next caller waits
on. On an install with a large cache that scan measured 6.1s, longer than the
TTL that is meant to amortise it, so steady traffic would keep rebuilding it.

Answer from the last built index instead and never rebuild from the request
path: a stale answer is fine here, since what is on disk barely moves and a
finished download already invalidates the index. The first request, before any
scan has completed, warms the index on a background thread and skips the check
rather than blocking on it. That also makes the lookup a dict read, so it no
longer needs handing to a thread.

Cold resolve on this box goes from 6152495us to 0.4us, and the whole hook now
costs the same for a foreign label as for the resident model.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: fix the admission hook's cold, stale and contended index paths

Five review items, four of them on the admission hook added here.

Skipping the check until the first scan lands also skipped explicit quant
mismatches, so the first request after startup could ask for :Q8_0 while
Q4_K_M was resident and be answered by it. The early return was redundant as
well: with an empty index resolved is None and here is False, so the gate below
already lets a bare name through and refuses an explicit quant, which is what
the except branch has always concluded. Dropped it and index_is_built with it.

index_is_built took _lock, which _index holds for the whole scan, so once a
warm was running every later request blocked on the event loop for exactly as
long as the scan it was there to avoid. The warm now has its own lock and reads
the timestamp unlocked, which is safe because _scan is only ever rebound.

Warming only when the index had never been built left a model fetched in the
Hub UI, or dropped into a scan folder, invisible for the life of the process,
since only the auto-download watcher calls invalidate_index. Warm on staleness
too, and unconditionally, so it refreshes within a TTL without a scan on the
request path. Rescanning is capped at a tenth of the scan's own duration: a big
install takes longer to scan than the TTL, and warming on the TTL alone would
keep a thread scanning continuously.

An Ollama-style tag names no quant, so the resolver misses it and auto-download
saw a model the resident one already answers to, then 404'd it for having no
such quant. Return early when the loaded model satisfies the reference.

Frontend: a cancelled download said "Model download failed", because the label
collapsed everything non-completed into failure.

The backend tests get an autouse fixture that stops the warm from walking the
developer's real HF caches; that scan starved the loop under the timing
sensitive streaming tests.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: make /v1/models and the admission hook agree on what is local

Three review items, all on the seam between the catalog scan and the resolver
index, which run on separate schedules.

/v1/models can advertise a local GGUF the resolver has not indexed yet. A bare
id carries no quant to refuse on, so a client asking for one it had just been
handed was answered by the resident model instead. The hook now reads the
catalog cache as evidence too, never scanning it. It takes the path rather than
a yes/no because the converse also happens: the catalog can list the resident
weights under an alias the loaded entry does not answer to, and those must stay
served.

That alias was also emitted twice by /v1/models, once as the loaded basename a
manual load records and once as publisher/model marked unloaded, because the
dedup only compared ids. Compare the path as well.

A directly loaded standalone .gguf takes its quant from the filename, but the
resolver stores such files with no quants, so the advertised <stem>:<quant>
stopped resolving as soon as anything else loaded. Advertise a quant only when
that reference resolves, and downgrade only on a definite answer so a cold
index leaves the metadata alone.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: tighten the comments this branch adds

Collapse the multi-line notes in the auto-download path, the /v1 admission
hook and their tests to one line each, keeping the reason and dropping the
restatement. No behaviour change.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: four admission and catalog fixes from review

Lowercasing paths in _resolves_to_resident made /srv/models/Foo and
/srv/models/foo the same weights on any case-sensitive filesystem, so a request
for one could be answered by the other and /v1/models could mark the wrong
entry loaded. That helper now backs residency as well as admission, so use
os.path.normcase, which folds case only where the filesystem does.

Advertising a quant whenever the resolver could not disprove it kept the bug it
was meant to fix: a standalone .gguf loaded before the first scan still got
<stem>:<quant> published, and the usage examples persist that. No proof is not
proof, so omit it and warm the index instead.

A 401 from an expired or invalid X-Unsloth-HF-Token skipped the 403 and 404
branches and surfaced as "could not reach Hugging Face, retry shortly". It now
says to replace the token, kept apart from the gated refusal since a rejected
credential is not an unaccepted licence.

An image request naming an undownloaded text-only GGUF started the whole
download and only then hit the capability guard, which never sees a remote
target, so every retry 400d and the bytes were wasted. Thread require_vision
into admission and check it against the mmproj companions the disk preflight
already asks build_gguf_variant_plans for.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: make the Hub error fixture carry a status on both hub majors

The 401 test built HfHubHTTPError directly, which works on 0.x and fails on 1.x
where response is required and keyword-only, so all four Python jobs failed
while the same test passed locally.

_hub_error already handled both constructors, but the 0.x branch left the
exception with no response at all, and hf_error_status reads the status off it
for the types that do not encode it in their name. So it could only produce a
usable error on 1.x, which is why the test bypassed it. Attach the status when
the constructed exception lacks it, and use the helper.

Cover the helper itself against stand-ins for both constructor shapes, since
whichever hub is installed only ever exercises one of them.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: invalidate on every download, resolve bare tags, keep polling

Three review items.

Only the API auto-download watcher dropped the resolver cache, so a GGUF
fetched in the Hub UI stayed absent to the cache-only request path and the
request was answered by whatever was resident. finalize_worker_exit is the one
point every download worker exits through, so invalidate there. That closes the
window without leaning on the TTL, which the scan-duration throttle can stretch
past 5s on an install where the scan itself takes longer than that.

A downloaded but unloaded GGUF asked for as org/model:latest missed the
resolver, since the suffix was always treated as an exact quant. With
auto-download on that probed the Hub and returned a 404 for a quant that was
never a quant; with it off it refused without switching. Fall back to the base
entry when the suffix is not quant-shaped, and keep exact matching for real
quants so a swap can never serve the wrong weights under the right name.

The usage examples stopped polling once a model was resident, but idle unload
frees one without touching the store, so nothing re-ran the effect and the
examples kept naming a model that could no longer be reloaded. Slow the poll to
60s instead of stopping it.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: hold the download slot while it is in use, and keep quants to llama.cpp

_loaded_satisfies refuses a quant reference against the Transformers backend by
name, but the path match did not carry that rule. A Transformers model active
from a directory that also holds GGUF exports therefore matched a request for
one of those quants and answered it with the safetensors weights. Only
llama.cpp has a quant identity, so admission now passes llama_only whenever the
reference is quant-qualified. A bare name still matches either backend, and
/v1/models residency keeps the default so a loaded Transformers model is still
reported loaded.

The 24 hour watch window was bounding ownership of the single-flight slot when
it should only have been bounding progress reporting, so a legitimately slow
download had its slot handed back while the worker was still writing, admitting
a second multi-gigabyte download beside it. Resolve the row on the clock, but
keep the slot on a slower poll until the job is actually terminal. Past the
deadline an unknown state does release it, since it means the worker cannot be
probed and holding it on that forever would wedge auto-download.

* Studio: keep what the resolver already knew when a download lands

Invalidating cleared the index to empty. The request path reads that cache
without scanning, so from a completed download until the rebuild landed it had
no evidence about any local model, not just the new one, and a bare request for
any of them was answered by whatever was resident. Wiring the hook into the
shared completion path in the last commit widened that from auto-download to
every download.

Mark the scan stale and keep the entries instead. Both _index and
warm_index_soon rebuild on a zero stamp, while the request path still sees
everything it knew a moment ago. Only a completed download invalidates, and
that only ever adds models, so nothing retained goes false.

Warm from the completion hook too, so the rebuild starts when the download
lands rather than when the next request happens to need it.

* Studio: match the quant, not just the directory, and default-select bare tags

Two quants of one repo share a directory, so the path match could not tell them
apart and an explicit :Q8_0 was answered by a resident Q4_K_M that
_loaded_satisfies had already refused by name. The llama_only fix in the last
commit only ruled out the wrong backend, not the wrong quant on the right one.
Both path matches now require the resident hf_variant to equal the requested
quant whenever the reference is quantified; a bare name still matches on the
path alone, since it claims nothing about the weights.

The local resolver already treated a tag that names no quant as meaning the
repo, but remote admission still looked for a quant literally called "latest",
so the same reference resolved locally and 404d remotely. Branch on
looks_like_quant there too. A real quant the repo does not have is still a 404
and never a substitution, which is what separates this from the loader's
low-disk fallback.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: one quant preference, and stop trusting a stale checkpoint

list_local_gguf_variants sorts by descending size, so the head of variants was
the biggest quant, often F16, while remote admission and a plain load both rank
through _pick_best_gguf. A bare id therefore meant a different quant depending
on which side answered it, and the local answer was the one that could evict a
working model and then fail or OOM starting an F16 next to a usable Q4.
/v1/models advertised that same head for pinning. Pull the ranking into one
preferred_quant helper and have both sides use it.

The usage examples returned a stored checkpoint without ever consulting
/v1/models, and the polling added last round was gated on not having one, so
for a stored checkpoint it never ran. An idle unload then left the panel
showing a snippet that could not run. Poll whenever mounted, and prefer the
checkpoint only while the catalog still backs it or switching can reload it. A
catalog that has not answered yet is not evidence against it.

The static contract pinned the old dependency array, so it now asserts the
intent it documents: a finished load re-runs the fetch, and the effect is not
gated on having no checkpoint.

* Studio: fix the Windows path compare, and advertise a label the worker knows

The case fix normalized the separator to "/" and then called os.path.normcase,
which on Windows folds case and rewrites the separator back to a backslash, so
the descendant checks compared against a "/" the path no longer had. A manually
loaded GGUF reached through an alias then read as a different model, giving a
false 404 and an alias marked unloaded. Run normcase first and normalize the
separator after it.

There are two quant-label extractors and they only agree while a recognized
quant token is present. With none, _extract_quant_label takes the last
hyphenated segment, "7b" of llama-7b, while build_gguf_variant_plans and the
worker key the whole stem: the plan lookup missed and the job exited on a
variant it had no shards for. Use the canonical extractor for the unrecognized
case only. Checked across real filenames first, the two match on every
recognized quant and part on bpw-qualified labels, which _extract_quant_label
keeps apart on purpose so byteshape's IQ4_XS at 3.53, 3.97 and 4.19 bpw stay
separate variants.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: a stored checkpoint needs catalog evidence, not just the switch setting

Preferring it whenever switching was on short-circuited the catalog check, so a
checkpoint the store still held after the model was deleted or moved kept being
named even though /v1/models had already proved it absent, and the snippets 404d
instead of falling back to a model that is actually there.

A lookup rather than a disjunction, which settles the whole matrix in one place:
no answer yet keeps the checkpoint, since that is not evidence against it; listed
and resident keeps it; listed but unloaded keeps it only when switching can
reload it; absent falls back whatever the setting says.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: normalize the quote style pre-commit would have rewritten

* Studio: cover the model that just landed, and pin the quant the catalog has

Retaining the index on invalidation protects what was already scanned and by
construction cannot contain the model that just finished downloading, so a bare
request for it in the window before the rebuild was still answered by the
resident model. Record the repo at the completion hook and treat that as
admission evidence alongside the resolver and the catalog; the next completed
scan clears the notes, since the index then covers them. Publishing a rebuilt
index before completion becomes observable would have closed it too, but that
blocks the download worker for the length of the scan.

Catalog membership proves the repo, not the saved quant, and the examples then
pinned the stored one. A quant deleted while another quant of the same repo
remained produced repo:deleted-quant, a missing-quant 404 with a runnable
alternative listed right beside it. Pin what the catalog advertises: for a
resident entry that is the resident quant, for an unloaded one it is a quant
actually on disk. The store is only consulted before /v1/models has answered.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: apply three rules everywhere they belong, not only where reported

The trust probe was the last credential handoff still passing a raw token.
huggingface_hub reads None as "use the cached login", so a caller-named repo
was read with this server's Hugging Face identity whenever the caller sent
none, which is exactly the isolation the metadata probe and the worker already
keep. It takes _hub_token now. Enumerated the rest of that path while there:
auth_check, model_info and spawn_worker were already correct.

finalize_worker_exit is shared with dataset downloads, so the resolver hook
fired for every completed dataset, scanning the model directories for nothing
and recording the dataset id as local-model evidence, which turns a bare /v1
request naming that id into a refusal instead of a foreign-id fallthrough.
Gated on repo_type.

_already_serving decided "bare" on the presence of a colon while
_loaded_satisfies and the resolver decide it on whether the suffix names a
quant, so org/model:latest against a serving Q8_0 read as a mismatch and
swapped in the preferred Q4_K_M for a request either one answers. That rule now
lives in four places, each fixed in its own round, so this time I looked for
the rest and found a fifth: describe_local_miss splits on the bare colon and
its docstring claims it splits like the resolver. It no longer did, and would
report a missing quant named "latest". Fixed here too, unreported.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: probe before refusing busy, and scan once when the index is cold

The busy refusal fired before anything established the requested label was a
model at all, so any namespaced id a drop-in client sends was told to wait out
an unrelated download for as long as it ran. Probe first and refuse only a
label the Hub actually serves as GGUF; anything else falls through to the
resident model as before. A probe failure answers "not downloadable", since
stranding ordinary traffic costs more than missing a busy refusal.

Treating an unbuilt index as "nothing here" let the first request after startup
be answered by the resident model under another model's name. That was a
deliberate trade to keep the scan off the request path, and it was the wrong
one. Cold, the scan now runs once on a thread, bounded so a pathological
install falls through rather than hanging the request. Built, the request path
still never scans, so the latency fix stands.

The watcher freed the slot the moment it saw an error, while Retry-After is
thirty times the poll interval, so the client came back to an empty slot and
restarted the same failing download instead of being told. Hold the failure on
the slot until a retry surfaces it, and let another repo take it after three
retry intervals so a client that never returns cannot keep it.

The watcher also invalidated on completion, which now lands after
finalize_worker_exit's warm and marks that fresh scan stale, pushing a
synchronous rescan onto the retry. Removed.

_loaded_satisfies lowercased paths as well as aliases, so it returned satisfied
before the case-preserving compare below could run. Both now go through one
helper: paths compare with normcase, aliases stay case-insensitive.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: an unfinished scan is not absence, and a decided refusal is not a failure

Bounding the cold scan then reading the bound as "not here" left the same hole
one branch over. A timeout now answers 503 model_indexing with a Retry-After
and leaves the warm running. A foreign label sent inside that window is asked
to retry rather than falling through, which is a real cost, but the window is
one request on an install whose scan exceeds ten seconds and it clears itself,
where answering with the wrong weights does not.

That uncovered a worse one. Every check here runs inside a broad except whose
job is "could not verify, so fall through", so an HTTPException raised in the
block was logged as a verification failure and the request was answered by the
resident model. Any refusal decided in there was being swallowed. Re-raise it
ahead of that handler.

Canonicalizing generic labels made them real variant keys, but the matcher
still decided on shape, so repo:llama-13b fell past an exact match and fetched
llama-7b. Match exactly first, whatever the shape; a quant-shaped suffix that
matches nothing is still a miss and never a swap.

Marking a catalog alias loaded while publishing the preferred on-disk quant
claimed alias:Q4 was loaded while Q8 was serving, and requiring the resident
quant to match then made pinning it a 404. Advertise the resident variant when
the entry resolves to the resident model.

* Studio: keep the asyncio.timeout fallback tests runnable on Python 3.10

Both tests deleted asyncio.timeout to force _wall_clock_timeout down its
pre-3.11 branch, but monkeypatch.delattr raises when the attribute is already
absent. On Python 3.10, the one version the fallback exists for, there is
nothing to delete, so the two tests errored with AttributeError before reaching
the code they cover. Passing raising=False makes the deletion a no-op there and
leaves the assertions running against the same branch on every version.

Every other delattr in the repo already passes raising=False for exactly this
reason. Verified with asyncio.timeout removed from the interpreter: the two
tests fail with the CI AttributeError before this change and pass after, and
the file still runs 89 passed on 3.13 where the deletion is real.

* Studio: decide GGUF residency, servability and variant keys by one rule each

Four admission and catalog fixes, each closing a gap between two places that
were answering the same question differently.

The /v1/models catalog asked _resolves_to_resident without llama_only, so a
Transformers model live from a directory that also holds GGUF exports marked a
GGUF alias loaded and gave it a GGUF quant. The usage examples then pinned
alias:quant that nothing could serve with switching off. Every entry in that
loop is advertised as GGUF, so residency there is llama.cpp residency.

The busy probe accepted any .gguf sibling while admission excludes mmproj, MTP
drafters and big-endian builds. A repo holding only companions is not
downloadable, so it was held at model_download_busy for the length of an
unrelated download instead of falling through to the resident model as it does
when no download is running. It now reuses _gguf_variants, the same filter.

split_model_ref refused any slash-bearing suffix, but an unrecognized GGUF below
a subdirectory keys on its path (build/llama-13b), which is_valid_gguf_variant
allows and the catalog advertises. Pinning such a variant could not parse, so
only the default-ranked one was reachable. A slash-bearing suffix is now a
variant exactly when a real Hub repo precedes it, which still leaves
C:/models/x.gguf a path rather than a quant.

The usage examples treated a downloaded-but-unloaded model as runnable only
under auto-switch, but a standalone UNSLOTH_MODEL_IDLE_TTL reloads exactly what
it freed on the next request. The panel hid runnable examples after an idle
unload. Tracked apart from auto-switch, because the stash restores the stored
checkpoint only and never an arbitrary catalog entry.

Also stub the index walk in the three cold-index tests that missed it: a real
multi-root scan inside the cold-wait budget made them time out into a 503 under
load rather than assert what they are there for. One of them flaked locally.

Verified each fix is load-bearing by reverting it and watching its test fail.
Backend CI command: 10195 passed, 0 failed. tsc -b and the frontend build clean.

* Studio: bound the Hub admission probes and stop guessing at nested model paths

Three review fixes plus a test-isolation one.

_resolves_to_resident matched on a path prefix, so two separately indexed models
that nest (/models/A alongside /models/A/sub/B) both satisfied it: loading B
made a request for A resident and answered it with B's weights, and the catalog
marked A loaded. A prefix match now counts only when no catalog entry sits
deeper, which is the innermost indexed model that actually owns the file. With
nothing indexed there is no nesting to tell apart, so the directory-to-weights
match this exists for is unchanged.

auth_check and hf_hub_download take no timeout of their own, and both ran while
the provisional single-flight slot was held, so an unresponsive Hub stalled the
request far past the metadata budget and reported every other model busy for the
duration. Both are bounded now. Each default errs the safe way: an unchecked
repo is not a cleared one, so the custom-code probe refuses on timeout, while a
slow gated-repo check stays inconclusive because the download's own auth is the
real gate.

The usage examples caught a failed refresh into an empty catalog and a disabled
auto-switch, which made a transient error authoritative and blanked every
example while the model was still servable. The catalog is deliberately
tri-state; a failure now keeps the last answer and retries.

Also start the backend tests from a built, empty model index. Stubbing only the
background warm still left the cold path walking real caches synchronously
inside the admission wait, so on a large install a test asserted against a 503
"still indexing" instead of its subject. _build_index is untouched, so the tests
that call it directly still exercise the real walk.

Verified each fix is load-bearing by reverting it and watching its test fail.
tsc -b clean. Backend CI command green apart from two failures reproduced only
on this box (a real model-dir scan and an orphan-process cleanup), neither
touched by this PR; staging CI is the gate for those.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Daniel Han 2026-07-27 05:02:06 -07:00 • committed by GitHub
commit da447d47ba
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
28 changed files with 5231 additions and 292 deletions
Download patch file Download diff file Expand all files Collapse all files

18
studio/backend/auth/authentication.py
View file

@ -164,6 +164,22 @@ async def get_current_subject_allow_password_change(
)
# The literal the examples ship with; pasting one unedited is likelier than a revoked key.
API_KEY_PLACEHOLDER = f"{API_KEY_PREFIX}YOUR_KEY"
def _invalid_api_key_detail(token: str) -> str:
"""Why the key failed. Only the unedited example placeholder is called out;
every real key still gets one indistinguishable message, so this reveals
nothing about which keys exist."""
if token == API_KEY_PLACEHOLDER:
return (
"This is the placeholder key from the example. Create an API key in "
f"Unsloth Studio under Settings > API and use it in place of {API_KEY_PLACEHOLDER}."
)
return "Invalid or expired API key"
async def _get_current_subject(
credentials: HTTPAuthorizationCredentials, *, allow_password_change: bool
) -> str:
@ -176,7 +192,7 @@ async def _get_current_subject(
if username is None:
raise HTTPException(
status_code = status.HTTP_401_UNAUTHORIZED,
detail = "Invalid or expired API key",
detail = _invalid_api_key_detail(token),
)
return username

127
studio/backend/core/inference/api_monitor.py
View file

@ -52,6 +52,13 @@ class ApiMonitorEntry:
total_tokens: Optional[int] = None
total_tokens_authoritative: bool = False
error: Optional[str] = None
# "request" (HTTP call) or "lifecycle" (model load/unload: event/reason, not a prompt; shared).
kind: str = "request"
event: Optional[str] = None
reason: Optional[str] = None
shared: bool = False
# 0-100 for a running download row; None when not applicable.
progress: Optional[float] = None
def snapshot(self, *, include_details: bool = True) -> dict[str, Any]:
duration_ms = None
@ -85,6 +92,10 @@ class ApiMonitorEntry:
"completion_tokens": self.completion_tokens,
"total_tokens": self.total_tokens,
"error": self.error,
"kind": self.kind,
"event": self.event,
"reason": self.reason,
"progress": self.progress,
}
if include_details:
payload["prompt"] = self.prompt
@ -127,6 +138,75 @@ class ApiMonitor:
self._trim_terminal_locked()
return entry.id
def record_lifecycle(
self,
*,
event: str,
model: str,
reason: Optional[str] = None,
running: bool = False,
) -> str:
"""Record a model load/unload alongside the request traffic that caused it.
``running=True`` opens the row (a load in progress) and the caller closes
it with the usual :meth:`finish` / :meth:`fail`; an unload is terminal on
arrival. Rows are shared, so every subject sees them, and share the same
retention budget as requests.
"""
now = time.time()
entry = ApiMonitorEntry(
id = f"apievt_{uuid.uuid4().hex[:12]}",
endpoint = f"model.{event}",
method = "",
model = model or "default",
prompt = "",
status = "running" if running else "completed",
started_at = now,
updated_at = now,
started_monotonic = time.monotonic(),
finished_at = None if running else now,
finished_monotonic = None if running else time.monotonic(),
kind = "lifecycle",
event = event,
reason = reason,
shared = True,
)
with self._lock:
self._entries.appendleft(entry)
self._trim_terminal_locked()
return entry.id
def relabel(self, entry_id: Optional[str], model: str) -> None:
"""Rename an open lifecycle row once the load resolves its real id (the
caller only has the load path up front, which may be an HF snapshot dir)."""
if not entry_id or not model:
return
with self._lock:
entry = self._find_locked(entry_id)
if entry is not None:
entry.model = model
entry.updated_at = time.time()
def set_progress(self, entry_id: Optional[str], progress: Optional[float]) -> None:
"""Update an open download row's percentage (clamped to 0-100)."""
if not entry_id or progress is None:
return
with self._lock:
entry = self._find_locked(entry_id)
if entry is not None and entry.status == "running":
entry.progress = min(100.0, max(0.0, float(progress)))
entry.updated_at = time.time()
def discard(self, entry_id: Optional[str]) -> None:
"""Drop a row that turned out not to be an event (a load that was already
satisfied, so nothing was actually loaded)."""
if not entry_id:
return
with self._lock:
entry = self._find_locked(entry_id)
if entry is not None:
self._entries.remove(entry)
def append_reply(self, entry_id: Optional[str], text: str) -> None:
if not entry_id or not text:
return
@ -212,6 +292,19 @@ class ApiMonitor:
self._entries.appendleft(entry)
self._trim_terminal_locked()
def fail_open(self, entry_id: Optional[str], error: str) -> None:
"""Fail only a still-open row. Unlike :meth:`fail` this never touches an
entry that already finished, so a catch-all in a ``finally`` cannot stamp
an error onto a request that in fact succeeded."""
if not entry_id:
return
with self._lock:
entry = self._find_locked(entry_id)
if entry is None or entry.finished_at is not None:
return
# Same lock as the check, so a finish() cannot land in between.
self._fail_locked(entry, error)
def fail(self, entry_id: Optional[str], error: str) -> None:
if not entry_id:
return
@ -224,15 +317,18 @@ class ApiMonitor:
if error:
entry.error = _trim(error, 1000)
return
now = time.time()
entry.status = "error"
entry.error = _trim(error, 1000)
entry.updated_at = now
entry.finished_at = now
entry.finished_monotonic = time.monotonic()
self._entries.remove(entry)
self._entries.appendleft(entry)
self._trim_terminal_locked()
self._fail_locked(entry, error)
def _fail_locked(self, entry: ApiMonitorEntry, error: str) -> None:
now = time.time()
entry.status = "error"
entry.error = _trim(error, 1000)
entry.updated_at = now
entry.finished_at = now
entry.finished_monotonic = time.monotonic()
self._entries.remove(entry)
self._entries.appendleft(entry)
self._trim_terminal_locked()
def snapshot(
self,
@ -244,7 +340,7 @@ class ApiMonitor:
return [
entry.snapshot(include_details = include_details)
for entry in self._entries
if subject is None or entry.subject == subject
if self._visible(entry, subject)
]
def get(
@ -257,22 +353,29 @@ class ApiMonitor:
entry = self._find_locked(entry_id)
if entry is None:
return None
if subject is not None and entry.subject != subject:
if not self._visible(entry, subject):
return None
return entry.snapshot(include_details = True)
def active_count(self, *, subject: Optional[str] = None) -> int:
# Lifecycle rows show as "running" while loading but are not in-flight API requests.
with self._lock:
return sum(
1
for entry in self._entries
if entry.status == "running" and (subject is None or entry.subject == subject)
if entry.status == "running"
and entry.kind != "lifecycle"
and (subject is None or entry.subject == subject)
)
def clear(self) -> None:
with self._lock:
self._entries.clear()
@staticmethod
def _visible(entry: ApiMonitorEntry, subject: Optional[str]) -> bool:
return subject is None or entry.subject == subject or entry.shared
def _find_locked(self, entry_id: str) -> Optional[ApiMonitorEntry]:
for entry in self._entries:
if entry.id == entry_id:

18
studio/backend/core/inference/llama_keepwarm.py
View file

@ -345,6 +345,22 @@ def _loaded_identity(backend):
return (backend.model_identifier, getattr(backend, "hf_variant", None), advertised)
def _note_idle_unload_event(freed) -> None:
"""Record an idle auto-unload in the API monitor, using the advertised repo id
from the stash so the row never shows the on-disk load path. Best-effort."""
try:
from core.inference.api_monitor import api_monitor
from core.inference.model_ids import public_model_id
identifier, variant, advertised = (list(freed) + [None, None, None])[:3]
label = public_model_id(advertised or identifier) or "model"
if variant and ":" not in label:
label = f"{label}:{variant}"
api_monitor.record_lifecycle(event = "unload", model = label, reason = "idle")
except Exception as exc:
logger.debug("idle unload monitor event failed: %s", exc)
async def idle_unload_loop(poll_seconds: float = 15.0) -> None:
"""Unload the loaded GGUF once idle past the configured TTL. Inert when off."""
from utils.openai_auto_switch_settings import (
@ -407,6 +423,8 @@ async def idle_unload_loop(poll_seconds: float = 15.0) -> None:
elif manifest:
_delete_resume_files(manifest)
logger.info("Idle auto-unload: freed GGUF after %ss idle", ttl)
# An idle unload stashes for reload and skips note_model_unloaded.
_note_idle_unload_event(freed)
seen_model = None
except Exception as exc:
logger.debug("idle_unload_loop iteration failed: %s", exc)

175
studio/backend/core/inference/local_model_resolver.py
View file

@ -34,6 +34,16 @@ class _LocalGgufEntry:
_CACHE_TTL_S = 5.0
_lock = threading.Lock()
_scan: tuple[float, dict[str, _LocalGgufEntry]] = (0.0, {})
# Not _lock: that is held for the whole scan, so the request path would wait on it.
_warm_lock = threading.Lock()
# Repos that finished downloading but are not in the published index yet. The
# retained index covers what was already known; nothing covers the one that just
# landed until the next scan, and the request path must not call it absent.
_just_downloaded: set[str] = set()
_warming = False
_last_scan_s = 0.0
# Rescan at most a tenth of the time: on the TTL alone a slow scan would run continuously.
_WARM_DUTY = 10.0
def _is_abs_path_id(value: str) -> bool:
@ -103,17 +113,28 @@ def _local_gguf_entry(loader_id: str, info) -> Optional[_LocalGgufEntry]:
load_dir = _resolve_load_dir(p)
variants, _ = list_local_gguf_variants(str(load_dir))
quants = tuple(v.quant for v in variants if getattr(v, "quant", None))
return _LocalGgufEntry(loader_id, str(load_dir), quants) if quants else None
if not quants:
return None
# That call orders by descending size, so the head is the biggest quant,
# often F16. A bare id means whichever quant a plain load would take, so put
# that first: everything downstream reads [0], and answering with the
# largest can evict a working model and then OOM starting it.
from core.inference.openai_auto_download import preferred_quant
best = preferred_quant(quants)
if best and quants[0] != best:
quants = (best, *(q for q in quants if q != best))
return _LocalGgufEntry(loader_id, str(load_dir), quants)
except Exception:
return None
def info_has_local_gguf(info) -> bool:
"""True when *info* (a LocalModelInfo) points to on-disk GGUF weights the
auto-switch path can load. Read from the files, not ``info.model_format``: the
HF-cache scanner leaves model_format unset for GGUF snapshots, so a
model_format filter would drop every cached GGUF. Lets /v1/models advertise
exactly what /v1 can serve."""
def local_gguf_quants(info) -> Optional[tuple[str, ...]]:
"""On-disk quant labels for *info*, or None when it is not a servable local
GGUF. Read from the files, not ``info.model_format``: the HF-cache scanner
leaves model_format unset for GGUF snapshots, so a model_format filter would
drop every cached GGUF. Lets /v1/models advertise exactly what /v1 can serve,
and which quant to name, from a single scan."""
from pathlib import Path
path = getattr(info, "path", None)
@ -123,8 +144,14 @@ def info_has_local_gguf(info) -> bool:
if isinstance(path, str) and any(
seg in (".studio_links", "ollama_links") for seg in Path(path).parts
):
return False
return _local_gguf_entry(getattr(info, "id", "") or "", info) is not None
return None
entry = _local_gguf_entry(getattr(info, "id", "") or "", info)
return entry.variants if entry is not None else None
def info_has_local_gguf(info) -> bool:
"""True when *info* points to on-disk GGUF weights the auto-switch path can load."""
return local_gguf_quants(info) is not None
def _build_index() -> dict[str, _LocalGgufEntry]:
@ -287,6 +314,36 @@ def _sibling_revision_entries(raw_id: str, loader_id: str):
yield sibling.name, entry
def note_downloaded(repo_id: Optional[str]) -> None:
"""Record a repo as present ahead of the scan that will index it."""
if not repo_id:
return
with _lock:
_just_downloaded.add(repo_id.strip().lower())
def recently_downloaded(repo_id: str) -> bool:
"""Whether *repo_id* finished downloading since the last completed scan."""
if not isinstance(repo_id, str) or not repo_id.strip():
return False
return repo_id.strip().lower() in _just_downloaded
def invalidate_index() -> None:
"""Mark the cached scan stale so the next resolve sees a just-finished
download, rather than waiting out the TTL.
Keeps the entries. Callers on the request path read this cache without
scanning, so emptying it would leave them with no evidence about any local
model until the rebuild lands, and a bare request for one of them would be
answered by whatever is resident. Only a completed download invalidates, and
that only ever adds models, so the retained entries stay true.
"""
global _scan
with _lock:
_scan = (0.0, _scan[1])
def _index() -> dict[str, _LocalGgufEntry]:
global _scan
# Build under the lock so concurrent callers with an expired cache don't all
@ -301,23 +358,78 @@ def _index() -> dict[str, _LocalGgufEntry]:
# an install with many local models can itself exceed the TTL, which would
# store the cache already expired and make every request rebuild the index.
_scan = (time.monotonic(), fresh)
# The scan supersedes the notes: whatever landed is in the index now.
_just_downloaded.clear()
return fresh
def resolve_local_gguf(requested: str) -> Optional[tuple[str, Optional[str], str]]:
def index_is_built() -> bool:
"""Whether a scan has ever completed, freshness aside.
Lock-free on purpose: ``_lock`` is held for the whole scan, so taking it here
would park the request path on the very scan it is trying to stay off. Reading
``_scan[0]`` is safe because ``_scan`` is only ever rebound, never mutated.
"""
return bool(_scan[0])
def warm_index_soon() -> None:
"""(Re)build the index off the request path when it is missing or past its TTL.
Callers that cannot afford the scan use this plus ``allow_scan=False``, so this
is the only thing that ever refreshes the index for them. It has to cover a
stale index and not just an absent one: a model downloaded through the Hub UI
or dropped into a scan folder has no invalidation hook, and would otherwise stay
invisible to those callers for the life of the process.
Never touches ``_lock``, which the scan holds throughout, and never blocks.
"""
global _warming
if time.monotonic() - _scan[0] < max(_CACHE_TTL_S, _last_scan_s * _WARM_DUTY):
return
with _warm_lock:
if _warming:
return
_warming = True
def _run() -> None:
global _warming, _last_scan_s
started = time.monotonic()
try:
_index()
except Exception:
pass
finally:
_last_scan_s = time.monotonic() - started
with _warm_lock:
_warming = False
threading.Thread(target = _run, name = "local-model-index-warm", daemon = True).start()
def resolve_local_gguf(
requested: str, *, allow_scan: bool = True
) -> Optional[tuple[str, Optional[str], str]]:
"""Return ``(load_path, gguf_variant, loader_id)`` for a local match, else None.
``load_path`` is the concrete on-disk path to hand /load (so it never fetches
a remote), ``loader_id`` is the advertised id used as the launch-override key.
``requested`` is ``repo`` or ``repo:VARIANT``. An exact id match wins first
(so ids containing a colon still resolve); else the last ``:VARIANT`` is split
off and resolves only when that quant is on disk.
off and resolves only when that quant is on disk, unless it names no quant at
all (an Ollama-style ":latest"), which means the repo.
``allow_scan=False`` answers from the last built index and never rebuilds,
for callers on the request path: the scan walks several model dirs and HF
caches, takes seconds on a large install, and holds a lock every other
caller queues behind. A stale answer is fine there, since what is on disk
barely moves and a finished download calls :func:`invalidate_index`.
"""
if not isinstance(requested, str) or not requested.strip():
return None
requested = requested.strip()
try:
index = _index()
index = _index() if allow_scan else _scan[1]
entry = index.get(requested.lower())
if entry is not None:
variant = entry.variants[0] if entry.variants else None
@ -333,8 +445,45 @@ def resolve_local_gguf(requested: str) -> Optional[tuple[str, Optional[str], str
for v in entry.variants:
if v.lower() == wanted:
return entry.load_path, v, entry.loader_id
return None
from core.inference.openai_auto_download import looks_like_quant
if looks_like_quant(variant):
return None
# ":latest" or ":8b" names no file, so it means the repo; a real quant that
# is not on disk still misses, or a swap would serve the wrong weights.
return entry.load_path, (entry.variants[0] if entry.variants else None), entry.loader_id
except Exception:
# Best-effort: any resolver failure falls through to the loaded model,
# so a malformed name can never turn a servable request into a 500.
return None
MISS_MODEL_NOT_FOUND = "model_not_found"
MISS_VARIANT_NOT_FOUND = "variant_not_found"
def describe_local_miss(requested: str) -> tuple[str, tuple[str, ...]]:
"""Why :func:`resolve_local_gguf` missed, so an error can say "wrong quant"
instead of "no such model".
``(MISS_VARIANT_NOT_FOUND, <local quants>)`` when the repo is downloaded but
the requested ``:VARIANT`` is not, else ``(MISS_MODEL_NOT_FOUND, ())``. Splits
the name like the resolver so the two agree. Fail-safe: a scan failure reports
the generic miss rather than raising into the handler.
"""
if not isinstance(requested, str) or not requested.strip():
return MISS_MODEL_NOT_FOUND, ()
base, sep, variant = requested.strip().rpartition(":")
from core.inference.openai_auto_download import looks_like_quant
# Split like the resolver or the two disagree: a tag naming no quant means the
# repo there, so reporting a missing quant for it would name one nobody asked for.
if not sep or not looks_like_quant(variant):
return MISS_MODEL_NOT_FOUND, ()
try:
entry = _index().get(base.strip().lower())
except Exception:
return MISS_MODEL_NOT_FOUND, ()
if entry is None or not entry.variants:
return MISS_MODEL_NOT_FOUND, ()
return MISS_VARIANT_NOT_FOUND, entry.variants

25
studio/backend/core/inference/model_ids.py
View file

@ -39,10 +39,30 @@ def _looks_like_path(identifier: str) -> bool:
return False
def hf_cache_repo_id(path: Optional[str]) -> Optional[str]:
"""``.../models--org--name/snapshots/<sha>`` -> ``org/name``, else None.
A model loaded straight out of the HF cache has a snapshot directory as its
identifier, whose basename is a commit hash. Recover the repo id so callers
show ``unsloth/gemma-4-31B-it-GGUF`` rather than ``c1ac76e99d55...``.
"""
if not path:
return None
parts = str(path).replace("\\", "/").split("/")
for index, part in enumerate(parts):
# Only inside the real cache layout: a "models--" name alone is not a repo id.
if part.startswith("models--") and parts[index + 1 : index + 2] == ["snapshots"]:
return part[len("models--") :].replace("--", "/")
return None
def public_model_id(identifier: Optional[str]) -> Optional[str]:
"""Return a clean, path-free public id for *identifier*.
- Local GGUF path -> the file stem with ``.gguf`` stripped, e.g.
- HF cache path -> the repo id it came from, e.g.
``~/.cache/huggingface/hub/models--unsloth--X-GGUF/snapshots/<sha>`` ->
``unsloth/X-GGUF``.
- Other local GGUF path -> the file stem with ``.gguf`` stripped, e.g.
``/srv/models/Qwen3-30B-A3B-Q4_K_M.gguf`` -> ``Qwen3-30B-A3B-Q4_K_M``.
- HF repo id (``org/model``) and already-clean names -> returned unchanged.
- ``None`` / empty -> returned unchanged.
@ -51,6 +71,9 @@ def public_model_id(identifier: Optional[str]) -> Optional[str]:
return identifier
if not _looks_like_path(identifier):
return identifier
repo_id = hf_cache_repo_id(identifier)
if repo_id:
return repo_id
name = os.path.basename(identifier.replace("\\", "/").rstrip("/"))
if name.lower().endswith(_GGUF_SUFFIX):
name = name[: -len(_GGUF_SUFFIX)]

831
studio/backend/core/inference/openai_auto_download.py Normal file
View file

@ -0,0 +1,831 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
"""Opt-in: fetch a GGUF a /v1 request names but this server doesn't have.
Auto-switch only loads models already on disk. With
``openai_api_auto_download_model`` on, a miss that looks like a real Hub repo is
downloaded in the background instead of erroring, and the request is told to
retry rather than being held open: a quant is routinely tens of GB, far longer
than any client (or the Cloudflare edge on ``--secure``) will wait, and the
inference lifecycle gate must not be held meanwhile. The resident model keeps
serving throughout, and the retry that lands after the download is served by the
new model through the ordinary auto-switch path.
Admission is deliberately narrow, since a request only needs an API key:
- ``namespace/name`` only, and only when the Hub confirms it is a GGUF repo.
``gpt-4`` and ``anthropic/claude-3.5-sonnet`` alike fall through to the
resident model as before: a namespace is not evidence of intent, since LiteLLM
and OpenRouter address every provider that way.
- GGUF repos only, decided from the remote file list, not the repo name. GGUF
runs under llama.cpp, which never imports repo Python.
- Anything declaring ``auto_map`` is refused, so ``trust_remote_code`` can only
ever be granted deliberately in the UI, never by an API call.
- One download at a time, so a key holder cannot fan out fetches.
"""
from __future__ import annotations
import asyncio
import shutil
import threading
import time
from dataclasses import dataclass
from typing import Optional
from loggers import get_logger
logger = get_logger(__name__)
# Keep the Hub probe short so a slow Hub can't stall the request path.
_MODEL_INFO_TIMEOUT_S = 8.0
# auth_check and hf_hub_download take no timeout of their own, and both run while the
# provisional slot is held, so an unresponsive Hub would pin the single flight and stall
# the request long past the metadata budget. The code probe fetches up to three small
# configs, so it gets more room than the single auth call.
_CODE_PROBE_TIMEOUT_S = 20.0
# Headroom left free after the download, so filling the disk can't wedge the box.
_DISK_RESERVE_BYTES = 5 * 1024**3
_WATCH_POLL_S = 2.0
# A stalled watcher must not pin the single-flight slot forever.
_MAX_WATCH_S = 24 * 60 * 60
# Past the watch window the row is already resolved, so poll only to see whether
# the worker is still alive and still owns the slot.
_TIMED_OUT_POLL_S = 60.0
_RETRY_AFTER_S = 30
# Long enough for a client honouring Retry-After to come back and be told, short
# enough that a client that never returns cannot hold the slot.
_FAILED_HOLD_S = 3 * _RETRY_AFTER_S
_MAX_LISTED_VARIANTS = 8
@dataclass(frozen = True)
class AutoDownloadRefusal:
"""Why this request cannot be served yet. The route turns it into an
HTTPException with the surface's own error envelope."""
status: int
code: str
message: str
retry_after: Optional[int] = None
@dataclass
class _Active:
repo_id: str
# None while the Hub probe is still deciding which quant to fetch.
variant: Optional[str] = None
expected_bytes: int = 0
monitor_id: Optional[str] = None
started_at: float = 0.0
# Set when the worker failed. The slot is kept until a retry surfaces it, since
# the advertised retry interval is far longer than the watcher's poll and the
# client would otherwise just restart the same failing download.
error: Optional[str] = None
failed_at: float = 0.0
_lock = threading.Lock()
_active: Optional[_Active] = None
# Repos the Hub says are not servable, so a "vendor/model" miss doesn't re-probe every request.
_NOT_SERVABLE_TTL_S = 10 * 60
_NOT_SERVABLE_MAX = 256
_cache_lock = threading.Lock()
_not_servable: dict[str, float] = {}
def _public_label(repo_id: str, variant: Optional[str]) -> str:
return f"{repo_id}:{variant}" if variant else repo_id
def split_model_ref(requested: str) -> tuple[str, Optional[str]]:
"""``org/repo:QUANT`` -> ``("org/repo", "QUANT")``; no suffix -> variant None.
Splits on the last colon. A slash-bearing suffix is only a variant when a real
Hub repo precedes it: an unrecognized GGUF below a subdirectory keys on its path
("build/llama-13b", which is_valid_gguf_variant allows and the catalog advertises),
while "C:/models/x.gguf" leaves a drive letter that is no repo id at all.
"""
text = (requested or "").strip()
base, sep, suffix = text.rpartition(":")
if not sep or not base or not suffix:
return text, None
stripped = base.strip()
if "/" in suffix:
from hub.utils.paths import is_valid_repo_id
if "/" not in stripped or not is_valid_repo_id(stripped):
return text, None
return stripped, suffix.strip()
def is_downloadable_ref(requested: str) -> bool:
"""Whether *requested* is shaped like a Hub repo we may fetch.
Requires an explicit namespace. That keeps ``gpt-4`` and other foreign ids
falling through untouched, and avoids the bare-name ``unsloth/`` prefixing in
ModelConfig.from_identifier turning an unrelated label into a real repo.
"""
from hub.utils.paths import is_valid_repo_id
repo_id, variant = split_model_ref(requested)
if "/" not in repo_id or not is_valid_repo_id(repo_id):
return False
if variant is not None:
from hub.utils.paths import is_valid_gguf_variant
return is_valid_gguf_variant(variant)
return True
def looks_like_quant(variant: Optional[str]) -> bool:
"""Whether a ``:suffix`` names a GGUF quant rather than a foreign tag.
``vendor/model`` is how LiteLLM and OpenRouter address every provider, and
``name:latest`` is how Ollama tags one, so neither a namespace nor a colon
proves a request was meant for this server. A real quant label does.
"""
import re
from utils.models.model_config import _GGUF_KNOWN_QUANT_RE
if not variant:
return False
# _extract_quant_label can append a bpw modifier (IQ4_XS-3.53bpw); still a quant.
label = re.sub(r"-[0-9]+(?:\.[0-9]+)?bpw$", "", variant.strip(), flags = re.IGNORECASE)
return _GGUF_KNOWN_QUANT_RE.fullmatch(label) is not None
def _hub_token(hf_token: Optional[str]):
"""The caller's token, or an explicit False.
None makes huggingface_hub fall back to a cached login, which here would be
the server owner's. False is what actually means anonymous.
"""
return hf_token or False
def _servable_key(repo_id: str, hf_token: Optional[str]) -> str:
"""Cache key, per credential.
The Hub answers 404 for a private repo the caller cannot see, so a verdict
reached without a token says nothing about a caller who has one. Keyed on a
digest so the token itself is never held here.
"""
import hashlib
seen_as = hashlib.sha256(hf_token.encode()).hexdigest()[:16] if hf_token else "anon"
return f"{repo_id.lower()}\n{seen_as}"
def _mark_not_servable(repo_id: str, hf_token: Optional[str]) -> None:
with _cache_lock:
if len(_not_servable) >= _NOT_SERVABLE_MAX:
_not_servable.clear()
_not_servable[_servable_key(repo_id, hf_token)] = time.monotonic() + _NOT_SERVABLE_TTL_S
def _is_not_servable(repo_id: str, hf_token: Optional[str]) -> bool:
key = _servable_key(repo_id, hf_token)
with _cache_lock:
expires = _not_servable.get(key)
if expires is None:
return False
if expires <= time.monotonic():
del _not_servable[key]
return False
return True
def _gated_refusal(repo_id: str) -> AutoDownloadRefusal:
return AutoDownloadRefusal(
status = 403,
code = "model_access_denied",
message = (
f"'{repo_id}' is gated on Hugging Face. Accept its licence, then retry with "
"your own token in the X-Unsloth-HF-Token header: automatic download never "
"uses this server's Hugging Face identity."
),
)
async def _bounded_probe(fn, *args, timeout: float, default):
"""Run a blocking Hub probe off the loop, bounding only the wait.
The thread is left to finish (a blocking socket read cannot be cancelled); the
caller stops waiting and takes *default*, which each call site chooses so that a
timeout errs the safe way.
"""
try:
return await asyncio.wait_for(asyncio.to_thread(fn, *args), timeout)
except (TimeoutError, asyncio.TimeoutError):
logger.debug("hub probe %s timed out after %ss", getattr(fn, "__name__", fn), timeout)
return default
def _auth_denied(repo_id: str, hf_token: Optional[str]) -> bool:
"""Whether this token lacks file access to a gated repo. False when the
check is inconclusive: the download's own auth is the real gate."""
from hub.utils.hf_errors import hf_error_status
try:
from huggingface_hub import auth_check
auth_check(repo_id, token = _hub_token(hf_token))
except Exception as exc:
return hf_error_status(exc) in (401, 403)
return False
def _gguf_variants(siblings) -> dict[str, int]:
"""Quant label -> bytes the download will actually fetch.
Mirrors list_gguf_variants for the selectable labels: companions (mmproj/MTP)
and big-endian builds are not quants of their own, and sharded quants sum
across their shards. The byte total comes from the download plan, which folds
the companions back into every quant, so the disk reserve is measured against
what the worker fetches rather than the main files alone.
"""
from hub.utils.gguf import extract_quant_label as canonical_quant_label
from hub.utils.gguf_plan import build_gguf_variant_plans
from utils.models.model_config import (
_extract_quant_label,
_is_big_endian_gguf_path,
_is_mmproj,
_is_mtp_drafter,
)
siblings = list(siblings or [])
plans = build_gguf_variant_plans(siblings)
sizes: dict[str, int] = {}
for sibling in siblings:
name = getattr(sibling, "rfilename", "") or ""
if not name.lower().endswith(".gguf"):
continue
quant = _extract_quant_label(name)
if not looks_like_quant(quant):
# With no recognized quant token the two extractors part ways: this one
# takes the last hyphenated segment ("7b" of llama-7b) while the plan and
# the worker key the whole stem. Advertising ours dispatches a variant the
# worker cannot resolve, so take theirs for the unrecognized case only.
quant = canonical_quant_label(name) or quant
if _is_mmproj(name) or _is_mtp_drafter(name) or _is_big_endian_gguf_path(name, quant):
continue
plan = plans.get(quant.lower())
if plan is not None:
sizes[quant] = plan.download_size_bytes
else:
sizes[quant] = sizes.get(quant, 0) + int(getattr(sibling, "size", 0) or 0)
return sizes
def _remaining_bytes(repo_id: str, plan, expected_bytes: int) -> int:
"""Bytes still to fetch: a resumed quant or a companion shared with another
quant is already on disk, and charging for it can 507 a download that fits."""
try:
from hub.utils.download_registry import existing_blob_bytes
hashes = frozenset(
file.sha256 for file in getattr(plan, "expected_files", ()) or () if file.sha256
)
if not hashes:
return expected_bytes
return max(0, expected_bytes - existing_blob_bytes("model", repo_id, hashes))
except Exception:
return expected_bytes
def _enough_disk(need_bytes: int) -> tuple[bool, int]:
"""(fits, free_bytes). Fail-open on an unreadable cache root: the download
worker runs its own preflight, this only adds the reserve margin."""
try:
from hub.utils.hf_cache_state import hf_cache_root
root = hf_cache_root(create = True)
if root is None:
return True, 0
free = shutil.disk_usage(root).free
except Exception:
return True, 0
return free >= need_bytes + _DISK_RESERVE_BYTES, free
def _gb(num_bytes: int) -> str:
return f"{num_bytes / 1024**3:.1f} GB"
async def _job_state(repo_id: str, variant: Optional[str]) -> tuple[str, Optional[str]]:
from hub.services.models import downloads
try:
status = await downloads.get_download_status_response(repo_id, variant or "")
return status.state, status.error
except Exception as exc:
# "unknown", not "idle": idle ends the watch, and a failed probe proves nothing.
logger.debug("auto-download: status probe failed for %r: %s", repo_id, exc)
return "unknown", None
async def _progress_percent(
repo_id: str, variant: Optional[str], expected_bytes: int, hf_token: Optional[str]
) -> Optional[float]:
"""0-100, or None. The hub service reports a 0-1 fraction, so scale it."""
from hub.services.models import downloads
try:
payload = await downloads.get_gguf_download_progress_response(
repo_id, variant or "", expected_bytes, hf_token
)
fraction = payload.get("progress")
if not isinstance(fraction, (int, float)):
return None
return min(100.0, max(0.0, float(fraction) * 100.0))
except Exception:
return None
def _release(active: Optional[_Active]) -> None:
"""Free the single-flight slot, but only while *active* still owns it.
Keying the release on ``repo_id`` alone let a stale operation clear a newer
one for the same repo: variant A errors, an adopting request frees the slot,
a retry starts variant B, and A's watcher then matches on the repo and clears
B on its way out -- admitting a second repository download alongside B.
Identity ties every release to the operation that actually took the slot.
"""
global _active
if active is None:
return
with _lock:
if _active is active:
_active = None
async def _watch(active: _Active, hf_token: Optional[str]) -> None:
"""Poll a dispatched job so the monitor row resolves and the resolver cache
is dropped the moment the weights land."""
from core.inference import api_monitor as monitor_module
api_monitor = monitor_module.api_monitor
deadline = time.monotonic() + _MAX_WATCH_S
timed_out = False
try:
while True:
await asyncio.sleep(_TIMED_OUT_POLL_S if timed_out else _WATCH_POLL_S)
state, error = await _job_state(active.repo_id, active.variant)
if state in ("running", "cancelling", "unknown"):
if timed_out:
# A worker still running still owns the slot: releasing it on the
# clock alone would admit a second multi-GB download alongside it.
# "unknown" cannot confirm it is alive, so stop holding it then,
# or a broken probe would wedge auto-download for good.
if state == "unknown":
return
continue
if time.monotonic() >= deadline:
api_monitor.fail_open(active.monitor_id, "Download timed out")
timed_out = True
continue
# Only "running" has progress; the others are still in flight, so keep the slot.
if state == "running":
api_monitor.set_progress(
active.monitor_id,
await _progress_percent(
active.repo_id, active.variant, active.expected_bytes, hf_token
),
)
continue
if state == "cancelled":
api_monitor.finish(active.monitor_id, status = "cancelled")
return
if state == "complete":
# No invalidate here: finalize_worker_exit already dropped the cache and
# started the warm, and a second one would mark that fresh scan stale and
# push a synchronous rescan onto the client's retry.
api_monitor.finish(active.monitor_id, status = "completed")
elif state == "idle":
# The job vanished without a terminal state (worker killed).
api_monitor.fail_open(active.monitor_id, "Download did not complete")
else:
api_monitor.fail_open(active.monitor_id, error or f"Download {state}")
# Keep the slot so the next retry is told it failed rather than
# silently starting the same download again.
active.error = error or f"Download {state}"
active.failed_at = time.monotonic()
return
return
except asyncio.CancelledError:
raise
except Exception as exc:
logger.warning("auto-download: watcher failed for %r: %s", active.repo_id, exc)
api_monitor.fail_open(active.monitor_id, "Download tracking failed")
finally:
if not active.failed_at:
_release(active)
def _downloading_refusal(label: str, percent: Optional[float]) -> AutoDownloadRefusal:
progress = f" ({percent:.0f}% done)" if percent is not None else ""
return AutoDownloadRefusal(
status = 503,
code = "model_downloading",
message = (f"Downloading '{label}'{progress}. Retry shortly. Track it in Unsloth Studio."),
retry_after = _RETRY_AFTER_S,
)
async def _is_downloadable_model(repo_id: str, hf_token: Optional[str]) -> bool:
"""Whether the Hub has this repo with GGUF weights we could fetch.
Only asked while another download holds the slot, to tell a second download
apart from an ordinary foreign label. Any failure answers False: falling
through to the resident model is what such a label does anyway, and refusing
it would strand normal traffic for the length of the download.
"""
if _is_not_servable(repo_id, hf_token):
return False
def _probe():
from huggingface_hub import HfApi
return HfApi(token = _hub_token(hf_token)).model_info(repo_id, timeout = _MODEL_INFO_TIMEOUT_S)
try:
info = await asyncio.to_thread(_probe)
except Exception:
return False
# The same filter admission uses, not a bare .gguf test: mmproj, MTP drafters and
# big-endian builds are companions rather than quants, so a repo holding only those
# is not downloadable here either. Answering otherwise would hold an ordinary
# foreign label at model_download_busy for the length of an unrelated download.
servable = bool(_gguf_variants(getattr(info, "siblings", None)))
if not servable:
_mark_not_servable(repo_id, hf_token)
return servable
async def maybe_auto_download(
requested_model: str,
*,
hf_token: Optional[str] = None,
require_vision: bool = False,
) -> Optional[AutoDownloadRefusal]:
"""Start (or report on) a background fetch of *requested_model*.
Returns None when the request should carry on unchanged, or a refusal the
caller must raise. Only called after the local resolver has already missed.
``require_vision`` refuses a target with no mmproj companion rather than
spending gigabytes on weights that cannot answer the request that asked for
them; the local capability guard only ever sees an already-downloaded model.
"""
global _active
repo_id, wanted_variant = split_model_ref(requested_model)
if not is_downloadable_ref(requested_model):
return None
if _is_not_servable(repo_id, hf_token) and not looks_like_quant(wanted_variant):
return None
# Settle the single-flight slot before the network, so retries during a download stay cheap.
busy: Optional[_Active] = None
with _lock:
current = _active
if current is not None and current.failed_at:
# A held failure only owns the slot until someone is told about it.
if current.repo_id != repo_id and time.monotonic() - current.failed_at > _FAILED_HOLD_S:
_active = current = None
if current is not None and current.repo_id == repo_id:
adopted = current
elif current is not None:
adopted = None
busy = current
else:
adopted = None
provisional = _Active(repo_id = repo_id, started_at = time.time())
_active = provisional
if busy is not None:
# Refusing before the probe blocks ordinary drop-in traffic: a namespaced label
# that is not a downloadable GGUF repo (LiteLLM/OpenRouter style) would be told
# to wait out a multi-hour download instead of falling through to the resident
# model. Only a label that could itself be downloaded is a second download.
if not await _is_downloadable_model(repo_id, hf_token):
return None
return AutoDownloadRefusal(
status = 503,
code = "model_download_busy",
message = (
f"Already downloading '{_public_label(busy.repo_id, busy.variant)}'. "
f"Retry '{requested_model}' once it finishes."
),
retry_after = _RETRY_AFTER_S,
)
if adopted is not None:
if adopted.variant is None:
# Still probing: no job yet, and a stale whole-repo error would free the probe's slot.
return _downloading_refusal(adopted.repo_id, None)
state, error = await _job_state(adopted.repo_id, adopted.variant)
if state in ("running", "cancelling", "unknown"):
return _downloading_refusal(
_public_label(adopted.repo_id, adopted.variant),
await _progress_percent(
adopted.repo_id, adopted.variant, adopted.expected_bytes, hf_token
),
)
if state == "error" or adopted.error:
error = error or adopted.error
# Surface once, then free the slot so a retry can start over.
_release(adopted)
return AutoDownloadRefusal(
status = 502,
code = "model_download_failed",
message = f"Downloading '{requested_model}' failed: {error or 'unknown error'}",
)
# complete/idle/cancelled: the watcher is about to free the slot, so retry once more.
return _downloading_refusal(
_public_label(adopted.repo_id, adopted.variant),
100.0 if state == "complete" else None,
)
try:
return await _admit_and_start(
repo_id, wanted_variant, requested_model, hf_token, provisional, require_vision
)
except BaseException:
# Not `except Exception`: a cancel mid-probe would otherwise wedge the provisional slot.
_release(provisional)
raise
async def _admit_and_start(
repo_id: str,
wanted_variant: Optional[str],
requested_model: str,
hf_token: Optional[str],
active: _Active,
require_vision: bool = False,
) -> Optional[AutoDownloadRefusal]:
from hub.utils.hf_errors import hf_error_status
def _probe():
from huggingface_hub import HfApi
return HfApi(token = _hub_token(hf_token)).model_info(
repo_id, files_metadata = True, timeout = _MODEL_INFO_TIMEOUT_S
)
try:
info = await asyncio.to_thread(_probe)
except Exception as exc:
_release(active)
status = hf_error_status(exc)
if status == 401:
return AutoDownloadRefusal(
status = 401,
code = "model_access_denied",
message = (
f"Hugging Face rejected the token sent for '{repo_id}'. Replace the "
"X-Unsloth-HF-Token header with a valid token; retrying will not help."
),
)
if status == 403:
return _gated_refusal(repo_id)
if status == 404:
_mark_not_servable(repo_id, hf_token)
# Unknown to the Hub reads as a foreign label; only an explicit quant makes it ours.
if not looks_like_quant(wanted_variant):
return None
# A private repo reads as absent without a token; don't confirm either way.
return AutoDownloadRefusal(
status = 404,
code = "model_not_found",
message = (
f"'{repo_id}' was not found on Hugging Face, or is not accessible. "
"If it is private, send a token in the X-Unsloth-HF-Token header."
),
)
logger.warning("auto-download: Hub lookup failed for %r: %s", repo_id, exc)
return AutoDownloadRefusal(
status = 503,
code = "model_lookup_failed",
message = f"Could not reach Hugging Face to look up '{repo_id}'. Retry shortly.",
retry_after = _RETRY_AFTER_S,
)
# Inconclusive on timeout: the download's own auth is the real gate.
if getattr(info, "gated", False) and await _bounded_probe(
_auth_denied, repo_id, hf_token, timeout = _MODEL_INFO_TIMEOUT_S, default = False
):
# Metadata for a gated repo is not file access; unchecked, the config read below lies.
_release(active)
return _gated_refusal(repo_id)
variants = _gguf_variants(getattr(info, "siblings", None))
if not variants:
_release(active)
_mark_not_servable(repo_id, hf_token)
if not looks_like_quant(wanted_variant):
return None
return AutoDownloadRefusal(
status = 400,
code = "model_not_supported",
message = (
f"'{repo_id}' has no GGUF weights. Automatic download serves GGUF only; "
"load other formats from Unsloth Studio."
),
)
# trust_remote_code gate: _config_has_auto_map is tri-state, so refuse on True and on None.
from utils.security.consent import _config_has_auto_map
# _hub_token, not the raw token: None lets huggingface_hub fall back to a cached
# server login, so a caller-named repo would be probed with this server's identity.
# Same rule as the metadata probe and the worker.
# None on timeout, which refuses: an unchecked repo is not a cleared one.
has_auto_map = await _bounded_probe(
_config_has_auto_map,
repo_id,
_hub_token(hf_token),
timeout = _CODE_PROBE_TIMEOUT_S,
default = None,
)
if has_auto_map is not False:
_release(active)
unknown = has_auto_map is None
return AutoDownloadRefusal(
status = 403,
code = "remote_code_consent_required",
message = (
f"'{repo_id}' "
+ (
"could not be checked for custom code"
if unknown
else "ships custom code that runs on load"
)
+ ". Load it once in Unsloth Studio to review and approve it, then retry."
),
)
variant = _match_variant(wanted_variant, variants)
if variant is None:
_release(active)
listed = sorted(variants)
shown = ", ".join(listed[:_MAX_LISTED_VARIANTS])
extra = len(listed) - _MAX_LISTED_VARIANTS
return AutoDownloadRefusal(
status = 404,
code = "model_not_found",
message = (
f"'{repo_id}' has no quant '{wanted_variant}'. Available quants: "
f"{shown}{f' and {extra} more' if extra > 0 else ''}."
),
)
expected_bytes = variants[variant]
from hub.utils.gguf_plan import build_gguf_variant_plans
plan = build_gguf_variant_plans(list(getattr(info, "siblings", None) or [])).get(
variant.lower()
)
if require_vision and not (plan and plan.mmproj_filenames):
_release(active)
return AutoDownloadRefusal(
status = 400,
code = "invalid_value",
message = (
f"'{_public_label(repo_id, variant)}' ships no mmproj companion, so it "
"cannot answer the image or audio input in this request. It was not "
"downloaded."
),
)
need_bytes = _remaining_bytes(repo_id, plan, expected_bytes)
fits, free = _enough_disk(need_bytes)
if not fits:
_release(active)
return AutoDownloadRefusal(
status = 507,
code = "insufficient_disk_space",
message = (
f"'{_public_label(repo_id, variant)}' needs {_gb(need_bytes)} plus "
f"{_gb(_DISK_RESERVE_BYTES)} headroom, but only {_gb(free)} is free."
),
)
return await _dispatch(repo_id, variant, expected_bytes, requested_model, hf_token, active)
def preferred_quant(labels) -> Optional[str]:
"""The quant a plain load would pick from *labels*, or None.
The one ranking for "which quant did they mean": local resolution, remote
admission and what /v1/models advertises all have to agree, or a bare id
means a different quant depending on which of them answered it.
"""
from utils.models.model_config import _pick_best_gguf
# _pick_best_gguf ranks filenames and matches upper-case tokens, so feed "<LABEL>.gguf".
synthetic: dict[str, str] = {}
for name in labels:
synthetic.setdefault(f"{name.upper()}.gguf", name)
best = _pick_best_gguf(list(synthetic))
return synthetic.get(best) if best else None
def _match_variant(wanted: Optional[str], variants: dict[str, int]) -> Optional[str]:
"""Resolve the requested quant against what the repo actually has.
An explicit quant matches case-insensitively and must exist: never quietly
substitute another, unlike the loader's low-disk fallback. A bare repo id, or
an Ollama-style tag that names no quant at all (":latest", ":8b"), uses the
same preference order as a manual load, matching what the local resolver does
with the same tag.
"""
if wanted:
# Exact first, whatever shape it is: a repo of generically named GGUFs has
# real variants like "llama-13b" that are valid worker keys but do not look
# like quants, and defaulting past one would fetch a model nobody asked for.
lowered = {name.lower(): name for name in variants}
exact = lowered.get(wanted.strip().lower())
if exact is not None or looks_like_quant(wanted):
# A quant-shaped suffix that matches nothing is a miss, never a swap.
return exact
return preferred_quant(variants)
async def _dispatch(
repo_id: str,
variant: str,
expected_bytes: int,
requested_model: str,
hf_token: Optional[str],
active: _Active,
) -> AutoDownloadRefusal:
global _active
from core.inference.api_monitor import api_monitor
from hub.schemas.downloads import DownloadModelRequest
from hub.services.models import downloads
label = _public_label(repo_id, variant)
busy = AutoDownloadRefusal(
status = 503,
code = "model_download_busy",
message = f"'{repo_id}' is already being downloaded or loaded. Retry shortly.",
retry_after = _RETRY_AFTER_S,
)
try:
dispatched = await downloads.download_model_response(
DownloadModelRequest(repo_id = repo_id, gguf_variant = variant),
hf_token,
allow_ambient_token = False,
)
except Exception as exc:
_release(active)
status = getattr(exc, "status_code", None)
if status == 409:
# A manual load or hub download already owns this repo.
return busy
logger.warning("auto-download: could not start %r: %s", label, exc)
return AutoDownloadRefusal(
status = 502,
code = "model_download_failed",
message = f"Could not start downloading '{requested_model}'.",
)
# accepted=False means no worker launched, so report the conflict instead of taking the slot.
if isinstance(dispatched, dict) and not dispatched.get("accepted", True):
_release(active)
logger.info("auto-download: dispatch refused for %s (%s)", label, dispatched.get("state"))
return busy
monitor_id = api_monitor.record_lifecycle(
event = "download", model = label, reason = "api", running = True
)
with _lock:
if _active is active:
active.variant = variant
active.expected_bytes = expected_bytes
active.monitor_id = monitor_id
tracked = active
else:
# Released underneath us: track the job we started, but never stomp a newer owner.
tracked = _Active(repo_id, variant, expected_bytes, monitor_id, time.time())
if _active is None:
_active = tracked
asyncio.create_task(_watch(tracked, hf_token))
logger.info("auto-download: started %s (%s)", label, _gb(expected_bytes))
return AutoDownloadRefusal(
status = 503,
code = "model_downloading",
message = (
f"Downloading '{label}' ({_gb(expected_bytes)}). Retry shortly. "
"Track it in Unsloth Studio."
),
retry_after = _RETRY_AFTER_S,
)
def reset_for_tests() -> None:
global _active
with _lock:
_active = None
with _cache_lock:
_not_servable.clear()

27
studio/backend/hub/services/download_lifecycle.py
View file

@ -58,6 +58,7 @@ def spawn_worker(
use_xet: bool,
protected_blob_hashes: Optional[frozenset[str]] = None,
cache_env: Optional[Mapping[str, str]] = None,
allow_ambient_token: bool = True,
) -> subprocess.Popen:
"""Spawn the download worker.
@ -83,7 +84,8 @@ def spawn_worker(
env["HF_HUB_DISABLE_XET"] = "0" if use_xet else "1"
# No token in Unsloth settings: fall back to the backend's own HF_TOKEN so
# private repos stay downloadable (needed while inkling repos are private).
if not hf_token:
# Not for a repo an API caller named: that would lend them the owner's identity.
if not hf_token and allow_ambient_token:
hf_token = os.environ.get("HF_TOKEN") or None
env["HF_HUB_DISABLE_IMPLICIT_TOKEN"] = "0" if hf_token else "1"
# hf_transfer's parallel Range chunks can leave sparse partials even in
@ -239,13 +241,32 @@ def finalize_worker_exit(
state = classify_exit(rc, cancel_requested = cancel_requested)
if state == "complete":
registry.set_job(key, "complete")
# Where /v1 learns a new model exists: its resolver answers the request path
# from a cached scan with no watcher, and would otherwise report the model
# absent and let the request be served by whatever is resident. Models only,
# since datasets share this path and noting one as a local model would refuse
# a bare request naming that id instead of letting a foreign id fall through.
if repo_type == "model":
try:
from core.inference.local_model_resolver import (
invalidate_index,
note_downloaded,
warm_index_soon,
)
note_downloaded(repo_id)
invalidate_index()
# Rebuild here rather than on the first request that needs it, so the
# new model resolves without a scan on the request path.
warm_index_soon()
except Exception:
pass
if transport == download_registry.TRANSPORT_HTTP:
registry.update_job_transport(key, download_registry.TRANSPORT_HTTP)
if stderr_text:
if download_manifest.MANIFEST_DEGRADED_MARKER in stderr_text:
logger.warning(
f"{log_prefix} complete with degraded diagnostics for "
f"{label}: {stderr_text}"
f"{log_prefix} complete with degraded diagnostics for {label}: {stderr_text}"
)
else:
logger.info(f"{log_prefix} worker diagnostics for {label}: {stderr_text}")

16
studio/backend/hub/services/models/downloads.py
View file

@ -91,6 +91,7 @@ def _spawn_download_worker(
use_xet: bool = True,
protected_blob_hashes: Optional[frozenset[str]] = None,
cache_env: Optional[dict[str, str]] = None,
allow_ambient_token: bool = True,
) -> subprocess.Popen:
args = ["--repo-id", repo_id]
if variant:
@ -101,11 +102,21 @@ def _spawn_download_worker(
use_xet = use_xet,
protected_blob_hashes = protected_blob_hashes,
cache_env = cache_env,
allow_ambient_token = allow_ambient_token,
)
async def download_model_response(body: DownloadModelRequest, hf_token: Optional[str] = None):
"""Start a background download for a HuggingFace model."""
async def download_model_response(
body: DownloadModelRequest,
hf_token: Optional[str] = None,
*,
allow_ambient_token: bool = True,
):
"""Start a background download for a HuggingFace model.
``allow_ambient_token=False`` keeps the worker anonymous when the caller
supplied no token, for repos named over the API rather than chosen here.
"""
repo_id = body.repo_id.strip()
if not _is_valid_repo_id(repo_id):
raise HTTPException(
@ -218,6 +229,7 @@ async def download_model_response(body: DownloadModelRequest, hf_token: Optional
use_xet = use_xet,
protected_blob_hashes = protected_blob_hashes,
cache_env = cache_env,
allow_ambient_token = allow_ambient_token,
),
hf_token = hf_token,
label = label,

693
studio/backend/routes/inference.py
View file

@ -28,7 +28,7 @@ import re as _re
# Model size extraction (shared with core/inference/llama_cpp.py)
from utils.models import extract_model_size_b as _extract_model_size_b
from utils.api_errors import openai_error_body, anthropic_error_body
from utils.api_errors import openai_error_body, anthropic_error_body, error_body_for_path
from utils.upload_limits import STT_AUDIO_B64_MAX_CHARS, STT_AUDIO_RAW_MAX_BYTES
from hub.dependencies import get_hf_token
from core.inference.orchestrator import GenStreamError, GenStreamErrorRaised
@ -3107,12 +3107,36 @@ def _monitor_context_length() -> Optional[int]:
return None
def _lifecycle_model_label(model: Optional[str], variant: Optional[str] = None) -> str:
"""A path-free ``repo`` / ``repo:QUANT`` label for a monitor lifecycle row."""
clean = public_model_id(model) or model or "model"
return f"{clean}:{variant}" if variant and ":" not in clean else clean
def _close_load_event(
entry_id: Optional[str], model: Optional[str], variant: Optional[str]
) -> None:
"""Close a monitor load row, relabelled with the id the load actually resolved
(the row opened on the request's model_path, which may be an HF snapshot dir)."""
api_monitor.relabel(entry_id, _lifecycle_model_label(model, variant))
api_monitor.finish(entry_id)
def _monitor_active_model() -> Optional[str]:
"""The loaded model as a client-facing id, quant included when known.
Cleaned like /v1/models: this is rendered in the settings UI and served over
the public --secure tunnel, so it must never be the on-disk load path.
"""
llama_backend = get_llama_cpp_backend()
if getattr(llama_backend, "is_loaded", False):
return getattr(llama_backend, "model_identifier", None)
model_id = _llama_public_model_id(llama_backend)
variant = getattr(llama_backend, "hf_variant", None)
if model_id and variant and ":" not in model_id:
return f"{model_id}:{variant}"
return model_id
backend = get_inference_backend()
return backend.active_model_name
return public_model_id(backend.active_model_name) or backend.active_model_name
def _validate_native_gguf_companion(
@ -3514,6 +3538,9 @@ _DISABLE_OPENAI_AUTO_SWITCH_SCOPE_KEY = "_unsloth_disable_openai_auto_switch"
# only restore an idle-freed model, never run the resolver (so a downloaded GGUF
# literally named "default" can't be swapped to). The NUL keeps it off any index.
_RELOAD_ONLY_MODEL = "\x00reload-only"
# One cold scan is worth paying to avoid answering a named model with another; a
# pathological install must not hang the request behind it forever.
_COLD_INDEX_WAIT_S = 10.0
def _switch_model_for_payload(payload) -> str:
@ -3599,6 +3626,477 @@ def _no_model_loaded_detail(base: str) -> str:
)
# Cap on ids listed by a "not downloaded" error, so it stays readable in a terminal.
_MAX_LISTED_AVAILABLE_MODELS = 8
def _raw_body_model(body) -> Optional[str]:
"""The ``model`` a raw-body endpoint was given, else None (same value
:func:`_auto_switch_from_request_body` fed the switch hook)."""
return body.get("model") if isinstance(body, dict) else None
async def _available_model_ids() -> list[str]:
"""Sorted ids a /v1 request may name, from the catalog ``GET /v1/models``
serves, so an error and the listing can't disagree."""
return sorted(
mid
for mid in (m.get("id") for m in await _openai_catalog_objects())
if isinstance(mid, str) and mid
)
def _format_available_models(ids: list[str]) -> str:
if not ids:
return ""
shown = ", ".join(ids[:_MAX_LISTED_AVAILABLE_MODELS])
extra = len(ids) - _MAX_LISTED_AVAILABLE_MODELS
return f"{shown} and {extra} more" if extra > 0 else shown
async def _unavailable_model_message(requested_model: str) -> str:
"""Why a named model can't serve this request, and what can.
Auto-switch only loads already-downloaded GGUFs, so a request naming a real
model usually fails because it is not on this machine. Pointing the caller at
/inference/load cannot fix that; say what is actually wrong.
"""
from core.inference.local_model_resolver import (
MISS_VARIANT_NOT_FOUND,
describe_local_miss,
)
reason, variants = await asyncio.to_thread(describe_local_miss, requested_model)
if reason == MISS_VARIANT_NOT_FOUND:
# Repo downloaded, only the quant missing: sibling quants beat the catalog.
base_id, _, wanted = requested_model.strip().rpartition(":")
return (
f"The model '{base_id}' is downloaded, but the quant '{wanted}' is not. "
f"Available quants: {', '.join(variants)}."
)
available = _format_available_models(await _available_model_ids())
if not available:
return (
f"The model '{requested_model}' is not downloaded on this server, and no "
"models are downloaded yet. Download one in Unsloth Studio."
)
return (
f"The model '{requested_model}' is not downloaded on this server. "
f"Available models: {available}. Download more in Unsloth Studio, "
"or list them with GET /v1/models."
)
async def _no_model_loaded_error(
base: str, requested_model: Optional[str], fastapi_request: Optional[Request], *, status: int
):
"""``(status, detail)`` for the /v1 sites that fail because nothing is loaded.
Changes only the case the generic text describes wrongly: auto-switch on, a
model named, and that name resolves to nothing local, so the switch silently
did nothing. That becomes a 404 model_not_found. Toggle off or no model named
keeps ``status`` and the :func:`_no_model_loaded_detail` text verbatim.
"""
from utils.openai_auto_switch_settings import get_openai_auto_switch_enabled
from core.inference.local_model_resolver import resolve_local_gguf
named = (
requested_model
if isinstance(requested_model, str)
and requested_model.strip()
and requested_model != _RELOAD_ONLY_MODEL
else None
)
if named is None or not get_openai_auto_switch_enabled():
return status, _no_model_loaded_detail(base)
try:
if _loaded_satisfies(named):
# Resident but on a backend this endpoint can't use, so "not downloaded" is false.
return status, _no_model_loaded_detail(base)
if await asyncio.to_thread(resolve_local_gguf, named) is not None:
# Resolvable but unloaded: the switch failed, which the generic text covers.
return status, _no_model_loaded_detail(base)
message = await _unavailable_model_message(named)
except Exception as exc:
# The diagnosis is a nicety; never let it turn a 4xx into a 500.
logger.debug("no-model-loaded diagnosis failed for %r: %s", named, exc)
return status, _no_model_loaded_detail(base)
path = getattr(getattr(fastapi_request, "url", None), "path", None)
if not isinstance(path, str):
# No request in hand: let the global /v1/* handler pick the envelope.
return 404, message
return 404, error_body_for_path(
path,
message,
status = 404,
code = "model_not_found",
param = "model",
)
def _auto_download_hf_token(fastapi_request: Optional[Request]) -> Optional[str]:
"""The token to fetch with: only one the caller sent themselves.
Never the server's ambient token. The repo here is named by whoever holds an
API key, so borrowing the owner's Hub identity would let that key pull the
owner's private repos and publish them in /v1/models for every other key.
The OpenAI bearer key is never used as an HF token either.
"""
from hub.dependencies import HUB_HF_TOKEN_HEADER, HUB_HF_TOKEN_MAX_LENGTH
headers = getattr(fastapi_request, "headers", None)
if headers is None:
return None
supplied = (headers.get(HUB_HF_TOKEN_HEADER) or "").strip()
if supplied and len(supplied) <= HUB_HF_TOKEN_MAX_LENGTH:
return supplied
return None
async def _maybe_auto_download_model(
requested_model: str,
fastapi_request: Optional[Request],
*,
require_vision: bool = False,
) -> None:
"""Opt-in: start fetching a named GGUF this server doesn't have.
Raises to stop the request when the model is downloading or cannot be
fetched. Off by default, and it never fires on a name that isn't shaped like
a Hub repo, so an unknown id like "gpt-4" still falls through to the resident
model as before.
"""
from utils.openai_auto_switch_settings import get_openai_auto_download_enabled
from core.inference.openai_auto_download import is_downloadable_ref, maybe_auto_download
if not requested_model or not get_openai_auto_download_enabled():
return
if not is_downloadable_ref(requested_model):
return
# An Ollama-style tag (":latest") names no quant, so the resolver misses a servable model.
if _loaded_satisfies(requested_model):
return
try:
refusal = await maybe_auto_download(
requested_model,
hf_token = _auto_download_hf_token(fastapi_request),
require_vision = require_vision,
)
except Exception as exc:
# Never turn a servable request into a 500 over the download attempt.
logger.warning("auto-download failed for %r: %s", requested_model, exc)
return
if refusal is None:
return
path = getattr(getattr(fastapi_request, "url", None), "path", None)
detail = (
error_body_for_path(
path,
refusal.message,
status = refusal.status,
code = refusal.code,
param = "model",
)
if isinstance(path, str)
else refusal.message
)
raise HTTPException(
status_code = refusal.status,
detail = detail,
headers = ({"Retry-After": str(refusal.retry_after)} if refusal.retry_after else None),
)
def _loaded_satisfies(requested: str) -> bool:
"""Whether what is serving right now actually answers to *requested*.
A bare ``org/model`` is satisfied by any loaded quant of that repo; an
explicit ``:QUANT`` must match the loaded one.
"""
from core.inference.openai_auto_download import looks_like_quant, split_model_ref
base, variant = split_model_ref(requested)
llama_backend = get_llama_cpp_backend()
if getattr(llama_backend, "is_loaded", False):
candidates = [
candidate
for candidate in (
getattr(llama_backend, "model_identifier", None),
getattr(llama_backend, "_openai_advertised_id", None),
_llama_public_model_id(llama_backend),
)
if candidate
]
if not _matches_any(base, candidates):
return False
if not looks_like_quant(variant):
# An Ollama-style tag (":latest", ":8b") names no file, so the repo is enough.
return True
return (getattr(llama_backend, "hf_variant", None) or "").lower() == variant.lower()
active = getattr(get_inference_backend(), "active_model_name", None)
if not active:
return False
# Only llama.cpp carries a quant identity, so this backend can only match on the repo.
if looks_like_quant(variant):
return False
return _matches_any(base, [active, public_model_id(active)])
def _raise_still_indexing(requested_model: str, fastapi_request) -> None:
"""Refuse a name we cannot yet place, rather than answer it with another model."""
path = getattr(getattr(fastapi_request, "url", None), "path", None)
message = (
f"This server is still indexing its local models, so it cannot confirm "
f"'{requested_model}' yet. Retry shortly."
)
raise HTTPException(
status_code = 503,
detail = (
error_body_for_path(path, message, status = 503, code = "model_indexing")
if isinstance(path, str)
else message
),
headers = {"Retry-After": "5"},
)
def _matches_any(requested: str, candidates) -> bool:
"""Whether *requested* names any of *candidates*.
A repo alias is case-insensitive, a filesystem path is not: lowercasing both
made /srv/models/foo.gguf and /srv/models/Foo.gguf the same weights, which is
the same trap _norm_path exists for one comparison further down.
"""
lowered = requested.strip().lower()
for candidate in candidates:
if not candidate:
continue
if _looks_like_local_path(requested) or _looks_like_local_path(candidate):
if _norm_path(requested) == _norm_path(candidate):
return True
continue
if lowered == str(candidate).strip().lower():
return True
return False
def _looks_like_local_path(value: str) -> bool:
"""A filesystem path rather than a repo id, so case matters."""
text = str(value)
return text.startswith("/") or text.startswith("~") or ":\\" in text or "\\" in text
def _norm_path(value: str) -> str:
"""Compare-ready path. normcase, not lower: on a case-sensitive filesystem
/srv/models/Foo and /srv/models/foo are different models."""
import os
# normcase after, not before: on Windows it folds case *and* rewrites the
# separator to a backslash, so normalizing first leaves the descendant checks
# below comparing a "/" against a path that no longer has any.
return os.path.normcase(str(value)).replace("\\", "/").rstrip("/")
def _resident_quant_is(variant: Optional[str]) -> bool:
"""Whether the loaded GGUF is that exact quant."""
resident = getattr(get_llama_cpp_backend(), "hf_variant", None) or ""
return bool(variant) and resident.lower() == variant.strip().lower()
def _resolves_to_resident(load_path: Optional[str], *, llama_only: bool = False) -> bool:
"""Whether a resolved on-disk path is what is already loaded.
``llama_only`` drops the Transformers backend from the comparison. Only
llama.cpp carries a quant identity, so a Transformers model active from a
directory that also holds GGUF exports would otherwise match a request for
one of those quants and answer it with the safetensors weights.
"""
if not load_path:
return False
target = _norm_path(load_path)
llama_backend = get_llama_cpp_backend()
for candidate in (
getattr(llama_backend, "gguf_path", None)
if getattr(llama_backend, "is_loaded", False)
else None,
getattr(llama_backend, "model_identifier", None)
if getattr(llama_backend, "is_loaded", False)
else None,
None if llama_only else getattr(get_inference_backend(), "active_model_name", None),
):
if not candidate:
continue
current = _norm_path(candidate)
if current == target:
return True
if current.startswith(f"{target}/"):
# A model directory holding the weights loaded from it. Nested entries
# (/models/A alongside /models/A/sub/B) satisfied this too, so a request
# for A was answered with B. The innermost indexed model owns the file;
# with none indexed there is no nesting to tell apart, so keep matching.
owner = _innermost_indexed_owner(current)
if owner is None or owner == target:
return True
continue
if target.startswith(f"{current}/"):
return True
return False
def _innermost_indexed_owner(path: str) -> Optional[str]:
"""Longest catalog-listed model path containing *path*, or None if none does."""
best = None
for info in _CATALOG_CACHE["models"] or ():
listed = getattr(info, "path", None)
if not listed:
continue
normalized = _norm_path(listed)
if path == normalized or path.startswith(f"{normalized}/"):
if best is None or len(normalized) > len(best):
best = normalized
return best
async def _reject_unservable_model(
requested_model: Optional[str], fastapi_request: Optional[Request]
) -> None:
"""Refuse rather than answer a named model with a different one.
Only for a reference this server can tell was meant for it: an explicit GGUF
quant, or a model that is actually here. A namespace decides nothing either
way. ``vendor/model`` is how LiteLLM and OpenRouter name every provider, so
``anthropic/claude-3.5-sonnet`` falls through like ``gpt-4``; a standalone or
custom-folder GGUF is advertised without one, so a slashless id that does
resolve locally is still a concrete reference. Only runs while something is
serving; with nothing loaded the caller's own :func:`_no_model_loaded_error`
already says the right thing.
"""
from core.inference.openai_auto_download import looks_like_quant, split_model_ref
if (
not isinstance(requested_model, str)
or not requested_model.strip()
or requested_model == _RELOAD_ONLY_MODEL
):
return
base, variant = split_model_ref(requested_model)
quantified = looks_like_quant(variant)
from core.inference.local_model_resolver import (
index_is_built,
recently_downloaded,
resolve_local_gguf,
warm_index_soon,
)
from utils.openai_auto_switch_settings import get_openai_auto_switch_enabled
still_indexing = False
try:
if _loaded_satisfies(requested_model):
return
if not (
get_llama_cpp_backend().is_loaded
or getattr(get_inference_backend(), "active_model_name", None)
):
return
# Refresh in the background and read the index as-is: scanning here would stall the
# request, and a cold index only costs evidence (the gate below fails safe without it).
if index_is_built():
warm_index_soon()
resolved = resolve_local_gguf(requested_model, allow_scan = False)
else:
# Before the first scan there is nothing cached to reason from, and falling
# through would answer a named model with the resident one. Pay the scan
# once, off the loop and bounded, rather than reading "not scanned yet" as
# "not here". Later requests take the cached branch above.
try:
resolved = await asyncio.wait_for(
asyncio.to_thread(resolve_local_gguf, requested_model),
_COLD_INDEX_WAIT_S,
)
except (TimeoutError, asyncio.TimeoutError):
# Still scanning, so nothing is known about this name. Falling through
# would put the resident model behind it, which is the failure this
# whole hook exists to stop, so say "not yet" instead of guessing.
warm_index_soon()
still_indexing = True
resolved = None
# A manual load stores the on-disk path the resolver advertises under an alias, so
# match on the path too.
# Quants of one repo share a directory, so the path alone cannot tell them
# apart: without the variant check an explicit :Q8_0 would be answered by a
# resident Q4_K_M, which _loaded_satisfies has already refused by name.
if (
resolved is not None
and _resolves_to_resident(resolved[0], llama_only = quantified)
and (not quantified or _resident_quant_is(variant))
):
return
downloaded = resolved is not None
# /v1/models may have advertised this id off its own scan while the index is cold.
advertised = _advertised_local_path(base)
if (
advertised is not None
and _resolves_to_resident(advertised, llama_only = quantified)
and (not quantified or _resident_quant_is(variant))
):
return
# The exact ref may miss on the quant alone, so ask about the repo too.
here = (
downloaded
or advertised is not None
# Just landed, so no scan has indexed it yet and neither of the above sees it.
or recently_downloaded(base)
or (variant is not None and resolve_local_gguf(base, allow_scan = False) is not None)
)
switchable = downloaded and get_openai_auto_switch_enabled()
except HTTPException:
# A refusal decided above is the answer, not a failure to decide. Without this
# the handler below would log it and fall through to the resident model.
raise
except Exception as exc:
# Can't verify: an explicit quant still proves intent, so refuse; let anything else by.
logger.debug("unservable-model check failed for %r: %s", requested_model, exc)
if not quantified:
return
downloaded = here = switchable = False
if still_indexing:
_raise_still_indexing(requested_model, fastapi_request)
if not (quantified or here):
return
if switchable:
# On disk and switching allowed, so the swap failed: the resident model is wrong weights.
status_code, code = 503, "model_switch_failed"
message = (
f"The model '{requested_model}' is downloaded, but this server could not "
"switch to it. Retry shortly, or load it in Unsloth Studio."
)
elif downloaded:
status_code, code = 404, "model_not_found"
message = (
f"The model '{requested_model}' is downloaded but not loaded, and "
"'Switch model by request' is off, so this server can only serve the "
"loaded model. Turn it on in Unsloth Studio under Settings > API."
)
else:
status_code, code = 404, "model_not_found"
try:
message = await _unavailable_model_message(requested_model)
except Exception as exc:
# Only the wording is uncertain; the mismatch is already established.
logger.debug("unavailable-model diagnosis failed for %r: %s", requested_model, exc)
message = f"The model '{requested_model}' is not the model this server is serving."
path = getattr(getattr(fastapi_request, "url", None), "path", None)
raise HTTPException(
status_code = status_code,
detail = (
error_body_for_path(path, message, status = status_code, code = code, param = "model")
if isinstance(path, str)
else message
),
headers = {"Retry-After": "5"} if status_code == 503 else None,
)
async def _maybe_auto_switch_model(
requested_model: Optional[str],
fastapi_request: Request,
@ -3610,7 +4108,8 @@ async def _maybe_auto_switch_model(
No-op unless enabled and ``requested_model`` resolves to a downloaded local
model different from the loaded one. Unknown names fall through (drop-in
compat) and no remote download is triggered. ``require_vision`` rejects a swap
compat); a miss only reaches the network when auto-download is also on, and
even then only for ``namespace/name`` ids. ``require_vision`` rejects a swap
to a text-only target before it runs, so an image request can't evict the
resident vision model only to 400 afterwards.
"""
@ -3640,6 +4139,8 @@ async def _maybe_auto_switch_model(
# loop freed is restored on the next request. The resolver-based switch still
# requires the auto-switch toggle.
if not auto_switch_on and get_auto_unload_idle_seconds() <= 0:
# No switching to do, but a named model must still not be answered by another.
await _reject_unservable_model(requested_model, fastapi_request)
return
async def _resolve_and_switch() -> None:
@ -3653,6 +4154,11 @@ async def _maybe_auto_switch_model(
else None
)
if resolved is None:
# Not on disk. Opt-in: fetch in the background and ask the caller to retry.
if auto_switch_on and not reload_only:
await _maybe_auto_download_model(
requested_model, fastapi_request, require_vision = require_vision
)
# Idle-unload may have freed the model; reload exactly what it freed
# (path + quant + advertised id) so an alias/unknown name stays servable
# and keeps the override keyed by the advertised id, not the load path.
@ -3679,7 +4185,13 @@ async def _maybe_auto_switch_model(
backend = get_llama_cpp_backend()
# A bare model id (no :VARIANT) is satisfied by any loaded quant of that
# repo, so it never reloads a different local quant that already serves it.
bare = ":" not in requested_model
from core.inference.openai_auto_download import looks_like_quant, split_model_ref
# A tag that names no quant (":latest", ":8b") means the repo, exactly as
# _loaded_satisfies and the resolver read it. Treating it as a quant tears
# down a serving Q8 to load the preferred Q4 for a request either satisfies.
_, _requested_variant = split_model_ref(requested_model)
bare = not looks_like_quant(_requested_variant)
def _already_serving() -> bool:
# Match against both the concrete load path and the advertised repo id,
@ -3783,6 +4295,8 @@ async def _maybe_auto_switch_model(
_note_switch_waiter(key, -1)
await _resolve_and_switch()
# The switch may have missed, so refuse rather than answer as whatever is resident.
await _reject_unservable_model(requested_model, fastapi_request)
async def _auto_switch_from_request_body(request: Request, current_subject: str):
@ -4332,6 +4846,13 @@ async def _load_model_impl(
# sampled step logs even if it reports 100% immediately (cached/small load).
_reset_load_progress_step()
# Live "loading" row: discarded if already loaded, relabelled on the real id, closed on exit.
_load_event = api_monitor.record_lifecycle(
event = "load",
model = _lifecycle_model_label(request.model_path, request.gguf_variant),
running = True,
)
native_grant_backed = False
model_log_label = request.model_path
gguf_load_stack = ExitStack()
@ -4425,6 +4946,8 @@ async def _load_model_impl(
and getattr(llama_backend, "_audio_probed", True)
):
llama_backend._record_matching_gpu_request(request.gpu_ids)
# Nothing was loaded, so the monitor must not show a load row.
api_monitor.discard(_load_event)
logger.info(
"Model already loaded (GGUF): "
f"{model_log_label} variant={request.gguf_variant or llama_backend.hf_variant}, skipping reload"
@ -4478,6 +5001,7 @@ async def _load_model_impl(
backend.active_model_name
and backend.active_model_name.lower() == model_identifier.lower()
):
api_monitor.discard(_load_event) # nothing loaded, no monitor row
logger.info(f"Model already loaded (Unsloth): {model_log_label}, skipping reload")
inference_config = load_inference_config(backend.active_model_name)
_model_info = backend.models.get(backend.active_model_name, {})
@ -4790,6 +5314,11 @@ async def _load_model_impl(
logger.info(
f"Loaded GGUF model via llama-server: {model_log_label if native_grant_backed else config.identifier}"
)
_close_load_event(
_load_event,
model_log_label if native_grant_backed else config.identifier,
request.gguf_variant or getattr(llama_backend, "hf_variant", None),
)
# Clear any idle-unload reload stash now, not only on the next poll.
from core.inference.llama_keepwarm import note_model_loaded
@ -4908,6 +5437,9 @@ async def _load_model_impl(
logger.info(
f"Loaded model: {model_log_label if native_grant_backed else config.identifier}"
)
_close_load_event(
_load_event, model_log_label if native_grant_backed else config.identifier, None
)
# Clear any idle-unload reload stash: a manual load supersedes an idle-freed
# GGUF, so the next /v1 request must not resurrect it. Mirror the GGUF branch
# above; without this a non-GGUF load leaves a stale stash until the idle
@ -5030,6 +5562,8 @@ async def _load_model_impl(
raise HTTPException(status_code = 500, detail = f"Failed to load model: {msg}")
finally:
gguf_load_stack.close()
# Catch-all: an error or cancelled load would otherwise leave the row "loading".
api_monitor.fail_open(_load_event, "Load did not complete")
def _requires_trust_remote_code_for_model(
@ -5646,10 +6180,18 @@ async def unload_model(request: UnloadRequest, current_subject: str = Depends(ge
)
or not llama_backend.is_loaded
):
# Read the identity before teardown clears it, so the row reads repo:QUANT.
_unloaded = _llama_public_model_id(llama_backend, request.model_path)
_unloaded_variant = getattr(llama_backend, "hf_variant", None)
# A manual unload is a deliberate user action: tear down now even if a
# request is mid-stream (only the automatic idle loop defers to it).
llama_backend.unload_model()
note_model_unloaded()
api_monitor.record_lifecycle(
event = "unload",
model = _lifecycle_model_label(_unloaded, _unloaded_variant),
reason = "manual",
)
logger.info(f"Unloaded GGUF model: {request.model_path}")
return UnloadResponse(status = "unloaded", model = request.model_path)
@ -5659,6 +6201,11 @@ async def unload_model(request: UnloadRequest, current_subject: str = Depends(ge
backend = get_inference_backend()
await asyncio.to_thread(backend.unload_model, request.model_path)
note_model_unloaded()
api_monitor.record_lifecycle(
event = "unload",
model = _lifecycle_model_label(request.model_path),
reason = "manual",
)
logger.info(f"Unloaded model: {request.model_path}")
return UnloadResponse(status = "unloaded", model = request.model_path)
@ -5913,6 +6460,9 @@ async def get_status(current_subject: str = Depends(get_current_subject)):
and os.path.isabs(_model_id)
):
_display_model_id = os.path.basename(_model_id)
elif not _native_grant_backed and _display_model_id == _model_id:
# No label registered, so report the clean public id, not the snapshot's sha.
_display_model_id = _llama_public_model_id(llama_backend) or _display_model_id
_inference_cfg = load_inference_config(_model_id) if _model_id else None
_audio_type = getattr(llama_backend, "_audio_type", None)
# Don't surface Unsloth's auto-applied bundled family template (e.g. the
@ -7791,10 +8341,13 @@ async def openai_chat_completions(
else:
backend = get_inference_backend()
if not backend.active_model_name:
raise HTTPException(
status_code = 400,
detail = _no_model_loaded_detail("No model loaded. Call POST /inference/load first."),
_status, _detail = await _no_model_loaded_error(
"No model loaded. Call POST /inference/load first.",
_switch_model_for_payload(payload),
request,
status = 400,
)
raise HTTPException(status_code = _status, detail = _detail)
# Clean public id so the response never echoes a local path; the audio
# branch below receives this sanitized label too.
model_name = public_model_id(backend.active_model_name) or payload.model
@ -10502,6 +11055,9 @@ def _openai_model_objects() -> list[dict]:
"created": _created,
"owned_by": _OWNED_BY,
}
_quant = getattr(llama_backend, "hf_variant", None)
if _quant and _quant_reference_resolves(entry["id"], _quant):
entry["quant"] = _quant
_ctx = _positive_int_or_none(getattr(llama_backend, "context_length", None))
if _ctx is not None:
entry["context_length"] = _ctx
@ -10542,6 +11098,50 @@ def _openai_model_objects() -> list[dict]:
# Brief cache for the local-model filesystem scan so repeated /v1/models calls
# don't rescan the HF cache and models dirs on every request.
_CATALOG_CACHE: dict = {"at": 0.0, "models": []}
# Ids the last catalog scan listed, rebuilt only when that scan is replaced.
_ADVERTISED_CACHE: dict = {"at": None, "paths": {}}
def _quant_reference_resolves(model_id: Optional[str], quant: str) -> bool:
"""Whether ``<model_id>:<quant>`` still resolves once this model is not resident.
A standalone .gguf takes its quant from the filename, but the resolver stores
such files with no quants at all, so advertising one hands out a pin that dies
the moment another model loads.
"""
from core.inference.local_model_resolver import (
index_is_built,
recently_downloaded,
resolve_local_gguf,
warm_index_soon,
)
if not model_id:
return False
# Cold index proves nothing, and publishing on no proof is what hands out the
# dead pin; warm so the next response carries the quant.
warm_index_soon()
return resolve_local_gguf(f"{model_id}:{quant}", allow_scan = False) is not None
def _advertised_local_path(model: str) -> Optional[str]:
"""On-disk path of *model* if the last /v1/models scan listed it, else None.
Cache-only, never scans. The catalog scans on its own schedule, so it can have
advertised a local model the resolver index has not picked up yet; having
advertised it is evidence the name means something other than the resident one.
"""
if _ADVERTISED_CACHE["at"] != _CATALOG_CACHE["at"]:
paths = {}
for info in _CATALOG_CACHE["models"] or ():
cid = getattr(info, "model_id", None) or public_model_id(getattr(info, "id", None))
path = getattr(info, "path", None)
if cid and path:
paths.setdefault(cid.strip().lower(), path)
_ADVERTISED_CACHE.update(at = _CATALOG_CACHE["at"], paths = paths)
return _ADVERTISED_CACHE["paths"].get(model.strip().lower())
_CATALOG_TTL_S = 30.0
# Per-loop lock (like _auto_switch_lock): a module-level asyncio.Lock ties its
# waiters to the loop that first awaited it, so a second event loop awaiting it
@ -10608,11 +11208,14 @@ async def _openai_catalog_objects() -> list[dict]:
# read from the on-disk files, not model_format: the HF-cache scanner leaves
# model_format unset for GGUF snapshots, so a model_format filter would drop
# every cached GGUF. The file checks run off the loop.
from core.inference.local_model_resolver import info_has_local_gguf
from core.inference.local_model_resolver import local_gguf_quants
catalog = await _cached_local_catalog()
servable = await asyncio.to_thread(lambda: [i for i in catalog if info_has_local_gguf(i)])
for info in servable:
# One scan yields both "is this servable" and its on-disk quants, so no second pass.
servable = await asyncio.to_thread(
lambda: [(i, q) for i in catalog if (q := local_gguf_quants(i)) is not None]
)
for info, quants in servable:
cid = getattr(info, "model_id", None) or public_model_id(getattr(info, "id", None))
if not cid or cid in by_id:
continue
@ -10621,8 +11224,22 @@ async def _openai_catalog_objects() -> list[dict]:
"object": "model",
"created": _created,
"owned_by": _OWNED_BY,
"loaded": False,
# A manual load keys the resident entry by path basename while the catalog uses
# the alias, so match on the path or the alias reads as not loaded. llama-only:
# these entries are advertised as GGUF with a GGUF quant, so a Transformers
# model live from a directory that also holds GGUF exports must not mark one
# loaded, or the examples pin a quant nothing can serve with switching off.
"loaded": _resolves_to_resident(getattr(info, "path", None), llama_only = True),
}
# The id stays bare for OpenAI compat; a client appends ":<quant>" to pin one.
# For the resident model that has to be the quant actually loaded, not the
# preferred one on disk, or the listing advertises alias:Q4 as loaded while
# Q8 is serving and pinning it 404s.
resident_quant = getattr(get_llama_cpp_backend(), "hf_variant", None)
if obj["loaded"] and resident_quant:
obj["quant"] = resident_quant
elif quants:
obj["quant"] = quants[0]
display = getattr(info, "display_name", None)
if display:
obj["display_name"] = display
@ -10758,10 +11375,13 @@ async def openai_completions(request: Request, current_subject: str = Depends(ge
# Opt-in: load the requested local GGUF before the loaded-state check.
body = await _auto_switch_from_request_body(request, current_subject)
if not llama_backend.is_loaded:
raise HTTPException(
status_code = 503,
detail = _no_model_loaded_detail("No GGUF model loaded. Load a GGUF model first."),
_status, _detail = await _no_model_loaded_error(
"No GGUF model loaded. Load a GGUF model first.",
_raw_body_model(body),
request,
status = 503,
)
raise HTTPException(status_code = _status, detail = _detail)
if not isinstance(body, dict):
# Re-read to re-raise a malformed-body error (post-503, pre-feature behavior);
# a valid non-dict body such as a list is a clean 400 rather than a 500.
@ -10978,10 +11598,13 @@ async def openai_embeddings(request: Request, current_subject: str = Depends(get
# a non-embedding target switches, then llama-server returns a no-pooling error.
body = await _auto_switch_from_request_body(request, current_subject)
if not llama_backend.is_loaded:
raise HTTPException(
status_code = 503,
detail = _no_model_loaded_detail("No GGUF model loaded. Load a GGUF model first."),
_status, _detail = await _no_model_loaded_error(
"No GGUF model loaded. Load a GGUF model first.",
_raw_body_model(body),
request,
status = 503,
)
raise HTTPException(status_code = _status, detail = _detail)
if not isinstance(body, dict):
# Re-read to re-raise a malformed-body error (post-503, pre-feature behavior);
# a valid non-dict body such as a list is a clean 400 rather than a 500.
@ -11718,14 +12341,15 @@ async def _responses_stream(
# double-layer asyncgen close pattern that produces "Attempted to exit
# cancel scope in a different task" on Python 3.13. Surface a typed 400
# so the client sees a useful error instead of a dangling stream.
raise HTTPException(
status_code = 400,
detail = _no_model_loaded_detail(
"Streaming /v1/responses requires a GGUF model loaded via "
"llama-server. Use non-streaming /v1/responses, "
"/v1/chat/completions, or load a GGUF model."
),
_status, _detail = await _no_model_loaded_error(
"Streaming /v1/responses requires a GGUF model loaded via "
"llama-server. Use non-streaming /v1/responses, "
"/v1/chat/completions, or load a GGUF model.",
_switch_model_for_payload(payload),
request,
status = 400,
)
raise HTTPException(status_code = _status, detail = _detail)
# Direct pass-through bypasses the openai_chat_completions image gate.
if not llama_backend.is_vision and any(
@ -12967,10 +13591,13 @@ async def anthropic_count_tokens(
llama_backend = get_llama_cpp_backend()
if not llama_backend.is_loaded:
raise HTTPException(
status_code = 503,
detail = _no_model_loaded_detail("No GGUF model loaded. Load a GGUF model first."),
_status, _detail = await _no_model_loaded_error(
"No GGUF model loaded. Load a GGUF model first.",
_switch_model_for_payload(payload),
request,
status = 503,
)
raise HTTPException(status_code = _status, detail = _detail)
# Same Anthropic → OpenAI translation as anthropic_messages: system is
# folded into the messages list, so pass system=None to the counter.
@ -13039,6 +13666,7 @@ async def anthropic_messages(
# before any request-shape check, exactly as the pre-feature endpoint did. When
# an automatic load can run (auto-switch or a standalone idle TTL), fall through
# so validation runs before the reload hook gets a chance to restore the model.
# Plain detail, not _no_model_loaded_error: that helper leaves this case unchanged.
if not llama_backend.is_loaded and not _automatic_model_load_may_run():
raise HTTPException(
status_code = 503,
@ -13138,10 +13766,13 @@ async def anthropic_messages(
require_vision = _anthropic_request_has_image(payload),
)
if not llama_backend.is_loaded:
raise HTTPException(
status_code = 503,
detail = _no_model_loaded_detail("No GGUF model loaded. Load a GGUF model first."),
_status, _detail = await _no_model_loaded_error(
"No GGUF model loaded. Load a GGUF model first.",
_switch_model_for_payload(payload),
request,
status = 503,
)
raise HTTPException(status_code = _status, detail = _detail)
# Advertised repo id after an auto-switch load, else a clean public id, never
# the local .gguf path (and a legacy raw path in payload.model is sanitized).

14
studio/backend/routes/settings.py
View file

@ -37,12 +37,14 @@ from utils.helper_precache_settings import (
from utils.coding_agents import CODING_AGENTS, detect_installed_coding_agents
from utils.openai_auto_switch_settings import (
DEFAULT_AUTO_UNLOAD_KEEP_KV,
DEFAULT_OPENAI_AUTO_DOWNLOAD_ENABLED,
DEFAULT_OPENAI_AUTO_SWITCH_ENABLED,
get_auto_unload_idle_seconds,
get_auto_unload_keep_kv,
get_model_overrides,
get_openai_auto_switch_enabled,
get_stored_auto_unload_idle_seconds,
get_stored_openai_auto_download_enabled,
set_model_override,
set_openai_auto_switch,
)
@ -112,6 +114,7 @@ class OpenAIAutoSwitchPayload(BaseModel):
# None leaves the stored value untouched (partial updates can't clobber it).
auto_unload_idle_seconds: Optional[int] = Field(default = None, ge = 0)
auto_unload_keep_kv: Optional[bool] = None
auto_download_model: Optional[bool] = None
class OpenAIAutoSwitchResponse(BaseModel):
@ -123,6 +126,8 @@ class OpenAIAutoSwitchResponse(BaseModel):
# is false, so the UI can show idle-unload as active instead of "needs enable".
idle_unload_active: bool = False
auto_unload_keep_kv: bool = DEFAULT_AUTO_UNLOAD_KEEP_KV
# Stored, not effective: the UI must round-trip the saved value across an auto-switch toggle.
auto_download_model: bool = DEFAULT_OPENAI_AUTO_DOWNLOAD_ENABLED
class ModelOverridePayload(BaseModel):
@ -245,6 +250,7 @@ def get_openai_auto_switch(
auto_unload_idle_seconds = get_stored_auto_unload_idle_seconds(),
idle_unload_active = get_auto_unload_idle_seconds() > 0,
auto_unload_keep_kv = get_auto_unload_keep_kv(),
auto_download_model = get_stored_openai_auto_download_enabled(),
)
@ -253,8 +259,11 @@ def update_openai_auto_switch(
payload: OpenAIAutoSwitchPayload, current_subject: str = Depends(get_current_subject)
) -> OpenAIAutoSwitchResponse:
try:
enabled, idle_seconds, keep_kv = set_openai_auto_switch(
payload.enabled, payload.auto_unload_idle_seconds, payload.auto_unload_keep_kv
enabled, idle_seconds, keep_kv, auto_download = set_openai_auto_switch(
payload.enabled,
payload.auto_unload_idle_seconds,
payload.auto_unload_keep_kv,
payload.auto_download_model,
)
except ValueError as exc:
raise log_and_http_error(
@ -274,6 +283,7 @@ def update_openai_auto_switch(
auto_unload_idle_seconds = idle_seconds,
idle_unload_active = idle_unload_active,
auto_unload_keep_kv = keep_kv,
auto_download_model = auto_download,
)

22
studio/backend/tests/conftest.py
View file

@ -58,6 +58,28 @@ def pytest_addoption(parser):
# E2E server fixtures
@pytest.fixture(autouse = True)
def _no_background_model_scan(monkeypatch):
"""Keep the /v1 admission hook from scanning the real HF cache during tests.
The hook warms the local-model index on a background thread. That is right in a
server and wrong here: it walks the developer's actual caches, which on a large
install takes seconds, and the resulting I/O starves the loop under the
timing-sensitive streaming tests. Tests that exercise the warm patch it back.
"""
import time
from core.inference import local_model_resolver
monkeypatch.setattr(local_model_resolver, "warm_index_soon", lambda: None)
# Start from a built, empty index. Stubbing only the background warm still left the
# cold path walking those caches synchronously inside the admission wait, so on a
# large install the assertion became a 503 "still indexing". Tests that want the
# cold path set _scan back themselves (and stub the scan). _build_index is left
# alone so the tests that call it directly still exercise the real walk.
monkeypatch.setattr(local_model_resolver, "_scan", (time.monotonic(), {}))
@pytest.fixture(scope = "session")
def studio_server(request):
"""Yield ``(base_url, api_key)`` for e2e tests.

97
studio/backend/tests/test_api_monitor.py
View file

@ -258,3 +258,100 @@ def test_api_monitor_append_reply_exact_cap_then_more_marks_truncated():
monitor.append_reply(entry_id, "y")
reply = monitor.snapshot()[0]["reply"]
assert len(reply) == m._MAX_REPLY_CHARS and reply.endswith("...")
# ── model lifecycle rows (load / unload) ────────────────────────────
def test_lifecycle_load_row_opens_running_then_closes():
monitor = ApiMonitor(max_entries = 5)
event_id = monitor.record_lifecycle(event = "load", model = "org/A-GGUF", running = True)
row = monitor.snapshot()[0]
assert row["kind"] == "lifecycle" and row["event"] == "load"
assert row["status"] == "running" and row["duration_ms"] is None
# A load in progress is not an in-flight API request.
assert monitor.active_count() == 0
monitor.relabel(event_id, "org/A-GGUF:Q4_K_M")
monitor.finish(event_id)
row = monitor.snapshot()[0]
assert row["status"] == "completed"
assert row["model"] == "org/A-GGUF:Q4_K_M"
assert row["duration_ms"] is not None
def test_lifecycle_unload_row_is_terminal_on_arrival():
monitor = ApiMonitor(max_entries = 5)
monitor.record_lifecycle(event = "unload", model = "org/A-GGUF", reason = "idle")
row = monitor.snapshot()[0]
assert row["status"] == "completed"
assert (row["event"], row["reason"]) == ("unload", "idle")
assert monitor.active_count() == 0
def test_lifecycle_rows_are_visible_to_every_subject():
# A load is server-wide, so it must not vanish for other API keys like a request does.
monitor = ApiMonitor(max_entries = 5)
monitor.start(
endpoint = "/v1/chat/completions",
method = "POST",
model = "m",
prompt = "hi",
subject = "alice",
)
event_id = monitor.record_lifecycle(event = "unload", model = "org/A-GGUF")
bob = monitor.snapshot(subject = "bob")
assert [r["kind"] for r in bob] == ["lifecycle"]
assert monitor.get(event_id, subject = "bob") is not None
assert len(monitor.snapshot(subject = "alice")) == 2
def test_request_rows_stay_private_to_their_subject():
monitor = ApiMonitor(max_entries = 5)
rid = monitor.start(
endpoint = "/v1/chat/completions",
method = "POST",
model = "m",
prompt = "hi",
subject = "alice",
)
assert monitor.snapshot(subject = "bob") == []
assert monitor.get(rid, subject = "bob") is None
def test_discard_drops_a_row_that_never_happened():
# A load that found the model already resident must leave no trace.
monitor = ApiMonitor(max_entries = 5)
event_id = monitor.record_lifecycle(event = "load", model = "org/A-GGUF", running = True)
monitor.discard(event_id)
assert monitor.snapshot() == []
monitor.discard(event_id) # idempotent
def test_fail_open_never_touches_a_finished_row():
# Called from a finally, so it must not stamp an error onto a load that succeeded.
monitor = ApiMonitor(max_entries = 5)
event_id = monitor.record_lifecycle(event = "load", model = "org/A-GGUF", running = True)
monitor.finish(event_id)
monitor.fail_open(event_id, "Load did not complete")
row = monitor.snapshot()[0]
assert row["status"] == "completed" and row["error"] is None
still_open = monitor.record_lifecycle(event = "load", model = "org/B-GGUF", running = True)
monitor.fail_open(still_open, "Load did not complete")
assert monitor.snapshot()[0]["status"] == "error"
def test_lifecycle_rows_share_the_retention_budget():
monitor = ApiMonitor(max_entries = 2)
for i in range(4):
monitor.record_lifecycle(event = "unload", model = f"org/M{i}")
models = [r["model"] for r in monitor.snapshot()]
assert models == ["org/M3", "org/M2"]
def test_request_rows_report_kind_request():
monitor = ApiMonitor(max_entries = 2)
monitor.start(endpoint = "/v1/chat/completions", method = "POST", model = "m", prompt = "hi")
assert monitor.snapshot()[0]["kind"] == "request"

17
studio/backend/tests/test_model_ids.py
View file

@ -37,6 +37,23 @@ def test_directory_path_uses_basename():
assert public_model_id("a/b/c") == "c"
def test_hf_cache_snapshot_recovers_the_repo_id():
from core.inference.model_ids import hf_cache_repo_id
# The snapshot basename is a commit sha, so recover org/name instead.
snapshot = (
"/home/u/.cache/huggingface/hub/models--unsloth--gemma-4-31B-it-GGUF"
"/snapshots/c1ac76e99d5513b141e8adde7288b85c3f9c32ec"
)
assert public_model_id(snapshot) == "unsloth/gemma-4-31B-it-GGUF"
# A file inside the snapshot resolves the same way, not to the file stem.
assert public_model_id(snapshot + "/gemma-4-31B-it-UD-Q5_K_XL.gguf") == (
"unsloth/gemma-4-31B-it-GGUF"
)
assert hf_cache_repo_id("/opt/models/plain.gguf") is None
assert hf_cache_repo_id(None) is None
def test_relative_and_home_paths_are_sanitized():
# ./ ../ ~ prefixed paths are local and must not be echoed raw.
assert public_model_id("./model.gguf") == "model"

1798
studio/backend/tests/test_openai_auto_download.py Normal file
View file

File diff suppressed because it is too large Load diff

555
studio/backend/tests/test_openai_auto_switch.py
View file

@ -11,6 +11,7 @@ import asyncio
import os
import pytest
from fastapi import HTTPException
import routes.inference as inference_route
from models.inference import LoadRequest
@ -18,6 +19,19 @@ from core.inference import local_model_resolver as resolver
from utils import openai_auto_switch_settings as settings
@pytest.fixture(autouse = True)
def _clean_resolver_index():
"""Drop the scan cache around every test.
The /v1 admission hook warms the index in the background, so without this a
test that exercises the hook can publish its own fixture's scan and, inside the
TTL, hand it to the next test that expects a fresh one.
"""
resolver.invalidate_index()
yield
resolver.invalidate_index()
class _FakeBackend:
effective_parallel_slots = 1
_slot_save_binary = None
@ -94,7 +108,7 @@ class _LoadRecorder:
def _wire(monkeypatch, *, enabled, resolves_to, backend, recorder):
monkeypatch.setattr(settings, "get_openai_auto_switch_enabled", lambda: enabled)
monkeypatch.setattr(resolver, "resolve_local_gguf", lambda _m: resolves_to)
monkeypatch.setattr(resolver, "resolve_local_gguf", lambda _m, **_kw: resolves_to)
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: backend)
# Auto-switch loads via _load_model_impl (the /load route holds the lifecycle
# gate that auto-switch already owns, so it calls the impl directly).
@ -116,7 +130,11 @@ def test_flag_off_never_loads(monkeypatch):
backend = backend,
recorder = rec,
)
_run_hook("unsloth/B-GGUF")
# Off means no load, but A must not answer as B either: say why instead.
with pytest.raises(HTTPException) as excinfo:
_run_hook("unsloth/B-GGUF")
assert excinfo.value.status_code == 404
assert "Switch model by request" in str(excinfo.value.detail)
assert rec.calls == []
@ -387,6 +405,45 @@ def test_resolver_nonstring_model_is_failsafe():
assert resolver.resolve_local_gguf(None) is None
def test_describe_local_miss_separates_missing_repo_from_missing_quant(monkeypatch):
# Two different misses: the repo isn't downloaded, or only that quant is absent.
monkeypatch.setattr(
resolver,
"_build_index",
lambda: {"unsloth/b-gguf": _entry("unsloth/B-GGUF", "UD-Q5_K_XL", "Q4_K_M")},
)
resolver._scan = (0.0, {})
assert resolver.describe_local_miss("unsloth/B-GGUF:Q8_0") == (
resolver.MISS_VARIANT_NOT_FOUND,
("UD-Q5_K_XL", "Q4_K_M"),
)
# Split the same way resolve_local_gguf does, so the two never disagree.
assert resolver.describe_local_miss("unsloth/b-gguf:q8_0")[0] == (
resolver.MISS_VARIANT_NOT_FOUND
)
# Unknown repo, and a bare id with no ":VARIANT" to blame.
assert resolver.describe_local_miss("totally/unknown:Q8_0") == (
resolver.MISS_MODEL_NOT_FOUND,
(),
)
assert resolver.describe_local_miss("unsloth/B-GGUF") == (resolver.MISS_MODEL_NOT_FOUND, ())
def test_describe_local_miss_is_failsafe(monkeypatch):
# Runs inside an error path, so a broken scan must degrade, not turn a 4xx into a 500.
def boom():
raise RuntimeError("scan blew up")
monkeypatch.setattr(resolver, "_build_index", boom)
resolver._scan = (0.0, {})
assert resolver.describe_local_miss("unsloth/B-GGUF:Q8_0") == (
resolver.MISS_MODEL_NOT_FOUND,
(),
)
assert resolver.describe_local_miss(123) == (resolver.MISS_MODEL_NOT_FOUND, ())
assert resolver.describe_local_miss("") == (resolver.MISS_MODEL_NOT_FOUND, ())
def test_resolver_exact_id_with_colon_wins(monkeypatch):
# A local id that itself contains a colon (e.g. a Windows path) must match
# exactly rather than being split at the drive-letter colon.
@ -537,7 +594,9 @@ def test_disabling_idle_unload_purges_saved_kv(monkeypatch, tmp_path):
"dir": str(tmp_path),
"slots": [{"id": 0, "filename": saved.name}],
}
monkeypatch.setattr(settings_route, "set_openai_auto_switch", lambda *a: (False, 300, True))
monkeypatch.setattr(
settings_route, "set_openai_auto_switch", lambda *a: (False, 300, True, False)
)
monkeypatch.setattr(settings_route, "get_auto_unload_idle_seconds", lambda: 0)
payload = settings_route.OpenAIAutoSwitchPayload(enabled = False)
@ -1877,7 +1936,10 @@ def test_env_idle_standalone_reloads_freed_model_with_auto_switch_off(monkeypatc
monkeypatch.setattr(settings, "get_auto_unload_idle_seconds", lambda: 600) # standalone env TTL
monkeypatch.setattr(kw, "_inflight", 0)
monkeypatch.setattr(kw, "_last_unloaded_model", ("/cache/snap/A", "Q4_K_M", "org/A-GGUF"))
_run_hook("org/B-GGUF")
# A is restored, but the request named B, so it is told so rather than served A.
with pytest.raises(HTTPException) as excinfo:
_run_hook("org/B-GGUF")
assert excinfo.value.status_code == 404
# Resolver skipped (auto-switch off), so only the stash reload runs: the freed A
# is restored, not the resolves_to target B.
assert len(rec.calls) == 1
@ -2947,9 +3009,13 @@ def test_require_vision_ignores_reload_stash(monkeypatch):
monkeypatch.setattr(
inference_route, "_target_is_vision", lambda _p: False
) # would reject if used
asyncio.run(
inference_route._maybe_auto_switch_model("org/B-GGUF", object(), "t", require_vision = True)
)
# 404 because the restored A is not the requested B, whose quant makes it a real reference.
with pytest.raises(HTTPException):
asyncio.run(
inference_route._maybe_auto_switch_model(
"org/B-GGUF:UD-Q6_K_XL", object(), "t", require_vision = True
)
)
assert len(rec.calls) == 1
assert rec.calls[0].model_path == "/cache/snap/A" # restored despite require_vision
@ -3290,13 +3356,19 @@ def test_no_model_loaded_detail_appends_hint_only_when_off(monkeypatch):
assert inference_route._no_model_loaded_detail(base) == base
def _run_responses_stream_no_model(monkeypatch, *, enabled, active_model_name):
# Drive _responses_stream's GGUF-not-loaded guard: llama backend unloaded,
# inference backend maybe holding a non-GGUF model. Returns the 400 detail.
def _run_responses_stream_no_model(
monkeypatch,
*,
enabled,
active_model_name,
resolves_to = None,
):
# Drive _responses_stream's GGUF-not-loaded guard. Returns (status, detail).
from fastapi import HTTPException
from models.inference import ResponsesRequest, ChatMessage
monkeypatch.setattr(settings, "get_openai_auto_switch_enabled", lambda: enabled)
monkeypatch.setattr(resolver, "resolve_local_gguf", lambda name: resolves_to)
monkeypatch.setattr(
inference_route, "get_llama_cpp_backend", lambda: _FakeBackend(loaded_id = None)
)
@ -3309,29 +3381,230 @@ def _run_responses_stream_no_model(monkeypatch, *, enabled, active_model_name):
messages = [ChatMessage(role = "user", content = "hi")]
with pytest.raises(HTTPException) as exc:
asyncio.run(inference_route._responses_stream(payload, messages, None))
assert exc.value.status_code == 400
return exc.value.detail
return exc.value.status_code, exc.value.detail
def test_responses_stream_hint_matches_toggle_regardless_of_active_model(monkeypatch):
# Streaming /v1/responses shares the GGUF-only 400 with the other "no model
# loaded" sites, so the auto-switch hint attaches whenever the toggle is
# off -- including while a non-GGUF model is active, since auto-switch
# evicts it to load a resolved GGUF (_maybe_auto_switch_model's resolver
# branch has no active-model guard, unlike its reload-stash branch). Only
# the toggle being on suppresses it.
hinted = _run_responses_stream_no_model(monkeypatch, enabled = False, active_model_name = None)
# The hint attaches whenever the toggle is off, whatever is active. With it on the name
# resolved to nothing local, so 404 rather than 400.
off_status, hinted = _run_responses_stream_no_model(
monkeypatch, enabled = False, active_model_name = None
)
assert off_status == 400
assert "Model auto-switch" in hinted
on = _run_responses_stream_no_model(monkeypatch, enabled = True, active_model_name = None)
on_status, on = _run_responses_stream_no_model(
monkeypatch, enabled = True, active_model_name = None
)
assert on_status == 404
assert "Model auto-switch" not in on
assert "unsloth/Qwen3.5-4B-GGUF" in on
non_gguf_loaded = _run_responses_stream_no_model(
non_gguf_status, non_gguf_loaded = _run_responses_stream_no_model(
monkeypatch, enabled = False, active_model_name = "unsloth/Llama-3.2-1B-Instruct"
)
assert non_gguf_status == 400
assert "Model auto-switch" in non_gguf_loaded
def _wire_unloaded_chat(
monkeypatch,
*,
enabled,
catalog = ("org/A-GGUF", "org/B-GGUF"),
):
# Nothing loaded, so a chat request hits "no model loaded". Pin the catalog for determinism.
async def _catalog():
return [{"id": mid} for mid in catalog]
monkeypatch.setattr(settings, "get_openai_auto_switch_enabled", lambda: enabled)
monkeypatch.setattr(resolver, "resolve_local_gguf", lambda _m, **_kw: None)
monkeypatch.setattr(
resolver, "describe_local_miss", lambda _m: (resolver.MISS_MODEL_NOT_FOUND, ())
)
monkeypatch.setattr(inference_route, "_openai_catalog_objects", _catalog)
monkeypatch.setattr(
inference_route, "get_llama_cpp_backend", lambda: _FakeBackend(loaded_id = None)
)
monkeypatch.setattr(
inference_route,
"get_inference_backend",
lambda: type("_B", (), {"active_model_name": None, "models": {}})(),
)
def _chat_error(payload):
from fastapi import HTTPException
with pytest.raises(HTTPException) as exc:
asyncio.run(inference_route.openai_chat_completions(payload, object(), "tester"))
return exc.value.status_code, exc.value.detail
def test_chat_names_undownloaded_model_404s_with_available_ids(monkeypatch):
# The reported bug: the model is not here, so the switch did nothing and /inference/load
# cannot fix it. Name it and list what can serve.
_wire_unloaded_chat(monkeypatch, enabled = True)
status, detail = _chat_error(_chat_request(model = "unsloth/gemma-4-E4B-it-GGUF:UD-Q5_K_XL"))
assert status == 404
assert "unsloth/gemma-4-E4B-it-GGUF:UD-Q5_K_XL" in detail
assert "org/A-GGUF, org/B-GGUF" in detail
assert "GET /v1/models" in detail
assert "POST /inference/load" not in detail
def test_chat_undownloaded_model_with_empty_catalog(monkeypatch):
# Nothing downloaded: an empty list would read as a bug, so say so plainly.
_wire_unloaded_chat(monkeypatch, enabled = True, catalog = ())
status, detail = _chat_error(_chat_request(model = "org/nope-GGUF"))
assert status == 404
assert "no models are downloaded yet" in detail
def test_chat_wrong_quant_lists_the_local_quants(monkeypatch):
# Repo downloaded, only the quant missing: sibling quants, not the catalog.
_wire_unloaded_chat(monkeypatch, enabled = True)
monkeypatch.setattr(
resolver,
"describe_local_miss",
lambda _m: (resolver.MISS_VARIANT_NOT_FOUND, ("Q4_K_M", "Q8_0")),
)
status, detail = _chat_error(_chat_request(model = "org/A-GGUF:UD-Q5_K_XL"))
assert status == 404
assert "'org/A-GGUF' is downloaded, but the quant 'UD-Q5_K_XL' is not" in detail
assert "Q4_K_M, Q8_0" in detail
def test_chat_error_unchanged_when_auto_switch_off(monkeypatch):
# Toggle off: nothing resolved, so keep the pre-existing status and text, hint included.
_wire_unloaded_chat(monkeypatch, enabled = False)
status, detail = _chat_error(_chat_request(model = "org/nope-GGUF"))
assert status == 400
assert detail.startswith("No model loaded. Call POST /inference/load first.")
assert "Model auto-switch" in detail
def test_chat_error_unchanged_when_no_model_named(monkeypatch):
# An omitted model means "serve whatever is loaded", so there is no name to report.
_wire_unloaded_chat(monkeypatch, enabled = True)
status, detail = _chat_error(_chat_request())
assert status == 400
assert detail == "No model loaded. Call POST /inference/load first."
def test_chat_not_downloaded_error_survives_a_broken_catalog_scan(monkeypatch):
# Layered onto an already-failing path, so a broken scan must not make it a 500.
async def _boom():
raise RuntimeError("catalog scan blew up")
_wire_unloaded_chat(monkeypatch, enabled = True)
monkeypatch.setattr(inference_route, "_openai_catalog_objects", _boom)
status, detail = _chat_error(_chat_request(model = "org/nope-GGUF"))
assert status == 400
assert detail.startswith("No model loaded. Call POST /inference/load first.")
def test_chat_available_id_list_is_capped(monkeypatch):
# A machine with 40 GGUFs must not print all 40 into a terminal error.
_wire_unloaded_chat(
monkeypatch, enabled = True, catalog = tuple(f"org/m{i:02d}-GGUF" for i in range(20))
)
status, detail = _chat_error(_chat_request(model = "org/nope-GGUF"))
assert status == 404
assert "and 12 more" in detail
assert "org/m08-GGUF" not in detail
def test_anthropic_undownloaded_model_uses_the_anthropic_envelope(monkeypatch):
# Shared with /v1/messages, so the 404 must not leak an OpenAI-shaped body.
from fastapi import HTTPException
async def _noop_switch(*a, **k):
return None
_wire_unloaded_chat(monkeypatch, enabled = True)
monkeypatch.setattr(inference_route, "_automatic_model_load_may_run", lambda: True)
monkeypatch.setattr(inference_route, "_maybe_auto_switch_model", _noop_switch)
request = type("_R", (), {"url": type("_U", (), {"path": "/v1/messages"})()})()
with pytest.raises(HTTPException) as exc:
asyncio.run(inference_route.anthropic_messages(_anthropic_payload(64), request, "tester"))
assert exc.value.status_code == 404
body = exc.value.detail
assert body["type"] == "error"
assert body["error"]["type"] == "not_found_error"
assert "claude-x" in body["error"]["message"]
def test_chat_undownloaded_model_uses_the_openai_envelope(monkeypatch):
# The OpenAI surface carries param/code so SDK clients can branch on it.
from fastapi import HTTPException
_wire_unloaded_chat(monkeypatch, enabled = True)
request = type("_R", (), {"url": type("_U", (), {"path": "/v1/chat/completions"})()})()
with pytest.raises(HTTPException) as exc:
asyncio.run(
inference_route.openai_chat_completions(
_chat_request(model = "org/nope-GGUF"), request, "tester"
)
)
assert exc.value.status_code == 404
err = exc.value.detail["error"]
assert err["type"] == "not_found_error"
assert err["code"] == "model_not_found"
assert err["param"] == "model"
def test_gguf_only_paths_keep_the_generic_error_for_the_resident_non_gguf_model(monkeypatch):
# resolve_local_gguf misses a resident Transformers model the catalog does list, so
# "not downloaded" would contradict itself.
resident = "unsloth/Qwen3.5-4B-GGUF" # the id _run_responses_stream_no_model asks for
async def _catalog():
return [{"id": resident}]
monkeypatch.setattr(inference_route, "_openai_catalog_objects", _catalog)
status, detail = _run_responses_stream_no_model(
monkeypatch, enabled = True, active_model_name = resident
)
assert status == 400
assert "requires a GGUF model" in detail
assert "not downloaded" not in detail
def test_completions_keeps_the_generic_error_for_the_resident_non_gguf_model(monkeypatch):
# Same contradiction on the raw-body surface, via _auto_switch_from_request_body.
from fastapi import HTTPException
resident = "unsloth/Llama-3.2-1B-Instruct"
_wire_unloaded_chat(monkeypatch, enabled = True, catalog = (resident,))
monkeypatch.setattr(
inference_route,
"get_inference_backend",
lambda: type("_B", (), {"active_model_name": resident, "models": {}})(),
)
with pytest.raises(HTTPException) as exc:
asyncio.run(
inference_route.openai_completions(
_json_body_request({"model": resident, "prompt": "hi"}), "tester"
)
)
assert exc.value.status_code == 503
assert exc.value.detail.startswith("No GGUF model loaded.")
assert "not downloaded" not in exc.value.detail
def test_responses_stream_keeps_generic_error_when_target_is_local(monkeypatch):
# Resolves locally yet nothing is loaded: the switch failed, so keep the generic 400.
status, detail = _run_responses_stream_no_model(
monkeypatch,
enabled = True,
active_model_name = None,
resolves_to = ("/p/A", "Q4_K_M", "unsloth/Qwen3.5-4B-GGUF"),
)
assert status == 400
assert "not downloaded" not in detail
# ── idle-unload KV persistence (slot save/restore) ──────────────────
@ -3784,10 +4057,11 @@ def test_keep_kv_only_update_leaves_env_idle_ttl_active(monkeypatch):
monkeypatch.setenv(settings.MODEL_IDLE_TTL_ENV_VAR, "600")
assert settings_route.OpenAIAutoSwitchPayload(enabled = False).auto_unload_idle_seconds is None
enabled, idle, keep_kv = settings.set_openai_auto_switch(False, None, False)
enabled, idle, keep_kv, auto_dl = settings.set_openai_auto_switch(False, None, False)
assert settings.AUTO_UNLOAD_IDLE_SETTING_KEY not in store # idle untouched
assert settings.OPENAI_AUTO_DOWNLOAD_SETTING_KEY not in store # nor auto-download
assert settings.get_auto_unload_idle_seconds() == 600 # env TTL still active
assert (enabled, idle, keep_kv) == (False, 600, False)
assert (enabled, idle, keep_kv, auto_dl) == (False, 600, False, False)
def test_load_impl_notes_loaded_with_backend_off_loop():
@ -3869,3 +4143,238 @@ def test_env_idle_below_floor_is_clamped(monkeypatch):
assert settings.get_auto_unload_idle_seconds() == 600
monkeypatch.delenv(settings.MODEL_IDLE_TTL_ENV_VAR)
assert settings.get_auto_unload_idle_seconds() == 0
def test_a_tag_that_names_no_quant_resolves_to_the_repo(monkeypatch):
# A downloaded but unloaded GGUF asked for as org/model:latest missed the
# resolver, so the switch path could not load it: with auto-download on it
# probed the Hub and 404d on a quant that was never a quant, and with it off it
# refused without switching. A real quant that is not on disk must still miss,
# or a swap would serve the wrong weights under the right name.
from core.inference.local_model_resolver import _LocalGgufEntry
import time
entry = _LocalGgufEntry("org/model", "/srv/models/org--model", ("Q4_K_M",))
# Fresh stamp so _index serves this instead of rescanning over it.
monkeypatch.setattr(resolver, "_scan", (time.monotonic(), {"org/model": entry}))
for tag in ("org/model:latest", "org/model:8b", "org/model"):
assert resolver.resolve_local_gguf(tag) == (
"/srv/models/org--model",
"Q4_K_M",
"org/model",
)
assert resolver.resolve_local_gguf("org/model:Q8_0") is None
assert resolver.resolve_local_gguf("org/model:Q4_K_M") == (
"/srv/models/org--model",
"Q4_K_M",
"org/model",
)
def test_any_finished_download_drops_the_resolver_cache(monkeypatch):
# Only the API auto-download watcher invalidated, so a GGUF fetched in the Hub
# UI stayed absent to the cache-only request path and the request was answered
# by the resident model instead. Every worker exits through here.
import logging
from hub.services import download_lifecycle
class _Proc:
stderr = None
def wait(self):
return 0
class _Registry:
def cancel_requested(self, key):
return False
def drop_process(self, key, proc):
return True
def get_job_metadata(self, key):
return None
def set_job(self, key, state):
self.state = state
resolver._scan = (1234.0, {"already-here": "entry"})
assert (
download_lifecycle.finalize_worker_exit(
_Registry(),
"org/model:Q4_K_M",
_Proc(),
hf_token = None,
label = "org/model",
log_prefix = "[test]",
logger = logging.getLogger(__name__),
repo_type = "model",
repo_id = "org/model",
)
== "complete"
)
stamp, entries = resolver._scan
assert stamp == 0.0, "a finished download left the scan looking fresh"
# Evidence for models already indexed has to survive, or a bare request for one
# of them during the rebuild is answered by whatever is resident.
assert entries == {"already-here": "entry"}
def test_invalidating_keeps_the_entries_it_already_had(monkeypatch):
# The request path reads this cache without scanning, so emptying it leaves it
# with no evidence about any local model until the rebuild lands. Only a
# completed download invalidates, and that only adds, so the entries stay true.
import time
entry = resolver._LocalGgufEntry("org/old", "/srv/models/org--old", ("Q4_K_M",))
monkeypatch.setattr(resolver, "_scan", (time.monotonic(), {"org/old": entry}))
resolver.invalidate_index()
assert resolver._scan[0] == 0.0
assert resolver.resolve_local_gguf("org/old", allow_scan = False) == (
"/srv/models/org--old",
"Q4_K_M",
"org/old",
)
def test_a_bare_local_id_takes_the_quant_a_plain_load_would(monkeypatch, tmp_path):
# list_local_gguf_variants orders by descending size, so the head is the biggest
# quant. Resolving a bare id to that could evict a working model and then OOM
# starting an F16 on a box sized for the Q4 sitting right next to it, and
# /v1/models advertised the same head for pinning.
from core.inference.local_model_resolver import _local_gguf_entry
for name, size in (("model-F16.gguf", 900), ("model-Q4_K_M.gguf", 100)):
(tmp_path / name).write_bytes(b"\0" * size)
entry = _local_gguf_entry("org/model", type("I", (), {"path": str(tmp_path)})())
assert entry is not None
assert set(entry.variants) == {"F16", "Q4_K_M"}
assert entry.variants[0] == "Q4_K_M", "a bare id would have resolved to F16"
def test_local_and_remote_agree_on_the_preferred_quant():
# A bare id must mean the same quant whichever side answered it.
from core.inference.openai_auto_download import _match_variant, preferred_quant
labels = ("F16", "Q8_0", "UD-Q4_K_XL", "Q4_K_M")
assert preferred_quant(labels) == _match_variant(None, dict.fromkeys(labels, 1))
assert preferred_quant(labels) not in ("F16",)
def test_a_just_downloaded_model_is_evidence_before_the_scan_indexes_it(monkeypatch):
# Retaining the old index covers what was already known, but nothing covers the
# model that just landed until the next scan finishes. A bare request for it in
# that window was answered by the unrelated resident model.
import logging
from hub.services import download_lifecycle
class _Proc:
stderr = None
def wait(self):
return 0
class _Registry:
def cancel_requested(self, key):
return False
def drop_process(self, key, proc):
return True
def get_job_metadata(self, key):
return None
def set_job(self, key, state):
pass
assert not resolver.recently_downloaded("org/fresh")
download_lifecycle.finalize_worker_exit(
_Registry(),
"org/fresh:Q4_K_M",
_Proc(),
hf_token = None,
label = "org/fresh",
log_prefix = "[test]",
logger = logging.getLogger(__name__),
repo_type = "model",
repo_id = "org/fresh",
)
assert resolver.recently_downloaded("org/fresh"), "no evidence for the new model"
assert resolver.recently_downloaded("ORG/Fresh"), "evidence must be case-insensitive"
assert not resolver.recently_downloaded("org/other")
# The scan that indexes it supersedes the note.
monkeypatch.setattr(resolver, "_build_index", dict)
resolver._index()
assert not resolver.recently_downloaded("org/fresh")
def test_a_finished_dataset_is_not_recorded_as_a_local_model(monkeypatch):
# finalize_worker_exit is shared with dataset downloads. Noting one as a local
# model would refuse a bare /v1 request naming that id while another model is
# resident, instead of letting a foreign id fall through, and would kick off a
# multi-directory model scan for nothing.
import logging
import time
from hub.services import download_lifecycle
class _Proc:
stderr = None
def wait(self):
return 0
class _Registry:
def cancel_requested(self, key):
return False
def drop_process(self, key, proc):
return True
def get_job_metadata(self, key):
return None
def set_job(self, key, state):
pass
stamp = time.monotonic()
monkeypatch.setattr(resolver, "_scan", (stamp, {"kept": "entry"}))
download_lifecycle.finalize_worker_exit(
_Registry(),
"org/corpus",
_Proc(),
hf_token = None,
label = "org/corpus",
log_prefix = "[test]",
logger = logging.getLogger(__name__),
repo_type = "dataset",
repo_id = "org/corpus",
)
assert not resolver.recently_downloaded("org/corpus")
assert resolver._scan == (stamp, {"kept": "entry"}), "a dataset invalidated the index"
def test_two_local_paths_differing_only_in_case_are_not_the_same_model(monkeypatch):
# _loaded_satisfies lowercased the request and every backend identifier, so on a
# case-sensitive filesystem /srv/models/foo.gguf counted as satisfied by a
# resident /srv/models/Foo.gguf and returned before the case-preserving compare
# further down ever ran. A repo alias must stay case-insensitive.
import os
loaded = _FakeBackend(loaded_id = "/srv/models/Foo.gguf")
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: loaded)
monkeypatch.setattr(
inference_route,
"get_inference_backend",
lambda: type("B", (), {"active_model_name": None})(),
)
assert inference_route._loaded_satisfies("/srv/models/Foo.gguf") is True
same = os.path.normcase("A") == os.path.normcase("a")
assert inference_route._loaded_satisfies("/srv/models/foo.gguf") is same
alias = _FakeBackend(loaded_id = "unsloth/Qwen3-4B-GGUF")
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: alias)
assert inference_route._loaded_satisfies("unsloth/qwen3-4b-gguf") is True

163
studio/backend/tests/test_openai_catalog.py
View file

@ -64,8 +64,10 @@ def test_catalog_lists_loaded_and_available(monkeypatch):
]
monkeypatch.setattr(inf, "_cached_local_catalog", _fake_catalog)
# GGUF-ness is read from the on-disk files; drive it off each info's flag here.
monkeypatch.setattr(resolver, "info_has_local_gguf", lambda info: info.is_gguf)
# GGUF-ness and the quant labels come from one on-disk scan; drive both off the flag.
monkeypatch.setattr(
resolver, "local_gguf_quants", lambda info: ("Q8_0",) if info.is_gguf else None
)
data = asyncio.run(inf._openai_catalog_objects())
ids = {m["id"]: m for m in data}
@ -73,8 +75,9 @@ def test_catalog_lists_loaded_and_available(monkeypatch):
# Loaded model is present, marked loaded, and keeps context fields.
assert ids["Qwen3-Q4"]["loaded"] is True
assert ids["Qwen3-Q4"]["context_length"] == 4096
# Available-but-not-loaded GGUF models are listed too.
# Not-loaded GGUFs are listed too, with the quant a client appends to pin them.
assert ids["Llama-8B-Q8"]["loaded"] is False
assert ids["Llama-8B-Q8"]["quant"] == "Q8_0"
# The HF-cache GGUF is listed despite model_format being unset.
assert ids["org/Foo"]["loaded"] is False
# The non-GGUF model is filtered out (/v1 can never serve it).
@ -205,3 +208,157 @@ def test_cached_local_catalog_offloads_and_caches(monkeypatch):
assert second is first or [i.id for i in second] == [i.id for i in first]
assert calls["scan"] == 1 # cached: scanned once for two calls
assert calls["threaded"] == 1 # offloaded to a worker thread
def test_monitor_active_model_is_a_public_id_not_a_host_path(monkeypatch):
# The settings UI renders this and --secure serves it publicly, so never a load path.
class _Llama:
is_loaded = True
model_identifier = "/home/me/.cache/huggingface/hub/models--org--A-GGUF/snapshots/abc"
hf_variant = "UD-Q4_K_XL"
_openai_advertised_id = "org/A-GGUF"
monkeypatch.setattr(inf, "get_llama_cpp_backend", lambda: _Llama())
assert inf._monitor_active_model() == "org/A-GGUF:UD-Q4_K_XL"
def test_monitor_active_model_cleans_a_path_with_no_advertised_id(monkeypatch):
class _Llama:
is_loaded = True
model_identifier = "/data/models/Llama-8B-Q8.gguf"
hf_variant = None
_openai_advertised_id = None
monkeypatch.setattr(inf, "get_llama_cpp_backend", lambda: _Llama())
label = inf._monitor_active_model()
assert "/" not in label and ".gguf" not in label
def test_lifecycle_label_recovers_the_repo_id_from_an_hf_cache_path():
# An auto-switch load gets the snapshot dir, whose basename is a commit sha.
snap = "/home/me/.cache/huggingface/hub/models--unsloth--gemma-4-E4B-it-GGUF/snapshots/bfc15c3"
assert (
inf._lifecycle_model_label(snap, "UD-Q4_K_XL") == "unsloth/gemma-4-E4B-it-GGUF:UD-Q4_K_XL"
)
def test_lifecycle_model_label_is_path_free():
label = inf._lifecycle_model_label("/data/models/Llama-8B-Q8.gguf", "Q8_0")
assert "/" not in label and ".gguf" not in label
assert inf._lifecycle_model_label("org/A-GGUF", "Q4_K_M") == "org/A-GGUF:Q4_K_M"
# An id that already carries a quant is not double-suffixed.
assert inf._lifecycle_model_label("org/A-GGUF:Q4_K_M", "Q8_0") == "org/A-GGUF:Q4_K_M"
def test_a_standalone_gguf_does_not_advertise_a_quant_that_stops_resolving(monkeypatch):
# llama.cpp reads hf_variant off the filename, but the resolver stores standalone files
# with no quants, so a pinned "<stem>:<quant>" would 404 once it is not resident.
from core.inference.local_model_resolver import _LocalGgufEntry
standalone = _LocalGgufEntry("Qwen3-Q4", "/srv/models/Qwen3-Q4.gguf", ())
repo = _LocalGgufEntry("org/Foo", "/hf/models--org--Foo/snapshots/a", ("Q4_K_M",))
monkeypatch.setattr(resolver, "_scan", (1.0, {"qwen3-q4": standalone, "org/foo": repo}))
monkeypatch.setattr(inf, "get_inference_backend", lambda: _FakeUnsloth())
llama = _FakeLlama()
llama.hf_variant = "Q4_K_M"
monkeypatch.setattr(inf, "get_llama_cpp_backend", lambda: llama)
assert "quant" not in inf._openai_model_objects()[0]
# The same quant on a repo the resolver does list stays advertised.
llama.model_identifier = "org/Foo"
assert inf._openai_model_objects()[0]["quant"] == "Q4_K_M"
# A cold index cannot prove the reference either, and publishing on no proof is
# exactly what hands out the pin that later fails to resolve.
monkeypatch.setattr(resolver, "_scan", (0.0, {}))
# Stub the walk: a real multi-root scan inside the cold-wait budget makes this
# test time out into a 503 under load instead of asserting what it is here for.
monkeypatch.setattr(resolver, "_build_index", lambda: {})
monkeypatch.setattr(resolver, "warm_index_soon", lambda: None)
assert "quant" not in inf._openai_model_objects()[0]
def test_a_loaded_alias_advertises_the_quant_that_is_actually_loaded(monkeypatch):
# Marking the alias loaded while still publishing the preferred on-disk quant said
# alias:Q4 was loaded while Q8 was serving, and pinning that 404s with switching off.
monkeypatch.setattr(inf, "get_inference_backend", lambda: _FakeUnsloth())
llama = _FakeLlama()
llama.hf_variant = "Q8_0"
monkeypatch.setattr(inf, "get_llama_cpp_backend", lambda: llama)
alias = _Info("/srv/models", "Qwen3", model_id = "publisher/Qwen3")
alias.path = "/srv/models" # holds the resident /srv/models/Qwen3-Q4.gguf
async def _fake_catalog():
return [alias]
monkeypatch.setattr(inf, "_cached_local_catalog", _fake_catalog)
monkeypatch.setattr(resolver, "local_gguf_quants", lambda info: ("Q4_K_M", "Q8_0"))
ids = {m["id"]: m for m in asyncio.run(inf._openai_catalog_objects())}
assert ids["publisher/Qwen3"]["loaded"] is True
assert ids["publisher/Qwen3"]["quant"] == "Q8_0"
def test_a_nested_model_directory_is_not_the_resident_one(monkeypatch):
# Two separately indexed models can nest (/models/A holding A, /models/A/sub/B
# holding B). A plain prefix test made loading B mark A resident, so a request for
# A was answered with B's weights. The innermost indexed model owns the file.
outer = _Info("/models/A", "A", model_id = "publisher/A")
outer.path = "/models/A"
inner = _Info("/models/A/sub/B", "B", model_id = "publisher/B")
inner.path = "/models/A/sub/B"
monkeypatch.setitem(inf._CATALOG_CACHE, "models", [outer, inner])
llama = _FakeLlama()
llama.gguf_path = "/models/A/sub/B/model-Q4_K_M.gguf"
llama.model_identifier = llama.gguf_path
monkeypatch.setattr(inf, "get_llama_cpp_backend", lambda: llama)
monkeypatch.setattr(inf, "get_inference_backend", lambda: _FakeUnsloth())
assert inf._resolves_to_resident("/models/A/sub/B") is True
assert inf._resolves_to_resident("/models/A") is False
# With nothing indexed there is no nesting to tell apart, so the directory-to-file
# match this exists for must still hold.
monkeypatch.setitem(inf._CATALOG_CACHE, "models", [])
assert inf._resolves_to_resident("/models/A") is True
def test_a_transformers_model_does_not_mark_a_gguf_alias_loaded(monkeypatch):
# Every entry in this loop is advertised as GGUF and carries a GGUF quant. A
# Transformers model live from a directory that also holds GGUF exports is not one
# of them, and marking the alias loaded had the usage examples pin alias:quant that
# nothing can serve while switching is off.
unsloth = _FakeUnsloth()
unsloth.active_model_name = "/srv/models"
monkeypatch.setattr(inf, "get_inference_backend", lambda: unsloth)
monkeypatch.setattr(inf, "get_llama_cpp_backend", lambda: _FakeLlama(loaded = False))
alias = _Info("/srv/models", "Qwen3", model_id = "publisher/Qwen3")
alias.path = "/srv/models" # also holds /srv/models/Qwen3-Q4.gguf
async def _fake_catalog():
return [alias]
monkeypatch.setattr(inf, "_cached_local_catalog", _fake_catalog)
monkeypatch.setattr(resolver, "local_gguf_quants", lambda info: ("Q4_K_M",))
ids = {m["id"]: m for m in asyncio.run(inf._openai_catalog_objects())}
assert ids["publisher/Qwen3"]["loaded"] is False
def test_an_alias_for_the_resident_weights_is_not_listed_as_unloaded(monkeypatch):
# A GGUF loaded by absolute path keys the resident entry by basename, so an id-only dedup
# would emit the alias again marked not loaded.
monkeypatch.setattr(inf, "get_llama_cpp_backend", lambda: _FakeLlama())
monkeypatch.setattr(inf, "get_inference_backend", lambda: _FakeUnsloth())
alias = _Info("/srv/models", "Qwen3", model_id = "publisher/Qwen3")
alias.path = "/srv/models" # holds the resident /srv/models/Qwen3-Q4.gguf
async def _fake_catalog():
return [alias]
monkeypatch.setattr(inf, "_cached_local_catalog", _fake_catalog)
monkeypatch.setattr(resolver, "local_gguf_quants", lambda info: ("Q4_K_M",))
ids = {m["id"]: m for m in asyncio.run(inf._openai_catalog_objects())}
assert ids["publisher/Qwen3"]["loaded"] is True

11
studio/backend/tests/test_openai_tool_passthrough.py
View file

@ -1611,7 +1611,7 @@ class TestOpenAICompatibilityHelpers:
def test_openai_stream_error_sse_closes_with_done(self):
error = {"error": {"message": "boom"}}
assert _openai_stream_error_sse(error) == (
'data: {"error": {"message": "boom"}}\n\n' "data: [DONE]\n\n"
'data: {"error": {"message": "boom"}}\n\ndata: [DONE]\n\n'
)
@pytest.mark.parametrize(
@ -6473,7 +6473,14 @@ class TestApiMonitorSafetensorsUsage:
nonlocal reset_called
reset_called = True
async def fake_to_thread(*_args, **_kwargs):
async def fake_to_thread(
func = None,
*_args,
**_kwargs,
):
# Only the generation hop should cancel; resolution runs before the row opens.
if getattr(func, "__name__", "") == "resolve_local_gguf":
return None
raise asyncio.CancelledError()
monitor = ApiMonitor(max_entries = 3)

33
studio/backend/tests/test_training_worker_flash_attn.py
View file

@ -209,7 +209,27 @@ def _force_missing_fla_imports(monkeypatch):
monkeypatch.setattr(builtins, "__import__", fake_import)
def _pin_fla_model_types(monkeypatch):
"""Pin the auto-discovered FLA allowlist to the Qwen GDN families.
`_discover_fla_model_types` scans the *installed* transformers for modeling
files importing `from fla.`, and `models/qwen3_5/` only exists from
transformers 5.x. The backend supports `transformers>=4.51`, so on a 4.x
install the gate returns False and every Qwen3.5 assertion below silently
passes through a no-op instead of exercising the install path. Pinning keeps
these tests hermetic across the whole supported transformers range, the same
way test_hook_does_not_install_tilelang_for_model_outside_allowlist pins it
against newly added FLA model_types.
"""
monkeypatch.setattr(
worker,
"_discover_fla_model_types",
lambda: frozenset({"qwen3_5", "qwen3_5_moe", "qwen3_6", "qwen3_next"}),
)
def test_flash_linear_attention_installs_pinned_pair_for_qwen3_5(monkeypatch):
_pin_fla_model_types(monkeypatch)
monkeypatch.setattr(worker.shutil, "which", lambda name: "/usr/bin/uv")
run_mock = mock.Mock(return_value = mock.Mock(returncode = 0, stdout = ""))
monkeypatch.setattr(worker._sp, "run", run_mock)
@ -315,6 +335,7 @@ def test_flash_linear_attention_skipped_via_env(monkeypatch):
def test_flash_linear_attention_skipped_below_torch_2_7(monkeypatch):
_pin_fla_model_types(monkeypatch)
monkeypatch.delenv(worker._FLA_SKIP_ENV, raising = False)
monkeypatch.setattr(worker, "_installed_torch_version_tuple", lambda: (2, 5))
run_mock = mock.Mock(return_value = mock.Mock(returncode = 0, stdout = ""))
@ -332,6 +353,7 @@ def test_flash_linear_attention_skipped_below_torch_2_7(monkeypatch):
def test_flash_linear_attention_install_includes_einops(monkeypatch):
_pin_fla_model_types(monkeypatch)
monkeypatch.delenv(worker._FLA_SKIP_ENV, raising = False)
monkeypatch.setattr(worker.shutil, "which", lambda name: "/usr/bin/uv")
monkeypatch.setattr(worker, "_installed_torch_version_tuple", lambda: (2, 9))
@ -358,6 +380,7 @@ def test_flash_linear_attention_install_includes_einops(monkeypatch):
def test_flash_linear_attention_logs_post_install_import_failure(monkeypatch):
"""pip exits 0 but `import fla.modules` still fails (missing transitive)."""
_pin_fla_model_types(monkeypatch)
monkeypatch.delenv(worker._FLA_SKIP_ENV, raising = False)
monkeypatch.setattr(worker.shutil, "which", lambda name: "/usr/bin/uv")
monkeypatch.setattr(worker, "_installed_torch_version_tuple", lambda: (2, 9))
@ -402,6 +425,7 @@ def test_tilelang_backend_skipped_on_unsupported_linux_arch(monkeypatch):
def test_tilelang_backend_pins_only_binary(monkeypatch):
_pin_fla_model_types(monkeypatch)
monkeypatch.delenv(worker._TILELANG_SKIP_ENV, raising = False)
monkeypatch.setattr(worker.shutil, "which", lambda name: "/usr/bin/uv")
monkeypatch.setattr(worker, "_installed_tvm_ffi_version", lambda: None)
@ -442,6 +466,7 @@ def _force_missing_tilelang_imports(monkeypatch):
def test_tilelang_backend_installs_pinned_pair_for_qwen3_5(monkeypatch):
_pin_fla_model_types(monkeypatch)
monkeypatch.delenv(worker._TILELANG_SKIP_ENV, raising = False)
monkeypatch.setattr(worker.shutil, "which", lambda name: "/usr/bin/uv")
monkeypatch.setattr(worker, "_installed_tvm_ffi_version", lambda: None)
@ -472,6 +497,7 @@ def test_tilelang_backend_reinstalls_when_tvm_ffi_is_broken(monkeypatch):
2 (install): plain apache-tvm-ffi + tilelang -- resolves missing transitive
deps without --force-reinstall, so it never replaces correct packages.
"""
_pin_fla_model_types(monkeypatch)
monkeypatch.delenv(worker._TILELANG_SKIP_ENV, raising = False)
monkeypatch.setattr(worker.shutil, "which", lambda name: "/usr/bin/uv")
monkeypatch.setattr(worker, "_installed_tvm_ffi_version", lambda: "0.1.11")
@ -533,6 +559,7 @@ def test_tilelang_backend_skipped_on_windows(monkeypatch):
def test_tilelang_backend_swallows_install_timeout(monkeypatch):
_pin_fla_model_types(monkeypatch)
monkeypatch.delenv(worker._TILELANG_SKIP_ENV, raising = False)
monkeypatch.setattr(worker.shutil, "which", lambda name: "/usr/bin/uv")
monkeypatch.setattr(worker, "_installed_tvm_ffi_version", lambda: None)
@ -586,6 +613,7 @@ def test_tilelang_backend_skipped_via_env(monkeypatch):
def test_tilelang_backend_swallows_install_failure(monkeypatch):
_pin_fla_model_types(monkeypatch)
monkeypatch.delenv(worker._TILELANG_SKIP_ENV, raising = False)
monkeypatch.setattr(worker.shutil, "which", lambda name: None)
monkeypatch.setattr(worker, "_installed_tvm_ffi_version", lambda: None)
@ -649,6 +677,7 @@ def _patch_iu_gates(monkeypatch, fla_gate, conv_gate):
def test_hook_installs_when_gate_returns_false(monkeypatch):
_pin_fla_model_types(monkeypatch)
fla_gate = _make_fake_gate(initial_return = False)
conv_gate = _make_fake_gate(initial_return = False)
_patch_iu_gates(monkeypatch, fla_gate, conv_gate)
@ -716,6 +745,7 @@ def test_hook_skips_install_when_gate_already_true(monkeypatch):
def test_hook_idempotent_on_repeat_call(monkeypatch):
_pin_fla_model_types(monkeypatch)
fla_gate = _make_fake_gate(initial_return = False)
conv_gate = _make_fake_gate(initial_return = False)
_patch_iu_gates(monkeypatch, fla_gate, conv_gate)
@ -924,6 +954,7 @@ def test_hook_does_not_install_tilelang_for_model_outside_allowlist(monkeypatch)
def test_hook_does_install_tilelang_for_qwen35(monkeypatch):
"""Positive control for finding #1: Qwen3.5 still gets tilelang."""
_pin_fla_model_types(monkeypatch)
fla_gate = _make_fake_gate(initial_return = False)
conv_gate = _make_fake_gate(initial_return = True)
_patch_iu_gates(monkeypatch, fla_gate, conv_gate)
@ -953,6 +984,7 @@ def test_tilelang_repair_does_not_touch_torch_cuda_stack(monkeypatch):
forced step so --force-reinstall doesn't cascade through
apache-tvm-ffi's dep graph and pull a different torch wheel.
"""
_pin_fla_model_types(monkeypatch)
monkeypatch.delenv(worker._TILELANG_SKIP_ENV, raising = False)
monkeypatch.setattr(worker.shutil, "which", lambda name: "/usr/bin/uv")
monkeypatch.setattr(worker, "_installed_tvm_ffi_version", lambda: "0.1.10")
@ -1065,6 +1097,7 @@ def test_hook_runs_tilelang_repair_when_fla_already_true(monkeypatch):
probe) but tilelang is missing or apache-tvm-ffi is on the broken
list, the post-available action must still run tilelang.
"""
_pin_fla_model_types(monkeypatch)
fla_gate = _make_fake_gate(initial_return = True)
conv_gate = _make_fake_gate(initial_return = True)
_patch_iu_gates(monkeypatch, fla_gate, conv_gate)

44
studio/backend/utils/openai_auto_switch_settings.py
View file

@ -3,10 +3,14 @@
"""Persisted opt-in controls for OpenAI-compatible model auto-switching.
Two settings, both off by default so existing API behavior is unchanged:
All off by default so existing API behavior is unchanged:
- ``openai_api_auto_switch_model``: when on, a ``/v1`` request whose ``model``
names a downloaded local GGUF different from the loaded one transparently
loads it before serving (llama-swap-style). Unknown names pass through.
- ``openai_api_auto_download_model``: when on (and auto-switch is too), a
``/v1`` request naming a GGUF repo that is *not* downloaded starts a
background download instead of failing. Gated on auto-switch, which is what
serves the model once it lands.
- ``openai_api_auto_unload_idle_seconds``: when > 0, the loaded GGUF is
unloaded after this many idle seconds to free VRAM. Enabled values have a
60s floor (0 stays "off"): a tiny TTL tears the model down between turns of
@ -29,12 +33,14 @@ import time
from typing import Any, Optional
OPENAI_AUTO_SWITCH_SETTING_KEY = "openai_api_auto_switch_model"
OPENAI_AUTO_DOWNLOAD_SETTING_KEY = "openai_api_auto_download_model"
AUTO_UNLOAD_IDLE_SETTING_KEY = "openai_api_auto_unload_idle_seconds"
AUTO_UNLOAD_KEEP_KV_SETTING_KEY = "openai_api_auto_unload_keep_kv"
MODEL_OVERRIDES_SETTING_KEY = "openai_api_auto_switch_overrides"
MODEL_IDLE_TTL_ENV_VAR = "UNSLOTH_MODEL_IDLE_TTL"
DEFAULT_OPENAI_AUTO_SWITCH_ENABLED = False
DEFAULT_OPENAI_AUTO_DOWNLOAD_ENABLED = False
DEFAULT_AUTO_UNLOAD_IDLE_SECONDS = 0
DEFAULT_AUTO_UNLOAD_KEEP_KV = True
MIN_AUTO_UNLOAD_IDLE_SECONDS = 60
@ -95,6 +101,25 @@ def get_openai_auto_switch_enabled() -> bool:
return parsed if parsed is not None else DEFAULT_OPENAI_AUTO_SWITCH_ENABLED
def get_stored_openai_auto_download_enabled() -> bool:
"""The persisted auto-download flag, independent of auto-switch.
The settings UI reads this so toggling auto-switch off displays and
round-trips the saved value rather than erasing it.
"""
parsed = _coerce_bool(_cached_setting(OPENAI_AUTO_DOWNLOAD_SETTING_KEY, None))
return parsed if parsed is not None else DEFAULT_OPENAI_AUTO_DOWNLOAD_ENABLED
def get_openai_auto_download_enabled() -> bool:
"""Whether a /v1 request may download a GGUF repo it names but doesn't have.
Gated on auto-switch: auto-switch is what loads the model once it lands, so
downloading without it would fetch gigabytes nothing can then serve.
"""
return get_stored_openai_auto_download_enabled() and get_openai_auto_switch_enabled()
def _stored_idle_seconds() -> Optional[int]:
"""The persisted idle TTL as an int, or None when never set."""
return _coerce_int(_cached_setting(AUTO_UNLOAD_IDLE_SETTING_KEY, None))
@ -170,7 +195,8 @@ def set_openai_auto_switch(
enabled: Any,
idle_seconds: Any,
keep_kv: Any = None,
) -> tuple[bool, int, bool]:
auto_download: Any = None,
) -> tuple[bool, int, bool, bool]:
"""One-transaction write; ``None`` leaves a stored value untouched."""
parsed_enabled = _coerce_bool(enabled)
if parsed_enabled is None:
@ -190,6 +216,11 @@ def set_openai_auto_switch(
parsed_keep_kv = _coerce_bool(keep_kv)
if parsed_keep_kv is None:
raise ValueError("Keep KV on idle unload must be true or false.")
parsed_auto_download = None
if auto_download is not None:
parsed_auto_download = _coerce_bool(auto_download)
if parsed_auto_download is None:
raise ValueError("Auto-download missing models must be true or false.")
from storage.studio_db import upsert_app_settings
updates: dict[str, Any] = {OPENAI_AUTO_SWITCH_SETTING_KEY: parsed_enabled}
@ -197,16 +228,25 @@ def set_openai_auto_switch(
updates[AUTO_UNLOAD_IDLE_SETTING_KEY] = parsed_idle
if parsed_keep_kv is not None:
updates[AUTO_UNLOAD_KEEP_KV_SETTING_KEY] = parsed_keep_kv
if parsed_auto_download is not None:
updates[OPENAI_AUTO_DOWNLOAD_SETTING_KEY] = parsed_auto_download
upsert_app_settings(updates)
_invalidate(OPENAI_AUTO_SWITCH_SETTING_KEY)
if parsed_idle is not None:
_invalidate(AUTO_UNLOAD_IDLE_SETTING_KEY)
if parsed_keep_kv is not None:
_invalidate(AUTO_UNLOAD_KEEP_KV_SETTING_KEY)
if parsed_auto_download is not None:
_invalidate(OPENAI_AUTO_DOWNLOAD_SETTING_KEY)
return (
parsed_enabled,
parsed_idle if parsed_idle is not None else get_stored_auto_unload_idle_seconds(),
parsed_keep_kv if parsed_keep_kv is not None else get_auto_unload_keep_kv(),
(
parsed_auto_download
if parsed_auto_download is not None
else get_stored_openai_auto_download_enabled()
),
)

6
studio/frontend/src/features/chat/types/api.ts
View file

@ -291,6 +291,12 @@ export interface ApiMonitorEntry {
completion_tokens?: number | null;
total_tokens?: number | null;
error?: string | null;
// "lifecycle" is a model load/unload/download: event/reason instead of a prompt.
kind?: "request" | "lifecycle";
event?: "load" | "unload" | "download" | null;
reason?: "manual" | "idle" | "api" | null;
// 0-100 while a download row is running.
progress?: number | null;
}
export interface ApiMonitorResponse {

10
studio/frontend/src/features/settings/api/openai-auto-switch.ts
View file

@ -13,6 +13,8 @@ export type OpenAIAutoSwitchSettings = {
idleUnloadActive: boolean;
// Persist the KV cache to disk on idle unload and restore it on reload.
autoUnloadKeepKv: boolean;
// Fetch a GGUF named in an API request; stored independently of `enabled`, gated on it.
autoDownloadModel: boolean;
};
type ApiOpenAIAutoSwitchSettings = {
@ -25,6 +27,8 @@ type ApiOpenAIAutoSwitchSettings = {
idle_unload_active?: boolean;
// biome-ignore lint/style/useNamingConvention: API schema
auto_unload_keep_kv?: boolean;
// biome-ignore lint/style/useNamingConvention: API schema
auto_download_model?: boolean;
};
let cachedSettings: OpenAIAutoSwitchSettings | null = null;
@ -39,6 +43,7 @@ function fromApi(
defaultEnabled: settings.default_enabled,
idleUnloadActive: settings.idle_unload_active ?? false,
autoUnloadKeepKv: settings.auto_unload_keep_kv ?? true,
autoDownloadModel: settings.auto_download_model ?? false,
};
}
@ -73,6 +78,7 @@ export async function updateOpenAIAutoSwitchSettings(
enabled: boolean,
autoUnloadIdleSeconds?: number,
autoUnloadKeepKv?: boolean,
autoDownloadModel?: boolean,
): Promise<OpenAIAutoSwitchSettings> {
const res = await authFetch("/api/settings/openai-auto-switch", {
method: "PUT",
@ -88,6 +94,10 @@ export async function updateOpenAIAutoSwitchSettings(
? {}
: // biome-ignore lint/style/useNamingConvention: API schema
{ auto_unload_keep_kv: autoUnloadKeepKv }),
...(autoDownloadModel === undefined
? {}
: // biome-ignore lint/style/useNamingConvention: API schema
{ auto_download_model: autoDownloadModel }),
}),
});
if (!res.ok) {

42
studio/frontend/src/features/settings/api/openai-models.ts Normal file
View file

@ -0,0 +1,42 @@
// SPDX-License-Identifier: AGPL-3.0-only
// Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
import { authFetch } from "@/features/auth";
export type OpenAIModel = {
id: string;
// Resident in memory now; the rest are downloaded and servable.
loaded?: boolean;
// On-disk GGUF quant. Ids stay bare for OpenAI compat, so append `:quant` to pin it.
quant?: string;
};
type ApiOpenAIModelList = {
data?: { id?: unknown; loaded?: unknown; quant?: unknown }[];
};
/**
* The models this server can serve: `/v1/models` returns exactly the ids
* `/v1/chat/completions` resolves against, and accepts the UI session JWT.
*/
export async function listOpenAIModels(): Promise<OpenAIModel[]> {
const res = await authFetch("/v1/models");
if (!res.ok) {
throw new Error(`Failed to list models (${res.status})`);
}
const body = (await res.json()) as ApiOpenAIModelList;
if (!Array.isArray(body?.data)) {
return [];
}
return body.data.flatMap((entry) =>
typeof entry?.id === "string" && entry.id
? [
{
id: entry.id,
loaded: entry.loaded === true,
quant: typeof entry.quant === "string" ? entry.quant : undefined,
},
]
: [],
);
}

213
studio/frontend/src/features/settings/components/api-monitor-console.tsx
View file

@ -7,6 +7,7 @@ import {
ActivityIcon,
ChevronDownIcon,
CircleIcon,
PowerOffIcon,
RefreshCwIcon,
} from "lucide-react";
import {
@ -17,11 +18,19 @@ import {
useRef,
useState,
} from "react";
import { getApiMonitor, getApiMonitorEntry } from "../../chat/api/chat-api";
import {
getApiMonitor,
getApiMonitorEntry,
getInferenceStatus,
unloadModel,
} from "../../chat/api/chat-api";
import { resolveInferenceCheckpointId } from "../../chat/lib/apply-inference-status-to-store";
import { useChatRuntimeStore } from "../../chat/stores/chat-runtime-store";
import type { ApiMonitorEntry, ApiMonitorResponse } from "../../chat/types/api";
const API_INFERENCE_PREFIX_RE = /^\/api\/inference/;
const V1_PREFIX_RE = /^\/v1\//;
const PAGE_SIZE = 5;
function formatTime(value: number): string {
return new Date(value * 1000).toLocaleTimeString([], {
@ -87,6 +96,65 @@ function UsageBar({ value }: { value?: number | null }): ReactElement | null {
);
}
function isLifecycle(entry: ApiMonitorEntry): boolean {
return entry.kind === "lifecycle";
}
function lifecycleLabel(entry: ApiMonitorEntry): string {
if (entry.event === "unload") {
return entry.reason === "idle" ? "Model unloaded (idle)" : "Model unloaded";
}
if (entry.event === "download") {
if (entry.status === "running") {
const pct = entry.progress;
return typeof pct === "number"
? `Downloading model (${Math.round(pct)}%)`
: "Downloading model";
}
if (entry.status === "completed") return "Model downloaded";
// A cancel is deliberate, so saying it failed misreads the user's own action.
return entry.status === "cancelled"
? "Model download cancelled"
: "Model download failed";
}
if (entry.status === "running") {
return "Loading model";
}
if (entry.status === "completed") {
return "Model loaded";
}
return "Model load failed";
}
// Load/unload rows: label, model and time. No prompt or detail, so nothing to expand.
function LifecycleEntry({ entry }: { entry: ApiMonitorEntry }): ReactElement {
return (
<article className="min-w-0 rounded-lg border border-border/70 bg-muted/25">
<div className="flex w-full min-w-0 items-start justify-between gap-3 p-3">
<div className="min-w-0">
<div className="flex min-w-0 items-center gap-2">
<ActivityIcon
className={cn("size-3.5 shrink-0", statusTone(entry.status))}
/>
<span className="truncate text-xs font-medium">
{lifecycleLabel(entry)}
</span>
</div>
<div className="mt-1 truncate text-ui-11 text-muted-foreground">
{entry.model}
</div>
</div>
<div className="shrink-0 text-right text-ui-11 text-muted-foreground">
<div>{formatTime(entry.started_at)}</div>
{entry.event === "load" || entry.event === "download" ? (
<div>{formatDuration(entry.duration_ms)}</div>
) : null}
</div>
</div>
</article>
);
}
function MonitorEntry({
entry,
detail,
@ -191,6 +259,7 @@ export function ApiMonitorConsole(): ReactElement {
const [data, setData] = useState<ApiMonitorResponse | null>(null);
const [error, setError] = useState<string | null>(null);
const [refreshing, setRefreshing] = useState(false);
const [unloading, setUnloading] = useState(false);
const [expandedIds, setExpandedIds] = useState<Set<string>>(() => new Set());
const [details, setDetails] = useState<Record<string, ApiMonitorEntry>>({});
const [loadingDetails, setLoadingDetails] = useState<Set<string>>(
@ -211,6 +280,28 @@ export function ApiMonitorConsole(): ReactElement {
}
}, []);
// /unload matches on the internal id, which the monitor omits, so read it from status.
const unloadActiveModel = useCallback(async (): Promise<void> => {
setUnloading(true);
try {
const status = await getInferenceStatus();
const checkpoint = resolveInferenceCheckpointId(status);
if (!checkpoint) {
setError(null);
return;
}
await unloadModel({ model_path: checkpoint });
// Same as the chat eject flow: the store still holds the freed checkpoint.
useChatRuntimeStore.getState().clearCheckpoint();
setError(null);
await loadMonitor();
} catch (err: unknown) {
setError(err instanceof Error ? err.message : "Failed to unload the model");
} finally {
setUnloading(false);
}
}, [loadMonitor]);
useEffect(() => {
let cancelled = false;
let timer: number | undefined;
@ -253,6 +344,48 @@ export function ApiMonitorConsole(): ReactElement {
const statusLabel = data?.status ?? "idle";
const hasActive = (data?.active_requests ?? 0) > 0;
const entries = useMemo(() => data?.entries ?? [], [data]);
// Page 1 tracks the live list; paging back freezes the id order so history holds still.
const [page, setPage] = useState(0);
const [frozenIds, setFrozenIds] = useState<string[] | null>(null);
const byId = useMemo(
() => new Map(entries.map((entry) => [entry.id, entry])),
[entries],
);
const ordered = useMemo(() => {
if (frozenIds === null) {
return entries;
}
return frozenIds.flatMap((id) => {
const entry = byId.get(id);
return entry ? [entry] : [];
});
}, [byId, entries, frozenIds]);
const pageCount = Math.max(1, Math.ceil(ordered.length / PAGE_SIZE));
const pageIndex = Math.min(page, pageCount - 1);
const visible = ordered.slice(
pageIndex * PAGE_SIZE,
pageIndex * PAGE_SIZE + PAGE_SIZE,
);
const newerCount =
frozenIds === null
? 0
: entries.filter((entry) => !frozenIds.includes(entry.id)).length;
const goToPage = useCallback(
(next: number): void => {
if (next <= 0) {
setFrozenIds(null);
setPage(0);
return;
}
// Freeze on the way off page 1 so the history under the cursor holds still.
setFrozenIds((prev) => prev ?? entries.map((entry) => entry.id));
setPage(next);
},
[entries],
);
const loadDetail = useCallback(
(id: string): void => {
if (loadingDetailsRef.current.has(id)) {
@ -305,8 +438,9 @@ export function ApiMonitorConsole(): ReactElement {
);
useEffect(() => {
for (const entry of entries) {
if (!expandedIds.has(entry.id)) {
// Only rows on screen: an expanded row on another page would keep polling.
for (const entry of visible) {
if (isLifecycle(entry) || !expandedIds.has(entry.id)) {
continue;
}
const cached = detailsRef.current[entry.id];
@ -314,7 +448,7 @@ export function ApiMonitorConsole(): ReactElement {
loadDetail(entry.id);
}
}
}, [entries, expandedIds, loadDetail]);
}, [visible, expandedIds, loadDetail]);
return (
<section className="flex min-w-0 flex-col rounded-lg border border-border/70 bg-background">
@ -339,6 +473,22 @@ export function ApiMonitorConsole(): ReactElement {
<div className="rounded-full border border-border px-2.5 py-1 text-xs capitalize text-muted-foreground">
{statusLabel}
</div>
{/* Always rendered, disabled when idle: the only manual release must stay visible. */}
<Button
type="button"
variant="ghost"
size="sm"
onClick={() => void unloadActiveModel()}
disabled={unloading || !data?.active_model}
title={
data?.active_model
? "Unload the model and free its VRAM"
: "No model is loaded"
}
>
<PowerOffIcon className="size-3.5" />
{unloading ? "Unloading" : "Unload"}
</Button>
<Button
type="button"
variant="ghost"
@ -375,19 +525,54 @@ export function ApiMonitorConsole(): ReactElement {
</div>
) : (
<div className="grid gap-3">
{entries.map((entry) => (
<MonitorEntry
key={entry.id}
entry={entry}
detail={details[entry.id]}
expanded={expandedIds.has(entry.id)}
loading={loadingDetails.has(entry.id)}
onToggle={() => toggleEntry(entry)}
/>
))}
{visible.map((entry) =>
isLifecycle(entry) ? (
<LifecycleEntry key={entry.id} entry={entry} />
) : (
<MonitorEntry
key={entry.id}
entry={entry}
detail={details[entry.id]}
expanded={expandedIds.has(entry.id)}
loading={loadingDetails.has(entry.id)}
onToggle={() => toggleEntry(entry)}
/>
),
)}
</div>
)}
</div>
{/* Also while frozen: retention can shrink that list below one page, and hiding the
pager would strand the console on a stale snapshot. */}
{ordered.length > PAGE_SIZE || frozenIds !== null ? (
<div className="flex items-center justify-between gap-2 border-t border-border/60 px-4 py-2 text-xs text-muted-foreground">
<span>
Page {pageIndex + 1} of {pageCount}
{newerCount > 0 ? ` (${newerCount.toLocaleString()} new)` : ""}
</span>
<div className="flex items-center gap-1">
<Button
variant="ghost"
size="sm"
className="h-7 px-2 text-xs"
onClick={() => goToPage(pageIndex - 1)}
disabled={pageIndex === 0 && frozenIds === null}
>
Newer
</Button>
<Button
variant="ghost"
size="sm"
className="h-7 px-2 text-xs"
onClick={() => goToPage(pageIndex + 1)}
disabled={pageIndex >= pageCount - 1}
>
Older
</Button>
</div>
</div>
) : null}
</section>
);
}

19
studio/frontend/src/features/settings/components/model-auto-switch-section.tsx
View file

@ -65,6 +65,7 @@ export function ModelAutoSwitchSection() {
idleSeconds: number | undefined,
syncDraft = true,
keepKv?: boolean,
autoDownload?: boolean,
) => {
setIsSaving(true);
setError(null);
@ -73,6 +74,7 @@ export function ModelAutoSwitchSection() {
enabled,
idleSeconds,
keepKv,
autoDownload,
);
setSettings(saved);
if (syncDraft) {
@ -117,6 +119,11 @@ export function ModelAutoSwitchSection() {
void persist(settings.enabled, undefined, false, keepKv);
};
const handleAutoDownloadToggle = (autoDownload: boolean) => {
if (!settings) return;
void persist(settings.enabled, undefined, false, undefined, autoDownload);
};
return (
<SettingsSection title={t("settings.general.modelAutoSwitch.sectionTitle")}>
<SettingsRow
@ -129,6 +136,18 @@ export function ModelAutoSwitchSection() {
onCheckedChange={handleToggle}
/>
</SettingsRow>
<SettingsRow
label={t("settings.general.modelAutoSwitch.autoDownload")}
description={t(
"settings.general.modelAutoSwitch.autoDownloadDescription",
)}
>
<Switch
checked={settings?.autoDownloadModel ?? false}
disabled={!settings?.enabled || isSaving}
onCheckedChange={handleAutoDownloadToggle}
/>
</SettingsRow>
<SettingsRow
label={t("settings.general.modelAutoSwitch.idleUnload")}
description={t(

330
studio/frontend/src/features/settings/components/usage-examples.tsx
View file

@ -29,11 +29,8 @@ import { HugeiconsIcon } from "@hugeicons/react";
import { useEffect, useMemo, useRef, useState } from "react";
import { Streamdown } from "streamdown";
import { loadCodingAgents } from "../api/coding-agents";
import {
type OpenAIAutoSwitchSettings,
loadOpenAIAutoSwitchSettings,
updateOpenAIAutoSwitchSettings,
} from "../api/openai-auto-switch";
import { loadOpenAIAutoSwitchSettings } from "../api/openai-auto-switch";
import { type OpenAIModel, listOpenAIModels } from "../api/openai-models";
import { buildAgentCommand, isLoopbackHost, normalizeHost } from "./agent-command";
type ExampleType =
@ -89,14 +86,7 @@ const JAVASCRIPT_TYPES = new Set<ExampleType>([
"javascriptAdvanced",
]);
const PROMPT = "Can Unsloth Studio do API calling?";
// Auto-switch demo: a second call naming a different downloaded GGUF so the
// example shows that the model field selects which model serves.
// A placeholder the user replaces with one of their downloaded GGUFs. A fixed
// repo is usually not one they have, so the resolver would fall through and the
// demo would keep serving the current model instead of switching.
const SWITCH_MODEL = "your-other-downloaded-GGUF";
const SWITCH_PROMPT = "Now answer as a different model.";
const PROMPT = "What is Unsloth Studio?";
// web_search + python + terminal are the reliable built-in tools.
const TOOLS = ["web_search", "python", "terminal"];
const ADV = {
@ -196,19 +186,13 @@ function winBody(model: string, variant: Variant): string {
return JSON.stringify(body, null, 2);
}
// A leading comment (valid in both bash and PowerShell) noting the model field
// selects the served model when auto-switch is on.
const SWITCH_NOTE =
'# "Switch model by request" is on: set "model" to any downloaded GGUF to switch.\n';
function curlUnix(
base: string,
key: string,
model: string,
variant: Variant,
autoSwitch: boolean,
): string {
return `${autoSwitch ? SWITCH_NOTE : ""}curl ${base}/v1/chat/completions \\
return `curl ${base}/v1/chat/completions \\
-H "Authorization: Bearer ${key}" \\
-H "Content-Type: application/json" \\
-d '${shSingle(curlBodyPretty(model, variant))}'`;
@ -219,9 +203,8 @@ function curlWindows(
key: string,
model: string,
variant: Variant,
autoSwitch: boolean,
): string {
return `${autoSwitch ? SWITCH_NOTE : ""}$body = '${psSingle(winBody(model, variant))}'
return `$body = '${psSingle(winBody(model, variant))}'
Set-Content -Path body.json -Value $body -Encoding ascii
curl.exe ${base}/v1/chat/completions \`
-H "Authorization: Bearer ${key}" \`
@ -229,29 +212,11 @@ curl.exe ${base}/v1/chat/completions \`
-d "@body.json"`;
}
// A second OpenAI call naming a different downloaded GGUF: with auto-switch on,
// Unsloth loads it before serving, so the model field selects the served model.
function pythonSwitchDemo(): string {
return `
# "Switch model by request" is on: replace the model below with another GGUF you
# have downloaded and Unsloth loads it before serving. Unknown names keep serving
# the current model.
response = client.chat.completions.create(
model=${j(SWITCH_MODEL)},
messages=[{"role": "user", "content": ${j(SWITCH_PROMPT)}}],
stream=True,
)
for chunk in response:
print(chunk.choices[0].delta.content or "", end="")`;
}
function pythonSnippet(
base: string,
key: string,
model: string,
variant: Variant,
autoSwitch: boolean,
): string {
const named =
variant === "advanced"
@ -296,7 +261,7 @@ response = client.chat.completions.create(
messages=[{"role": "user", "content": ${j(PROMPT)}}],${named}${extraBody}
stream=True,
)
${loop}${autoSwitch ? pythonSwitchDemo() : ""}`;
${loop}`;
}
function javascriptSnippet(
@ -304,7 +269,6 @@ function javascriptSnippet(
key: string,
model: string,
variant: Variant,
autoSwitch: boolean,
): string {
const options: string[] = [];
if (variant === "advanced") {
@ -343,23 +307,6 @@ const response = await client.chat.completions.create({
for await (const chunk of response) {
process.stdout.write(chunk.choices?.[0]?.delta?.content || "");
}${autoSwitch ? javascriptSwitchDemo() : ""}`;
}
function javascriptSwitchDemo(): string {
return `
// "Switch model by request" is on: replace the model below with another GGUF you
// have downloaded and Unsloth loads it before serving. Unknown names keep serving
// the current model.
const switchResponse = await client.chat.completions.create({
model: ${j(SWITCH_MODEL)},
messages: [{ role: "user", content: ${j(SWITCH_PROMPT)} }],
stream: true,
});
for await (const chunk of switchResponse) {
process.stdout.write(chunk.choices?.[0]?.delta?.content || "");
}`;
}
@ -368,31 +315,28 @@ function buildSnippets(
key: string,
model: string,
os: Os,
autoSwitch: boolean,
): Record<ExampleType, string> {
const curl = os === "windows" ? curlWindows : curlUnix;
return {
curl: curl(base, key, model, "plain", autoSwitch),
python: pythonSnippet(base, key, model, "plain", autoSwitch),
javascript: javascriptSnippet(base, key, model, "plain", autoSwitch),
curlTools: curl(base, key, model, "tools", autoSwitch),
pythonTools: pythonSnippet(base, key, model, "tools", autoSwitch),
javascriptTools: javascriptSnippet(base, key, model, "tools", autoSwitch),
curlAdvanced: curl(base, key, model, "advanced", autoSwitch),
pythonAdvanced: pythonSnippet(base, key, model, "advanced", autoSwitch),
javascriptAdvanced: javascriptSnippet(
base,
key,
model,
"advanced",
autoSwitch,
),
curl: curl(base, key, model, "plain"),
python: pythonSnippet(base, key, model, "plain"),
javascript: javascriptSnippet(base, key, model, "plain"),
curlTools: curl(base, key, model, "tools"),
pythonTools: pythonSnippet(base, key, model, "tools"),
javascriptTools: javascriptSnippet(base, key, model, "tools"),
curlAdvanced: curl(base, key, model, "advanced"),
pythonAdvanced: pythonSnippet(base, key, model, "advanced"),
javascriptAdvanced: javascriptSnippet(base, key, model, "advanced"),
};
}
const KEY_PLACEHOLDER = "sk-unsloth-YOUR_KEY";
const MODEL_FALLBACK = "unsloth/gemma-4-E4B-it-GGUF:UD-Q5_K_XL";
const USE_TUNNEL_KEY = "unsloth_api_use_tunnel";
// Slow retry while /v1 has nothing to name: a download or load moves no store state.
const CATALOG_RETRY_MS = 15000;
// Slower beat once something is servable: idle unload frees a model without
// touching the store, so residency is never settled for good.
const CATALOG_IDLE_MS = 60000;
function readUseTunnelPref(): boolean {
if (typeof window === "undefined") return true;
@ -412,18 +356,112 @@ function writeUseTunnelPref(value: boolean): void {
}
}
function useLoadedModelName(): string {
// A checkpoint can be an on-disk load path, which /v1 never advertises. Mirrors _looks_like_path.
function looksLikePath(id: string): boolean {
return (
id.startsWith("/") ||
id.startsWith("~") ||
id.startsWith(".") ||
id.includes("\\") ||
id.toLowerCase().endsWith(".gguf") ||
(id.match(/\//g)?.length ?? 0) >= 2
);
}
// Same model, ignoring any ":quant" a caller pinned.
function sameBaseModelId(a: string, b: string): boolean {
const base = (id: string) => id.trim().toLowerCase().split(":")[0];
return a.trim().toLowerCase() === b.trim().toLowerCase() || base(a) === base(b);
}
// The model the examples name: always an id /v1 resolves against, null when there is none.
function useExampleModelName(): string | null {
const checkpoint = useChatRuntimeStore((s) => s.params.checkpoint);
const ggufVariant = useChatRuntimeStore((s) => s.activeGgufVariant);
return useMemo(() => {
if (!checkpoint || checkpoint.startsWith("external::")) {
return MODEL_FALLBACK;
}
if (ggufVariant && !checkpoint.includes(":")) {
return `${checkpoint}:${ggufVariant}`;
}
return checkpoint;
// null until /v1/models answers: "not asked yet" must not read as "holds nothing".
const [catalog, setCatalog] = useState<OpenAIModel[] | null>(null);
// A downloaded but unloaded model is only runnable when switching is on.
const [autoSwitch, setAutoSwitch] = useState(false);
// Idle-unload running on its own (UNSLOTH_MODEL_IDLE_TTL, switching off) still
// reloads exactly what it freed on the next request. That restores the stored
// checkpoint only, never an arbitrary catalog entry, so it is tracked apart.
const [idleReload, setIdleReload] = useState(false);
const usableCheckpoint =
!!checkpoint && !checkpoint.startsWith("external::") && !looksLikePath(checkpoint);
// Always: a stored checkpoint can stop being servable without the store changing.
// biome-ignore lint/correctness/useExhaustiveDependencies: a load or unload must refetch the servable ids
useEffect(() => {
let cancelled = false;
let timeoutId: number | null = null;
const update = () => {
// null on failure, never [] or false: a transient error is not evidence that the
// server holds nothing, and feeding those negatives in blanked every example while
// the model was still servable. Keep the last answer and retry.
void Promise.all([
listOpenAIModels().catch(() => null),
loadOpenAIAutoSwitchSettings()
.then((s) => [s.enabled, s.idleUnloadActive] as const)
.catch(() => null),
])
.then(([models, settings]) => {
if (cancelled) return true;
if (models !== null) setCatalog(models);
if (settings !== null) {
setAutoSwitch(settings[0]);
setIdleReload(settings[1]);
}
// Resident only slows the polling; it never stops it.
return models !== null && models.some((m) => m.loaded);
})
.then((resolved) => {
if (cancelled) return;
timeoutId = window.setTimeout(
update,
resolved ? CATALOG_IDLE_MS : CATALOG_RETRY_MS,
);
});
};
update();
return () => {
cancelled = true;
if (timeoutId !== null) window.clearTimeout(timeoutId);
};
}, [checkpoint, ggufVariant]);
return useMemo(() => {
// Name something held here, with its quant to pin the file on disk.
const fromCatalog = (): string | null => {
const pick =
catalog?.find((m) => m.loaded) ?? (autoSwitch ? catalog?.[0] : undefined);
if (!pick) {
return null;
}
return pick.quant && !pick.id.includes(":")
? `${pick.id}:${pick.quant}`
: pick.id;
};
// The store keeps a checkpoint across an idle unload, and across the model
// being deleted, so it only names a runnable model while the catalog still
// lists it: resident, or downloaded with switching able to reload it. A null
// catalog means /v1/models has not answered, which is not evidence against it.
const entry = catalog?.find((m) => sameBaseModelId(m.id, checkpoint ?? ""));
const backed =
catalog === null || (!!entry && (entry.loaded || autoSwitch || idleReload));
if (usableCheckpoint && checkpoint && backed) {
if (checkpoint.includes(":")) {
return checkpoint;
}
// Pin the quant the catalog advertises, not the stored one: membership proves
// the repo, and the saved quant can name a file deleted while another quant of
// the same repo remains. Fall back to the store only before /v1/models answers.
const quant = catalog === null ? ggufVariant : entry?.quant;
return quant ? `${checkpoint}:${quant}` : checkpoint;
}
return fromCatalog();
}, [autoSwitch, catalog, checkpoint, ggufVariant, idleReload, usableCheckpoint]);
}
// Backend PATH detection is only safe in the desktop app, where the UI owns
@ -493,11 +531,6 @@ export function UsageExamples({ apiKey }: { apiKey?: string | null }) {
const base =
useTunnel && cloudflareUrl ? cloudflareUrl : (serverUrl ?? origin);
const localAgentDetection = canUseLocalAgentDetection(base);
// null while loading; the same setting the General tab exposes (shared cache).
const [autoSwitch, setAutoSwitch] = useState<OpenAIAutoSwitchSettings | null>(
null,
);
const [savingAutoSwitch, setSavingAutoSwitch] = useState(false);
useEffect(() => {
void fetchDeviceType({ force: true });
@ -575,27 +608,13 @@ export function UsageExamples({ apiKey }: { apiKey?: string | null }) {
}
}, [agent, detectedAgents, activeGgufVariant, activeNativePathToken, ggufContextLength]);
useEffect(() => {
let cancelled = false;
void loadOpenAIAutoSwitchSettings()
.then((s) => {
if (!cancelled) setAutoSwitch(s);
})
.catch(() => {
// Best-effort: leave the toggle off if the setting can't be read.
});
return () => {
cancelled = true;
};
}, []);
const model = useLoadedModelName();
const model = useExampleModelName();
const key = apiKey || KEY_PLACEHOLDER;
const autoSwitchOn = autoSwitch?.enabled ?? false;
// Null model: nothing is servable, so there is no snippet worth copying.
const snippets = useMemo(
() => buildSnippets(base, key, model, os, autoSwitchOn),
[base, key, model, os, autoSwitchOn],
() => (model ? buildSnippets(base, key, model, os) : null),
[base, key, model, os],
);
// Agent command must target the server the panel shows, not the :8888 default.
const agentCommand = useMemo(
@ -613,6 +632,7 @@ export function UsageExamples({ apiKey }: { apiKey?: string | null }) {
: "python";
const handleCopy = async () => {
if (!snippets) return;
if (await copyToClipboard(snippets[lang])) {
setCopied(true);
setTimeout(() => setCopied(false), 1800);
@ -624,20 +644,6 @@ export function UsageExamples({ apiKey }: { apiKey?: string | null }) {
writeUseTunnelPref(next);
};
// Same setting as the General tab; persist optimistically and revert on failure
// so the examples reflect the live model-switch behavior.
const handleToggleAutoSwitch = (next: boolean) => {
const idle = autoSwitch?.autoUnloadIdleSeconds ?? 0;
setAutoSwitch((prev) => (prev ? { ...prev, enabled: next } : prev));
setSavingAutoSwitch(true);
void updateOpenAIAutoSwitchSettings(next, idle)
.then(setAutoSwitch)
.catch(() => {
setAutoSwitch((prev) => (prev ? { ...prev, enabled: !next } : prev));
})
.finally(() => setSavingAutoSwitch(false));
};
const handleCopyUrl = async () => {
if (cloudflareUrl && (await copyToClipboard(cloudflareUrl))) {
setCopiedUrl(true);
@ -658,41 +664,7 @@ export function UsageExamples({ apiKey }: { apiKey?: string | null }) {
{t("settings.apiKeys.usageExamples")}
</h2>
<div className="min-w-0 max-w-full overflow-hidden rounded-lg border border-border bg-muted/20">
{/* Same setting as the General tab; surfaced here so the request `model`
actually switches the served model, which the examples below show. */}
<div className="flex min-w-0 items-center justify-between gap-2 border-b border-border px-2 py-1.5">
<div className="flex shrink-0 items-center gap-1.5">
<Switch
size="sm"
checked={autoSwitchOn}
disabled={autoSwitch === null || savingAutoSwitch}
onCheckedChange={handleToggleAutoSwitch}
aria-label={t("settings.general.modelAutoSwitch.enable")}
/>
<span className="text-ui-11 font-medium text-foreground">
{t("settings.general.modelAutoSwitch.enable")}
</span>
<Tooltip>
<TooltipTrigger asChild={true}>
<button
type="button"
className="flex items-center rounded text-muted-foreground transition-colors hover:text-foreground focus-visible:outline-none focus-visible:ring-1 focus-visible:ring-ring"
aria-label={t(
"settings.general.modelAutoSwitch.enableDescription",
)}
>
<HugeiconsIcon
icon={InformationCircleIcon}
className="size-3.5"
/>
</button>
</TooltipTrigger>
<TooltipContent className="max-w-[260px] text-ui-11 leading-snug">
{t("settings.general.modelAutoSwitch.enableDescription")}
</TooltipContent>
</Tooltip>
</div>
</div>
{/* No model-auto-switch row: ModelAutoSwitchSection renders that setting just below. */}
{cloudflareUrl ? (
<div className="flex min-w-0 items-center justify-between gap-2 border-b border-border px-2 py-1.5">
<div className="flex shrink-0 items-center gap-1.5">
@ -802,25 +774,33 @@ export function UsageExamples({ apiKey }: { apiKey?: string | null }) {
</button>
</div>
) : null}
<div className="relative min-w-0">
<button
type="button"
onClick={handleCopy}
className="absolute right-2 top-2 z-10 flex items-center gap-1 rounded border border-border bg-background/80 px-1.5 py-1 text-ui-11 text-muted-foreground backdrop-blur transition-colors hover:text-foreground focus-visible:outline-none focus-visible:ring-1 focus-visible:ring-ring"
aria-label={t("settings.apiKeys.copySnippet")}
>
<HugeiconsIcon
icon={copied ? Tick02Icon : Copy01Icon}
className={cn("size-3.5", copied && "text-emerald-600")}
{snippets ? (
<div className="relative min-w-0">
<button
type="button"
onClick={handleCopy}
className="absolute right-2 top-2 z-10 flex items-center gap-1 rounded border border-border bg-background/80 px-1.5 py-1 text-ui-11 text-muted-foreground backdrop-blur transition-colors hover:text-foreground focus-visible:outline-none focus-visible:ring-1 focus-visible:ring-ring"
aria-label={t("settings.apiKeys.copySnippet")}
>
<HugeiconsIcon
icon={copied ? Tick02Icon : Copy01Icon}
className={cn("size-3.5", copied && "text-emerald-600")}
/>
{copied
? t("settings.apiKeys.copied")
: t("settings.apiKeys.copy")}
</button>
<HighlightedCode
key={snippets[lang]}
code={snippets[lang]}
language={shikiLang}
/>
{copied ? t("settings.apiKeys.copied") : t("settings.apiKeys.copy")}
</button>
<HighlightedCode
key={snippets[lang]}
code={snippets[lang]}
language={shikiLang}
/>
</div>
</div>
) : (
<div className="min-w-0 px-3 py-2.5 text-ui-11 leading-snug text-muted-foreground">
{t("settings.apiKeys.usageNoModel")}
</div>
)}
<div className="flex min-w-0 flex-col gap-1.5 border-t border-border px-3 py-2.5">
<span className="text-ui-11 font-semibold text-foreground">
{t("settings.apiKeys.codingAgents")}

4
studio/frontend/src/features/settings/tabs/api-keys-tab.tsx
View file

@ -168,12 +168,12 @@ export function ApiKeysTab() {
)}
</section>
<ModelAutoSwitchSection />
<ApiMonitorConsole />
<UsageExamples apiKey={revealed} />
<ModelAutoSwitchSection />
<Dialog open={revokeTarget !== null} onOpenChange={(o) => !o && setRevokeTarget(null)}>
<DialogContent className="max-w-md">
<DialogHeader>

17
studio/frontend/src/i18n/locales/en.ts
View file

@ -284,20 +284,21 @@ export const en = {
sectionTitle: "Model auto-switch (OpenAI API)",
enable: "Switch model by request",
enableDescription:
"When an OpenAI-compatible request names a different downloaded GGUF, load it before serving. Off by default; unknown names keep serving the loaded model.",
"Load a downloaded GGUF named in an API request before serving. Off by default.",
autoDownload: "Download missing models",
autoDownloadDescription:
"Fetch a GGUF named in an API request that is not downloaded yet. Anyone with an API key can then use disk and bandwidth.",
idleUnload: "Idle auto-unload",
idleUnloadDescription:
"Unload the model after this many idle seconds to free VRAM; the next request reloads it. 0 keeps it loaded. Minimum 60 seconds.",
idleNeedsEnable:
"Turn on Switch model by request so an unloaded model reloads on next use.",
idleActiveViaEnv:
"Idle auto-unload is active via the UNSLOTH_MODEL_IDLE_TTL environment variable.",
"Free VRAM after this many idle seconds. 0 keeps it loaded, minimum 60.",
idleNeedsEnable: "Turn on Switch model by request first.",
idleActiveViaEnv: "Active via UNSLOTH_MODEL_IDLE_TTL.",
loadError: "Failed to load model auto-switch settings.",
saveError: "Failed to save model auto-switch settings.",
idleError: "Enter 0 to keep the model loaded, or at least 60 seconds.",
keepKv: "Keep chat context across idle unload",
keepKvDescription:
"Save the model's KV cache to disk before an idle unload and restore it on reload, so resumed chats skip re-reading their history. Chat context is written to disk (up to 10 GB) until it is restored or cleaned up.",
"Save the KV cache before an idle unload so resumed chats skip re-reading history. Up to 10 GB on disk.",
},
previewSharing: {
sectionTitle: "Preview sharing",
@ -818,6 +819,8 @@ export const en = {
copyAccessToken: "Copy access token",
copyNow: "Copy now - this won't be shown again.",
usageExamples: "Usage examples",
usageNoModel:
"Load or download a model to see runnable examples. This server has no model to name yet.",
usageTools: "Tools",
exampleCurlTools: "curl + tools",
examplePythonTools: "Python + tools",

200
tests/studio/test_usage_examples_model_source_contract.py Normal file
View file

@ -0,0 +1,200 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
"""Static contract for which model the API usage examples name, and for the
model-auto-switch control living in exactly one place on the API keys tab."""
import re
from pathlib import Path
REPO = Path(__file__).resolve().parents[2]
SETTINGS = REPO / "studio/frontend/src/features/settings"
USAGE_EXAMPLES_TSX = SETTINGS / "components/usage-examples.tsx"
OPENAI_MODELS_TS = SETTINGS / "api/openai-models.ts"
API_KEYS_TAB_TSX = SETTINGS / "tabs/api-keys-tab.tsx"
def test_examples_name_a_model_the_server_can_serve():
# A hardcoded repo id made copied curls 404; read the servable ids from /v1/models.
src = USAGE_EXAMPLES_TSX.read_text(encoding = "utf-8")
assert 'from "../api/openai-models"' in src
assert "function useExampleModelName(): string" in src
hook = src[src.find("function useExampleModelName") : src.find("// Backend PATH detection")]
assert "listOpenAIModels()" in hook
# Precedence: live checkpoint, then a loaded entry, then any entry if switching is on.
assert "catalog?.find((m) => m.loaded) ?? (autoSwitch ? catalog?.[0] : undefined)" in hook
# The snippet pins the quant so the request names the file on disk.
assert "`${pick.id}:${pick.quant}`" in hook
api = OPENAI_MODELS_TS.read_text(encoding = "utf-8")
assert 'authFetch("/v1/models")' in api
def test_examples_never_print_a_hardcoded_model_id():
# The bug this exists for: a `[]` catalog printed a snippet before /v1/models answered.
# It is tri-state now, and the panel asks for a model instead.
src = USAGE_EXAMPLES_TSX.read_text(encoding = "utf-8")
assert "MODEL_FALLBACK" not in src
# No repo-shaped literal anywhere: a snippet may only name what /v1 returns.
assert re.search(r'"unsloth/[^"]+"', src) is None
assert "function useExampleModelName(): string | null" in src
assert "useState<OpenAIModel[] | null>(null)" in src
# Nothing servable means nothing is built, so there is nothing to copy.
assert "(model ? buildSnippets(base, key, model, os) : null)" in src
assert "if (!snippets) return;" in src
assert "{snippets ? (" in src
assert 't("settings.apiKeys.usageNoModel")' in src
en = EN_TS.read_text(encoding = "utf-8")
assert "usageNoModel:" in en
def test_catalog_refresh_follows_the_loaded_model():
# A dep list that misses these never re-ran, so a finished load left the first
# fetch's name. It must not be gated on having no checkpoint either: the store
# keeps one across an idle unload, which changes nothing React can see.
src = USAGE_EXAMPLES_TSX.read_text(encoding = "utf-8")
hook = src[src.find("function useExampleModelName") : src.find("// Backend PATH detection")]
assert "}, [checkpoint, ggufVariant]);" in hook
assert "needsCatalog" not in hook
# A finishing download moves no store state, so the fetch retries on a timer too,
# and residency only slows that timer rather than stopping it.
assert "CATALOG_RETRY_MS" in hook and "CATALOG_IDLE_MS" in hook
assert "window.clearTimeout(timeoutId)" in hook
assert "const CATALOG_RETRY_MS = 15000;" in src
assert "const CATALOG_IDLE_MS = 60000;" in src
def test_a_stored_checkpoint_needs_catalog_evidence():
# The store keeps a checkpoint across an idle unload and across the model being
# deleted. Preferring it on the switch setting alone kept naming one /v1/models
# had already proved absent, so the snippets 404d instead of falling back.
src = USAGE_EXAMPLES_TSX.read_text(encoding = "utf-8")
hook = src[src.find("function useExampleModelName") : src.find("// Backend PATH detection")]
assert 'const entry = catalog?.find((m) => sameBaseModelId(m.id, checkpoint ?? ""));' in hook
# Resident, or downloaded with something able to reload it. Never the setting alone.
assert "(!!entry && (entry.loaded || autoSwitch || idleReload))" in hook
assert "autoSwitch ||\n" not in hook
def test_standalone_idle_unload_still_names_the_stored_checkpoint():
# UNSLOTH_MODEL_IDLE_TTL without auto-switch reloads exactly what it freed, so the
# stored checkpoint stays runnable after an idle unload and the panel must keep
# showing it. The stash restores only that model, so it can never pick catalog[0].
src = USAGE_EXAMPLES_TSX.read_text(encoding = "utf-8")
hook = src[src.find("function useExampleModelName") : src.find("// Backend PATH detection")]
assert "const [idleReload, setIdleReload] = useState(false);" in hook
assert "setIdleReload(settings[1])" in hook
assert "s.idleUnloadActive" in hook
# fromCatalog stays gated on auto-switch alone.
assert "?? (autoSwitch ? catalog?.[0] : undefined)" in hook
assert "idleReload ? catalog" not in hook
def test_a_failed_refresh_does_not_erase_what_the_server_holds():
# Catching into [] and false made a transient error authoritative: the panel
# dropped a still-servable model and printed "No model" until the next poll.
# The catalog is deliberately tri-state, and a failure must stay the unknown one.
src = USAGE_EXAMPLES_TSX.read_text(encoding = "utf-8")
hook = src[src.find("function useExampleModelName") : src.find("// Backend PATH detection")]
assert "listOpenAIModels().catch(() => null)" in hook
assert ".catch(() => null)," in hook
assert "if (models !== null) setCatalog(models);" in hook
assert "if (settings !== null) {" in hook
# The old negatives must be gone entirely.
assert "catch(() => [] as OpenAIModel[])" not in hook
assert "catch(() => [false, false] as const)" not in hook
assert "catch(() => false)" not in hook
def test_the_pinned_quant_comes_from_the_catalog():
# Catalog membership proves the repo, not the saved quant. The stored one can
# name a file deleted while another quant of the same repo remains, and pinning
# it emitted repo:deleted-quant, a missing-quant 404 with a runnable one listed.
src = USAGE_EXAMPLES_TSX.read_text(encoding = "utf-8")
hook = src[src.find("function useExampleModelName") : src.find("// Backend PATH detection")]
assert "const quant = catalog === null ? ggufVariant : entry?.quant;" in hook
assert "`${checkpoint}:${ggufVariant}`" not in hook
def test_usage_examples_has_no_duplicate_auto_switch_control():
# ModelAutoSwitchSection renders this setting just below and shares no state with it.
src = USAGE_EXAMPLES_TSX.read_text(encoding = "utf-8")
# Reading the setting is fine; writing it here is what would be a second control.
assert "updateOpenAIAutoSwitchSettings" not in src
assert "SWITCH_NOTE" not in src
assert "Switch model by request" not in src
assert "pythonSwitchDemo" not in src
assert "javascriptSwitchDemo" not in src
assert "modelAutoSwitch" not in src
tab = API_KEYS_TAB_TSX.read_text(encoding = "utf-8")
assert "<ModelAutoSwitchSection />" in tab
API_MONITOR_TSX = SETTINGS / "components/api-monitor-console.tsx"
def test_api_monitor_pages_five_at_a_time():
# The backend retains 50 terminal entries; the console used to dump them all at once.
src = API_MONITOR_TSX.read_text(encoding = "utf-8")
assert "const PAGE_SIZE = 5;" in src
assert "ordered.slice(" in src
# Paging back must freeze the id order, or live traffic reorders history under it.
assert "frozenIds" in src
assert "setFrozenIds((prev) => prev ?? entries.map((entry) => entry.id))" in src
def test_api_monitor_renders_lifecycle_rows():
src = API_MONITOR_TSX.read_text(encoding = "utf-8")
assert "function LifecycleEntry(" in src
assert 'entry.kind === "lifecycle"' in src
for label in ("Loading model", "Model loaded", "Model unloaded"):
assert label in src
# Lifecycle rows have no prompt/reply to fetch.
assert "isLifecycle(entry) || !expandedIds.has(entry.id)" in src
def test_auto_switch_section_sits_above_the_monitor():
tab = API_KEYS_TAB_TSX.read_text(encoding = "utf-8")
assert tab.index("<ModelAutoSwitchSection />") < tab.index("<ApiMonitorConsole />")
assert tab.index("<ApiMonitorConsole />") < tab.index("<UsageExamples")
AUTO_SWITCH_TSX = SETTINGS / "components/model-auto-switch-section.tsx"
EN_TS = REPO / "studio/frontend/src/i18n/locales/en.ts"
def test_api_monitor_renders_download_rows():
src = API_MONITOR_TSX.read_text(encoding = "utf-8")
assert 'entry.event === "download"' in src
for label in ("Downloading model", "Model downloaded", "Model download failed"):
assert label in src
def test_monitor_can_unload_the_loaded_model():
src = API_MONITOR_TSX.read_text(encoding = "utf-8")
assert "unloadActiveModel" in src
# Always rendered so the manual release stays discoverable; disabled, not hidden.
assert "disabled={unloading || !data?.active_model}" in src
assert "{data?.active_model ? (" not in src
# /unload matches on the internal id, omitted here (a host path), so read it from status.
assert "resolveInferenceCheckpointId(status)" in src
assert "unloadModel({ model_path: checkpoint })" in src
def test_auto_download_toggle_is_gated_on_auto_switch():
# Downloading what auto-switch cannot load fetches gigabytes nothing can serve.
src = AUTO_SWITCH_TSX.read_text(encoding = "utf-8")
assert "modelAutoSwitch.autoDownload" in src
assert "settings?.autoDownloadModel ?? false" in src
row = src[src.find("modelAutoSwitch.autoDownload") :]
assert "disabled={!settings?.enabled || isSaving}" in row[: row.find("</SettingsRow>")]
def test_auto_download_copy_warns_about_api_key_holders():
en = EN_TS.read_text(encoding = "utf-8")
start = en.find("autoDownloadDescription:")
assert start != -1
description = en[start : en.find("\n", en.find('",', start))]
assert "API key" in description