da447d47ba* Studio: say which model is missing instead of "No model loaded" A /v1 request naming a model that is not downloaded returned the generic "No model loaded. Call POST /inference/load first.", which cannot fix it. Return 404 model_not_found naming the model and listing what can serve, and make the API usage examples name a model the server actually has. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: page the API monitor, show model load/unload, pin the example quant The monitor rendered all 50 retained entries in one scroller: page it 5 at a time, freezing history while paged back so live traffic cannot reorder it. Add model load/unload rows so the feed shows what the server is doing, and stop the header rendering the loaded model as a raw host path. Advertise each model's GGUF quant on /v1/models so the example pins repo:QUANT, and move the auto-switch section above the monitor with shorter copy. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: optionally download a model named in an OpenAI API request Auto-switch only ever loaded models already on disk, so naming one this server does not have either 404s or, when something else is loaded, gets quietly answered by the resident model. Add openai_api_auto_download_model (off by default, gated on auto-switch). When on, a /v1 request naming a GGUF repo that is not downloaded starts a background fetch and returns 503 with Retry-After and a typed model_downloading code. The resident model keeps serving in the meantime, and the retry after the download completes is served by the new model through the existing auto-switch path. The download reuses the Hub manager's service layer, which already does repo-id validation, casing, claim bookkeeping, disk preflight, resume and cancel. The in-loader download is deliberately not used: it silently falls back to a smaller quant under low disk, which is wrong when the caller named an exact one. Admission is narrow, since a request only needs an API key: - namespace/name only, so gpt-4 and other foreign ids fall through to the resident model exactly as before - GGUF only, decided from the remote file list rather than the repo name - anything declaring auto_map is refused, so trust_remote_code stays a deliberate opt-in in the UI and can never be granted over the API - a single download at a time, plus a free-disk reserve - one model_info call answers existence, gating and the quant list, so a missing repo, a gated repo and a wrong quant each get their own error With the setting off every one of these paths is byte-identical to before. Also: - monitor rows for downloads, with a live percentage - public_model_id resolves an HF cache snapshot to its repo id, so a cache-loaded model is no longer labelled with a commit sha; this drops the duplicate helper added for the monitor and fixes the same leak in the inference status response - the unedited sk-unsloth-YOUR_KEY from the copyable examples now says so instead of "Invalid or expired API key"; every other bad key keeps the generic message * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add an Unload button to the API monitor The monitor names the loaded model but offered no way to free it. Idle auto-unload is the only existing release path, and it needs a TTL and a wait. The button sits next to Refresh, appears only while a model is loaded and is disabled mid-unload. /unload matches on the internal identifier, which this response deliberately omits because it would be a host path, so the click reads it from /api/inference/status the same way the chat runtime does rather than widening the monitor payload. Also stamp the manual unload row with the quant, read before the teardown clears it, so it reads repo:QUANT like the load row it pairs with. * Studio: keep the API monitor Unload button visible when idle It only rendered while a model was loaded, which hid the one manual release path at exactly the moment someone goes looking for it. Render it always, disabled with a "No model is loaded" tooltip when there is nothing to free. * Studio: never answer a named model with a different one Asking for a model this server is not serving returned 200 from whatever was resident. Requesting gemma-4-E2B-it-GGUF:UD-Q6_K_XL while UD-Q4_K_XL was loaded got a confident answer from the wrong quant, with nothing in the response saying so. A name carrying a namespace (org/model, optionally :QUANT) is a concrete reference, so 404 instead, with the reason: - wrong quant -> names the quants that are actually downloaded - not on disk -> lists what is available - on disk but auto-switch off -> says to turn it on Ids without a namespace (gpt-4, claude-3, default) are foreign labels rather than references, so they still fall through to the resident model and drop-in clients are unaffected. A bare org/model is still satisfied by any loaded quant of that repo; only an explicit :QUANT must match. The check runs whatever the auto-switch and auto-download toggles are, since serving the wrong weights is wrong in every configuration. It is skipped when nothing is loaded, where the existing no-model-loaded error already says the right thing, and when the model is on disk with auto-switch on, where a failed swap should still fall back. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: use a simpler prompt in the API usage examples "What is Unsloth Studio?" rather than "Can Unsloth Studio do API calling?". One constant feeds all nine snippet tabs. * Studio: only refuse a model reference meant for this server A namespace alone was treated as a concrete model reference, so a /v1 request naming anthropic/claude-3.5-sonnet, openai/gpt-4o or any other LiteLLM or OpenRouter style vendor/model id started returning 404 instead of being answered by the resident model. Refuse only on evidence the caller meant this server: an explicit GGUF quant label, or a repo that is actually on disk here. gpt-4 and vendor/model alike fall through again, while the wrong-quant and wrong-repo cases this PR exists for still refuse. Also from review: - Release the single download slot by object identity, not repo id. A stale watcher could clear a newer download of the same repo and let a second multi-GB fetch start alongside it. - Catch BaseException around admission: CancelledError is not an Exception, so a cancelled request stranded the slot for the process lifetime. - Honour the download service's accepted=False, which it returns without raising for a cross-variant conflict, instead of promising a download that was never dispatched. - Treat a failed status probe as unknown rather than idle, so a transient read cannot fail the monitor row and free the slot under a live worker. - Check gated repos with auth_check. The Hub serves metadata for a gated repo without granting its files, so the licence gate was being reported as the unrelated custom-code refusal. - Size the disk reserve from the download plan, which includes the mmproj and MTP companions the worker fetches with every quant. - Never fetch under the server's own HF token. The repo is named by whoever holds an API key, so the ambient token let that key pull the owner's private repos. - Refuse an explicit quant on a backend with no quant identity, gated on the suffix really being a quant so Ollama style :latest tags still match. - Raise instead of falling through when the diagnosis fails: the mismatch is already established by then, only the wording is uncertain. - Report a failed switch as 503 model_switch_failed rather than answering as the resident model. - Fail an open monitor row under the same lock as the check, so a finish landing in between cannot stamp an error onto a completed row. - Usage examples never emit a hardcoded model id: the catalog is tri-state and the panel asks for a model to be loaded instead of printing one the server cannot serve. It also refreshes when the loaded model changes. - Keep the monitor pager reachable while frozen entries expire. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: scope the auto-download 404 cache to the caller's credentials The Hub answers 404 for a private repo the caller cannot see, so caching that verdict per repo alone let one anonymous request mark a private repo unservable for everyone for the whole TTL. A later caller sending a valid X-Unsloth-HF-Token skipped the probe and fell through to the resident model instead of downloading what it asked for. Keyed on the repo id plus a digest of the token now, so the token itself is never held. Two more from the same review: - Clear the chat runtime checkpoint after unloading from the API monitor, as the chat eject flow already does. The store went on treating the freed checkpoint as loaded and the usage examples kept naming it. - Point gated and not-found callers at the X-Unsloth-HF-Token header. Automatic download deliberately ignores the server's own Hugging Face identity, so telling the user to add a token in Studio sent them round the same 403 forever. * Studio: tighten the comments added by this branch * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep API auto-download off the server's Hugging Face identity Passing None for the caller's token was not anonymous. spawn_worker substitutes the backend's HF_TOKEN for a falsy one, and HfApi(token=None) falls back to a cached login, so a repo named by an API-key holder could still be fetched under the owner's Hub identity and land in the shared catalog. The metadata probe and auth_check now pass an explicit False, and dispatch threads allow_ambient_token=False so the worker stays anonymous too. The flag defaults to True, so the UI download path keeps the ambient fallback that private repos rely on. Three more from the same review: - Require an exact hf_variant match only when the suffix is really a quant. The llama.cpp branch still compared Ollama style :latest and :8b against the loaded quant and refused the resident model, which is the opposite of what looks_like_quant classifies them as. - Decode an HF cache repo id only when the models-- component is followed by snapshots. An ordinary directory whose name merely starts with models-- was being read as an encoded repo id. - Return the probing response before consulting the job registry when an adopted claim has no variant yet. A stale error on the whole-repo key could otherwise release the slot the first request's probe still holds, letting a second large download start beside it. * Studio: stop treating a namespace as what decides model intent The rule refused a reference only when it carried a namespace, which was wrong in both directions. vendor/model is how LiteLLM and OpenRouter name every provider, and a standalone or custom-folder GGUF is advertised without one, so asking for a path-free local id such as model-Q4_K_M was answered by whatever else happened to be resident. The slashless early return is gone and the same evidence test now applies to every id: an explicit quant, or a model that actually resolves here. gpt-4 and default still fall through because they are not local, not because of their shape. Also: - Recognise bits-per-weight quant labels. _extract_quant_label emits IQ4_XS-3.53bpw and the resolver and downloader both accept it, but _GGUF_KNOWN_QUANT_RE has no bpw group, so looks_like_quant rejected a reference the rest of the machinery understands. - Upper-case the synthetic names handed to _pick_best_gguf. Its preference tokens are upper case and matched case-sensitively, so a repo with lower-case filenames skipped the preference and took the first entry, which can be F16. - Only offer a downloaded but unloaded model as a runnable example when auto-switch is on. It is off by default, so the copied snippet hit the no-model-loaded error, which is the failure this branch exists to fix. The tool-passthrough cancel test stubbed asyncio.to_thread module-wide, so it cancelled at the first thread hop rather than the generation hop it means to test. Model resolution runs off the loop before the monitor row opens, so that stub now passes the resolver through. * Studio: tighten the comments added since the last pass * Studio: match a resident model through its resolver alias A manual load stores the model by its on-disk path while the resolver and /v1/models advertise it as publisher/model, so _loaded_satisfies could not recognise the alias. Reducing the resolution to a boolean then threw away the load path that would have proved the match, and the request was refused with 404 for a model the server was serving at that moment. Common for LM Studio models and custom-folder aliases. The resolved path is compared against the resident backend before anything is refused. Also: - Size disk admission on what is left to fetch. expected_bytes is the whole plan, so a resumed quant or a companion already pulled in by another quant was charged for twice and could 507 a download that fits. Cached blobs are subtracted through existing_blob_bytes, the same accounting the worker's own preflight does, and it falls open to the full size when no blob hashes are available. - Report a cancelled download as cancelled. The catch-all sent every state other than complete or idle through fail_open, so a deliberate cancel rendered as a download failure rather than the monitor's cancelled state. - Keep polling the servable ids while nothing is loaded. The poll settled as soon as auto-switch was on, so turning it back off left the examples naming an unloaded model until something else remounted the panel. * Studio: shorten the comments added in the last pass * Studio: keep the FLA fast-path tests hermetic across transformers versions _discover_fla_model_types scans the *installed* transformers for modeling files importing `from fla.`, so `models/qwen3_5/` only exists from transformers 5.x. The backend supports transformers>=4.51, and on a 4.x install the Qwen3.5 gate returns False, so 14 tests in test_training_worker_flash_attn.py silently exercised a no-op instead of the install path and failed their call-count assertions. Pin the discovered model_type set in those 14 tests, the same way test_hook_does_not_install_tilelang_for_model_outside_allowlist already pins it against newly added FLA model_types. Test-only change: the production gate and the _discover_fla_model_types unit tests are untouched. * Studio: keep the /v1 admission check off the model-scanning path The admission check added here runs on every /v1 request, including with auto-switch off, where the route used to return straight away. It called resolve_local_gguf, whose index is cached for 5s and otherwise rebuilt by walking ./models and every HF cache root, under a lock the next caller waits on. On an install with a large cache that scan measured 6.1s, longer than the TTL that is meant to amortise it, so steady traffic would keep rebuilding it. Answer from the last built index instead and never rebuild from the request path: a stale answer is fine here, since what is on disk barely moves and a finished download already invalidates the index. The first request, before any scan has completed, warms the index on a background thread and skips the check rather than blocking on it. That also makes the lookup a dict read, so it no longer needs handing to a thread. Cold resolve on this box goes from 6152495us to 0.4us, and the whole hook now costs the same for a foreign label as for the resident model. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: fix the admission hook's cold, stale and contended index paths Five review items, four of them on the admission hook added here. Skipping the check until the first scan lands also skipped explicit quant mismatches, so the first request after startup could ask for :Q8_0 while Q4_K_M was resident and be answered by it. The early return was redundant as well: with an empty index resolved is None and here is False, so the gate below already lets a bare name through and refuses an explicit quant, which is what the except branch has always concluded. Dropped it and index_is_built with it. index_is_built took _lock, which _index holds for the whole scan, so once a warm was running every later request blocked on the event loop for exactly as long as the scan it was there to avoid. The warm now has its own lock and reads the timestamp unlocked, which is safe because _scan is only ever rebound. Warming only when the index had never been built left a model fetched in the Hub UI, or dropped into a scan folder, invisible for the life of the process, since only the auto-download watcher calls invalidate_index. Warm on staleness too, and unconditionally, so it refreshes within a TTL without a scan on the request path. Rescanning is capped at a tenth of the scan's own duration: a big install takes longer to scan than the TTL, and warming on the TTL alone would keep a thread scanning continuously. An Ollama-style tag names no quant, so the resolver misses it and auto-download saw a model the resident one already answers to, then 404'd it for having no such quant. Return early when the loaded model satisfies the reference. Frontend: a cancelled download said "Model download failed", because the label collapsed everything non-completed into failure. The backend tests get an autouse fixture that stops the warm from walking the developer's real HF caches; that scan starved the loop under the timing sensitive streaming tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make /v1/models and the admission hook agree on what is local Three review items, all on the seam between the catalog scan and the resolver index, which run on separate schedules. /v1/models can advertise a local GGUF the resolver has not indexed yet. A bare id carries no quant to refuse on, so a client asking for one it had just been handed was answered by the resident model instead. The hook now reads the catalog cache as evidence too, never scanning it. It takes the path rather than a yes/no because the converse also happens: the catalog can list the resident weights under an alias the loaded entry does not answer to, and those must stay served. That alias was also emitted twice by /v1/models, once as the loaded basename a manual load records and once as publisher/model marked unloaded, because the dedup only compared ids. Compare the path as well. A directly loaded standalone .gguf takes its quant from the filename, but the resolver stores such files with no quants, so the advertised <stem>:<quant> stopped resolving as soon as anything else loaded. Advertise a quant only when that reference resolves, and downgrade only on a definite answer so a cold index leaves the metadata alone. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten the comments this branch adds Collapse the multi-line notes in the auto-download path, the /v1 admission hook and their tests to one line each, keeping the reason and dropping the restatement. No behaviour change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: four admission and catalog fixes from review Lowercasing paths in _resolves_to_resident made /srv/models/Foo and /srv/models/foo the same weights on any case-sensitive filesystem, so a request for one could be answered by the other and /v1/models could mark the wrong entry loaded. That helper now backs residency as well as admission, so use os.path.normcase, which folds case only where the filesystem does. Advertising a quant whenever the resolver could not disprove it kept the bug it was meant to fix: a standalone .gguf loaded before the first scan still got <stem>:<quant> published, and the usage examples persist that. No proof is not proof, so omit it and warm the index instead. A 401 from an expired or invalid X-Unsloth-HF-Token skipped the 403 and 404 branches and surfaced as "could not reach Hugging Face, retry shortly". It now says to replace the token, kept apart from the gated refusal since a rejected credential is not an unaccepted licence. An image request naming an undownloaded text-only GGUF started the whole download and only then hit the capability guard, which never sees a remote target, so every retry 400d and the bytes were wasted. Thread require_vision into admission and check it against the mmproj companions the disk preflight already asks build_gguf_variant_plans for. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make the Hub error fixture carry a status on both hub majors The 401 test built HfHubHTTPError directly, which works on 0.x and fails on 1.x where response is required and keyword-only, so all four Python jobs failed while the same test passed locally. _hub_error already handled both constructors, but the 0.x branch left the exception with no response at all, and hf_error_status reads the status off it for the types that do not encode it in their name. So it could only produce a usable error on 1.x, which is why the test bypassed it. Attach the status when the constructed exception lacks it, and use the helper. Cover the helper itself against stand-ins for both constructor shapes, since whichever hub is installed only ever exercises one of them. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: invalidate on every download, resolve bare tags, keep polling Three review items. Only the API auto-download watcher dropped the resolver cache, so a GGUF fetched in the Hub UI stayed absent to the cache-only request path and the request was answered by whatever was resident. finalize_worker_exit is the one point every download worker exits through, so invalidate there. That closes the window without leaning on the TTL, which the scan-duration throttle can stretch past 5s on an install where the scan itself takes longer than that. A downloaded but unloaded GGUF asked for as org/model:latest missed the resolver, since the suffix was always treated as an exact quant. With auto-download on that probed the Hub and returned a 404 for a quant that was never a quant; with it off it refused without switching. Fall back to the base entry when the suffix is not quant-shaped, and keep exact matching for real quants so a swap can never serve the wrong weights under the right name. The usage examples stopped polling once a model was resident, but idle unload frees one without touching the store, so nothing re-ran the effect and the examples kept naming a model that could no longer be reloaded. Slow the poll to 60s instead of stopping it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: hold the download slot while it is in use, and keep quants to llama.cpp _loaded_satisfies refuses a quant reference against the Transformers backend by name, but the path match did not carry that rule. A Transformers model active from a directory that also holds GGUF exports therefore matched a request for one of those quants and answered it with the safetensors weights. Only llama.cpp has a quant identity, so admission now passes llama_only whenever the reference is quant-qualified. A bare name still matches either backend, and /v1/models residency keeps the default so a loaded Transformers model is still reported loaded. The 24 hour watch window was bounding ownership of the single-flight slot when it should only have been bounding progress reporting, so a legitimately slow download had its slot handed back while the worker was still writing, admitting a second multi-gigabyte download beside it. Resolve the row on the clock, but keep the slot on a slower poll until the job is actually terminal. Past the deadline an unknown state does release it, since it means the worker cannot be probed and holding it on that forever would wedge auto-download. * Studio: keep what the resolver already knew when a download lands Invalidating cleared the index to empty. The request path reads that cache without scanning, so from a completed download until the rebuild landed it had no evidence about any local model, not just the new one, and a bare request for any of them was answered by whatever was resident. Wiring the hook into the shared completion path in the last commit widened that from auto-download to every download. Mark the scan stale and keep the entries instead. Both _index and warm_index_soon rebuild on a zero stamp, while the request path still sees everything it knew a moment ago. Only a completed download invalidates, and that only ever adds models, so nothing retained goes false. Warm from the completion hook too, so the rebuild starts when the download lands rather than when the next request happens to need it. * Studio: match the quant, not just the directory, and default-select bare tags Two quants of one repo share a directory, so the path match could not tell them apart and an explicit :Q8_0 was answered by a resident Q4_K_M that _loaded_satisfies had already refused by name. The llama_only fix in the last commit only ruled out the wrong backend, not the wrong quant on the right one. Both path matches now require the resident hf_variant to equal the requested quant whenever the reference is quantified; a bare name still matches on the path alone, since it claims nothing about the weights. The local resolver already treated a tag that names no quant as meaning the repo, but remote admission still looked for a quant literally called "latest", so the same reference resolved locally and 404d remotely. Branch on looks_like_quant there too. A real quant the repo does not have is still a 404 and never a substitution, which is what separates this from the loader's low-disk fallback. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: one quant preference, and stop trusting a stale checkpoint list_local_gguf_variants sorts by descending size, so the head of variants was the biggest quant, often F16, while remote admission and a plain load both rank through _pick_best_gguf. A bare id therefore meant a different quant depending on which side answered it, and the local answer was the one that could evict a working model and then fail or OOM starting an F16 next to a usable Q4. /v1/models advertised that same head for pinning. Pull the ranking into one preferred_quant helper and have both sides use it. The usage examples returned a stored checkpoint without ever consulting /v1/models, and the polling added last round was gated on not having one, so for a stored checkpoint it never ran. An idle unload then left the panel showing a snippet that could not run. Poll whenever mounted, and prefer the checkpoint only while the catalog still backs it or switching can reload it. A catalog that has not answered yet is not evidence against it. The static contract pinned the old dependency array, so it now asserts the intent it documents: a finished load re-runs the fetch, and the effect is not gated on having no checkpoint. * Studio: fix the Windows path compare, and advertise a label the worker knows The case fix normalized the separator to "/" and then called os.path.normcase, which on Windows folds case and rewrites the separator back to a backslash, so the descendant checks compared against a "/" the path no longer had. A manually loaded GGUF reached through an alias then read as a different model, giving a false 404 and an alias marked unloaded. Run normcase first and normalize the separator after it. There are two quant-label extractors and they only agree while a recognized quant token is present. With none, _extract_quant_label takes the last hyphenated segment, "7b" of llama-7b, while build_gguf_variant_plans and the worker key the whole stem: the plan lookup missed and the job exited on a variant it had no shards for. Use the canonical extractor for the unrecognized case only. Checked across real filenames first, the two match on every recognized quant and part on bpw-qualified labels, which _extract_quant_label keeps apart on purpose so byteshape's IQ4_XS at 3.53, 3.97 and 4.19 bpw stay separate variants. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: a stored checkpoint needs catalog evidence, not just the switch setting Preferring it whenever switching was on short-circuited the catalog check, so a checkpoint the store still held after the model was deleted or moved kept being named even though /v1/models had already proved it absent, and the snippets 404d instead of falling back to a model that is actually there. A lookup rather than a disjunction, which settles the whole matrix in one place: no answer yet keeps the checkpoint, since that is not evidence against it; listed and resident keeps it; listed but unloaded keeps it only when switching can reload it; absent falls back whatever the setting says. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: normalize the quote style pre-commit would have rewritten * Studio: cover the model that just landed, and pin the quant the catalog has Retaining the index on invalidation protects what was already scanned and by construction cannot contain the model that just finished downloading, so a bare request for it in the window before the rebuild was still answered by the resident model. Record the repo at the completion hook and treat that as admission evidence alongside the resolver and the catalog; the next completed scan clears the notes, since the index then covers them. Publishing a rebuilt index before completion becomes observable would have closed it too, but that blocks the download worker for the length of the scan. Catalog membership proves the repo, not the saved quant, and the examples then pinned the stored one. A quant deleted while another quant of the same repo remained produced repo:deleted-quant, a missing-quant 404 with a runnable alternative listed right beside it. Pin what the catalog advertises: for a resident entry that is the resident quant, for an unloaded one it is a quant actually on disk. The store is only consulted before /v1/models has answered. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: apply three rules everywhere they belong, not only where reported The trust probe was the last credential handoff still passing a raw token. huggingface_hub reads None as "use the cached login", so a caller-named repo was read with this server's Hugging Face identity whenever the caller sent none, which is exactly the isolation the metadata probe and the worker already keep. It takes _hub_token now. Enumerated the rest of that path while there: auth_check, model_info and spawn_worker were already correct. finalize_worker_exit is shared with dataset downloads, so the resolver hook fired for every completed dataset, scanning the model directories for nothing and recording the dataset id as local-model evidence, which turns a bare /v1 request naming that id into a refusal instead of a foreign-id fallthrough. Gated on repo_type. _already_serving decided "bare" on the presence of a colon while _loaded_satisfies and the resolver decide it on whether the suffix names a quant, so org/model:latest against a serving Q8_0 read as a mismatch and swapped in the preferred Q4_K_M for a request either one answers. That rule now lives in four places, each fixed in its own round, so this time I looked for the rest and found a fifth: describe_local_miss splits on the bare colon and its docstring claims it splits like the resolver. It no longer did, and would report a missing quant named "latest". Fixed here too, unreported. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: probe before refusing busy, and scan once when the index is cold The busy refusal fired before anything established the requested label was a model at all, so any namespaced id a drop-in client sends was told to wait out an unrelated download for as long as it ran. Probe first and refuse only a label the Hub actually serves as GGUF; anything else falls through to the resident model as before. A probe failure answers "not downloadable", since stranding ordinary traffic costs more than missing a busy refusal. Treating an unbuilt index as "nothing here" let the first request after startup be answered by the resident model under another model's name. That was a deliberate trade to keep the scan off the request path, and it was the wrong one. Cold, the scan now runs once on a thread, bounded so a pathological install falls through rather than hanging the request. Built, the request path still never scans, so the latency fix stands. The watcher freed the slot the moment it saw an error, while Retry-After is thirty times the poll interval, so the client came back to an empty slot and restarted the same failing download instead of being told. Hold the failure on the slot until a retry surfaces it, and let another repo take it after three retry intervals so a client that never returns cannot keep it. The watcher also invalidated on completion, which now lands after finalize_worker_exit's warm and marks that fresh scan stale, pushing a synchronous rescan onto the retry. Removed. _loaded_satisfies lowercased paths as well as aliases, so it returned satisfied before the case-preserving compare below could run. Both now go through one helper: paths compare with normcase, aliases stay case-insensitive. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: an unfinished scan is not absence, and a decided refusal is not a failure Bounding the cold scan then reading the bound as "not here" left the same hole one branch over. A timeout now answers 503 model_indexing with a Retry-After and leaves the warm running. A foreign label sent inside that window is asked to retry rather than falling through, which is a real cost, but the window is one request on an install whose scan exceeds ten seconds and it clears itself, where answering with the wrong weights does not. That uncovered a worse one. Every check here runs inside a broad except whose job is "could not verify, so fall through", so an HTTPException raised in the block was logged as a verification failure and the request was answered by the resident model. Any refusal decided in there was being swallowed. Re-raise it ahead of that handler. Canonicalizing generic labels made them real variant keys, but the matcher still decided on shape, so repo:llama-13b fell past an exact match and fetched llama-7b. Match exactly first, whatever the shape; a quant-shaped suffix that matches nothing is still a miss and never a swap. Marking a catalog alias loaded while publishing the preferred on-disk quant claimed alias:Q4 was loaded while Q8 was serving, and requiring the resident quant to match then made pinning it a 404. Advertise the resident variant when the entry resolves to the resident model. * Studio: keep the asyncio.timeout fallback tests runnable on Python 3.10 Both tests deleted asyncio.timeout to force _wall_clock_timeout down its pre-3.11 branch, but monkeypatch.delattr raises when the attribute is already absent. On Python 3.10, the one version the fallback exists for, there is nothing to delete, so the two tests errored with AttributeError before reaching the code they cover. Passing raising=False makes the deletion a no-op there and leaves the assertions running against the same branch on every version. Every other delattr in the repo already passes raising=False for exactly this reason. Verified with asyncio.timeout removed from the interpreter: the two tests fail with the CI AttributeError before this change and pass after, and the file still runs 89 passed on 3.13 where the deletion is real. * Studio: decide GGUF residency, servability and variant keys by one rule each Four admission and catalog fixes, each closing a gap between two places that were answering the same question differently. The /v1/models catalog asked _resolves_to_resident without llama_only, so a Transformers model live from a directory that also holds GGUF exports marked a GGUF alias loaded and gave it a GGUF quant. The usage examples then pinned alias:quant that nothing could serve with switching off. Every entry in that loop is advertised as GGUF, so residency there is llama.cpp residency. The busy probe accepted any .gguf sibling while admission excludes mmproj, MTP drafters and big-endian builds. A repo holding only companions is not downloadable, so it was held at model_download_busy for the length of an unrelated download instead of falling through to the resident model as it does when no download is running. It now reuses _gguf_variants, the same filter. split_model_ref refused any slash-bearing suffix, but an unrecognized GGUF below a subdirectory keys on its path (build/llama-13b), which is_valid_gguf_variant allows and the catalog advertises. Pinning such a variant could not parse, so only the default-ranked one was reachable. A slash-bearing suffix is now a variant exactly when a real Hub repo precedes it, which still leaves C:/models/x.gguf a path rather than a quant. The usage examples treated a downloaded-but-unloaded model as runnable only under auto-switch, but a standalone UNSLOTH_MODEL_IDLE_TTL reloads exactly what it freed on the next request. The panel hid runnable examples after an idle unload. Tracked apart from auto-switch, because the stash restores the stored checkpoint only and never an arbitrary catalog entry. Also stub the index walk in the three cold-index tests that missed it: a real multi-root scan inside the cold-wait budget made them time out into a 503 under load rather than assert what they are there for. One of them flaked locally. Verified each fix is load-bearing by reverting it and watching its test fail. Backend CI command: 10195 passed, 0 failed. tsc -b and the frontend build clean. * Studio: bound the Hub admission probes and stop guessing at nested model paths Three review fixes plus a test-isolation one. _resolves_to_resident matched on a path prefix, so two separately indexed models that nest (/models/A alongside /models/A/sub/B) both satisfied it: loading B made a request for A resident and answered it with B's weights, and the catalog marked A loaded. A prefix match now counts only when no catalog entry sits deeper, which is the innermost indexed model that actually owns the file. With nothing indexed there is no nesting to tell apart, so the directory-to-weights match this exists for is unchanged. auth_check and hf_hub_download take no timeout of their own, and both ran while the provisional single-flight slot was held, so an unresponsive Hub stalled the request far past the metadata budget and reported every other model busy for the duration. Both are bounded now. Each default errs the safe way: an unchecked repo is not a cleared one, so the custom-code probe refuses on timeout, while a slow gated-repo check stays inconclusive because the download's own auth is the real gate. The usage examples caught a failed refresh into an empty catalog and a disabled auto-switch, which made a transient error authoritative and blanked every example while the model was still servable. The catalog is deliberately tri-state; a failure now keeps the last answer and retries. Also start the backend tests from a built, empty model index. Stubbing only the background warm still left the cold path walking real caches synchronously inside the admission wait, so on a large install a test asserted against a 503 "still indexing" instead of its subject. _build_index is untouched, so the tests that call it directly still exercise the real walk. Verified each fix is load-bearing by reverting it and watching its test fail. tsc -b clean. Backend CI command green apart from two failures reproduced only on this box (a real model-dir scan and an orphan-process cleanup), neither touched by this PR; staging CI is the gate for those. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1798 lines
72 KiB
Python
1798 lines
72 KiB
Python
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
|
|
|
|
"""Opt-in auto-download of a GGUF a /v1 request names but this server lacks.
|
|
|
|
No network: huggingface_hub, the consent probe and the Hub download service are
|
|
all mocked. The invariant these guard is that with the setting off nothing here
|
|
runs at all, and with it on a name that isn't shaped like a repo still falls
|
|
through to the resident model.
|
|
"""
|
|
|
|
import asyncio
|
|
import time
|
|
|
|
import pytest
|
|
from fastapi import HTTPException
|
|
|
|
import routes.inference as inference_route
|
|
from core.inference import openai_auto_download as auto_dl
|
|
from core.inference.local_model_resolver import warm_index_soon as _real_warm_index_soon
|
|
from utils import openai_auto_switch_settings as settings
|
|
|
|
|
|
class _Sibling:
|
|
def __init__(
|
|
self,
|
|
rfilename,
|
|
size = 0,
|
|
blob_id = None,
|
|
):
|
|
self.rfilename = rfilename
|
|
self.size = size
|
|
self.blob_id = blob_id
|
|
|
|
|
|
class _Info:
|
|
def __init__(
|
|
self,
|
|
siblings,
|
|
sha = "abc123",
|
|
gated = False,
|
|
private = False,
|
|
):
|
|
self.siblings = siblings
|
|
self.sha = sha
|
|
self.gated = gated
|
|
self.private = private
|
|
|
|
|
|
def _gguf_repo_info():
|
|
gb = 1024**3
|
|
return _Info(
|
|
[
|
|
_Sibling("model-UD-Q4_K_XL.gguf", 4 * gb),
|
|
_Sibling("model-UD-Q5_K_XL.gguf", 5 * gb),
|
|
_Sibling("model-Q8_0-00001-of-00002.gguf", 4 * gb),
|
|
_Sibling("model-Q8_0-00002-of-00002.gguf", 4 * gb),
|
|
_Sibling("mmproj-F16.gguf", 1 * gb),
|
|
_Sibling("mtp-model.gguf", 1 * gb),
|
|
_Sibling("README.md", 1024),
|
|
]
|
|
)
|
|
|
|
|
|
@pytest.fixture(autouse = True)
|
|
def _clean_slot():
|
|
from core.inference import local_model_resolver
|
|
|
|
auto_dl.reset_for_tests()
|
|
# The hook warms the index in the background; drop it so a scan never leaks between tests.
|
|
local_model_resolver.invalidate_index()
|
|
yield
|
|
auto_dl.reset_for_tests()
|
|
local_model_resolver.invalidate_index()
|
|
|
|
|
|
def _repo_not_found_error():
|
|
from huggingface_hub.utils import RepositoryNotFoundError
|
|
return RepositoryNotFoundError
|
|
|
|
|
|
def _gated_error():
|
|
from huggingface_hub.utils import GatedRepoError
|
|
return GatedRepoError
|
|
|
|
|
|
def _hub_error(error_type, status_code: int, message: str):
|
|
"""Build a Hub exception across huggingface_hub majors.
|
|
|
|
huggingface_hub 1.x made ``response`` a required keyword-only argument, and
|
|
the project floor is 0.34, so construct positionally and fall back. The
|
|
positional form carries no response, and hf_error_status reads the status off
|
|
it for the types that do not encode it in their name, so attach one either way.
|
|
"""
|
|
try:
|
|
exc = error_type(message)
|
|
except TypeError:
|
|
import httpx
|
|
exc = error_type(
|
|
message,
|
|
response = httpx.Response(
|
|
status_code,
|
|
request = httpx.Request("GET", "https://huggingface.co/api/models/org/repo"),
|
|
),
|
|
)
|
|
if getattr(getattr(exc, "response", None), "status_code", None) != status_code:
|
|
from types import SimpleNamespace
|
|
try:
|
|
exc.response = SimpleNamespace(status_code = status_code)
|
|
except AttributeError:
|
|
pass
|
|
return exc
|
|
|
|
|
|
def test_the_hub_error_helper_carries_a_status_on_both_majors():
|
|
# CI runs huggingface_hub 1.x and this box runs 0.x, and only one of the two
|
|
# constructor shapes works on each. hf_error_status reads the status off the
|
|
# response, so a helper that silently produced one without it would make an
|
|
# error-mapping test pass here and fail there.
|
|
from hub.utils.hf_errors import hf_error_status
|
|
|
|
class _Legacy(Exception):
|
|
"""0.x: response is optional and unset when built positionally."""
|
|
|
|
class _Modern(Exception):
|
|
"""1.x: response is required and keyword-only."""
|
|
|
|
def __init__(self, message, *, response):
|
|
super().__init__(message)
|
|
self.response = response
|
|
|
|
for error_type in (_Legacy, _Modern):
|
|
assert hf_error_status(_hub_error(error_type, 401, "unauthorized")) == 401
|
|
|
|
|
|
@pytest.fixture
|
|
def hub(monkeypatch):
|
|
"""Wire the whole remote surface to fakes and record what was dispatched."""
|
|
import huggingface_hub
|
|
from hub.services.models import downloads
|
|
|
|
state = {
|
|
"info": _gguf_repo_info(),
|
|
"raise": None,
|
|
"auto_map": False,
|
|
"started": [],
|
|
"watched": [],
|
|
# What the hub service returns; accepted=False means no worker was launched.
|
|
"dispatch_result": {"job_key": "k", "state": "running", "accepted": True},
|
|
"on_probe": None,
|
|
"probes": 0,
|
|
"auth_denied": False,
|
|
"allow_ambient": None,
|
|
}
|
|
|
|
class _FakeApi:
|
|
def __init__(self, token = None):
|
|
state["token"] = token
|
|
|
|
def model_info(self, repo_id, **kwargs):
|
|
state["probes"] += 1
|
|
if state["on_probe"] is not None:
|
|
state["on_probe"]()
|
|
if state["raise"] is not None:
|
|
raise state["raise"]
|
|
return state["info"]
|
|
|
|
async def _start(
|
|
body,
|
|
hf_token = None,
|
|
*,
|
|
allow_ambient_token = True,
|
|
):
|
|
state["started"].append((body.repo_id, body.gguf_variant, hf_token))
|
|
state["allow_ambient"] = allow_ambient_token
|
|
return state["dispatch_result"]
|
|
|
|
async def _no_watch(active, hf_token):
|
|
state["watched"].append(active)
|
|
return None
|
|
|
|
monkeypatch.setattr(huggingface_hub, "HfApi", _FakeApi)
|
|
monkeypatch.setattr(downloads, "download_model_response", _start)
|
|
# Keep the real watcher reachable: one test drives its cleanup directly.
|
|
state["real_watch"] = auto_dl._watch
|
|
monkeypatch.setattr(auto_dl, "_watch", _no_watch)
|
|
monkeypatch.setattr(auto_dl, "_enough_disk", lambda need: (True, 10 * 1024**4))
|
|
monkeypatch.setattr(auto_dl, "_auth_denied", lambda repo, token: state["auth_denied"])
|
|
monkeypatch.setattr(
|
|
"utils.security.consent._config_has_auto_map",
|
|
lambda repo, token = None: state["auto_map"],
|
|
)
|
|
return state
|
|
|
|
|
|
def _run(model, hf_token = None):
|
|
return asyncio.run(auto_dl.maybe_auto_download(model, hf_token = hf_token))
|
|
|
|
|
|
# --- pure helpers ------------------------------------------------------------
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"raw,expected",
|
|
[
|
|
("org/repo:UD-Q4_K_XL", ("org/repo", "UD-Q4_K_XL")),
|
|
("org/repo", ("org/repo", None)),
|
|
("gpt-4", ("gpt-4", None)),
|
|
# A colon followed by a path segment is not a quant.
|
|
("C:/models/x.gguf", ("C:/models/x.gguf", None)),
|
|
("org/repo:", ("org/repo:", None)),
|
|
# An unrecognized GGUF below a subdirectory keys on its path, and that key is
|
|
# what the catalog advertises, so pinning it has to parse.
|
|
("org/repo:build/llama-13b", ("org/repo", "build/llama-13b")),
|
|
# Still a path, not a variant: no Hub repo precedes the colon.
|
|
("/home/me/models/x:build/llama-13b", ("/home/me/models/x:build/llama-13b", None)),
|
|
("D:/models/repo:build/llama-13b", ("D:/models/repo:build/llama-13b", None)),
|
|
],
|
|
)
|
|
def test_split_model_ref(raw, expected):
|
|
assert auto_dl.split_model_ref(raw) == expected
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"raw",
|
|
[
|
|
"gpt-4", # no namespace: a foreign id, must keep falling through
|
|
"gpt-4o-mini",
|
|
"../../etc/passwd",
|
|
"https://evil.example/x",
|
|
"/abs/path/model.gguf",
|
|
"org/repo/extra",
|
|
"org/re..po",
|
|
"org/repo\nX-Injected: 1",
|
|
"",
|
|
],
|
|
)
|
|
def test_not_downloadable(raw):
|
|
assert auto_dl.is_downloadable_ref(raw) is False
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"raw", ["unsloth/gemma-4-31B-it-GGUF", "unsloth/gemma-4-31B-it-GGUF:UD-Q5_K_XL"]
|
|
)
|
|
def test_downloadable(raw):
|
|
assert auto_dl.is_downloadable_ref(raw) is True
|
|
|
|
|
|
def test_gguf_variants_skips_companions():
|
|
variants = auto_dl._gguf_variants(_gguf_repo_info().siblings)
|
|
# Companions are not quants of their own...
|
|
assert set(variants) == {"UD-Q4_K_XL", "UD-Q5_K_XL", "Q8_0"}
|
|
# ...but every quant fetches them, so they count, and shards sum on top.
|
|
companions = 2 * 1024**3 # mmproj + MTP drafter
|
|
assert variants["Q8_0"] == 8 * 1024**3 + companions
|
|
assert variants["UD-Q4_K_XL"] == 4 * 1024**3 + companions
|
|
|
|
|
|
def test_looks_like_quant_separates_quants_from_foreign_tags():
|
|
assert auto_dl.looks_like_quant("UD-Q6_K_XL")
|
|
assert auto_dl.looks_like_quant("q4_k_m")
|
|
assert auto_dl.looks_like_quant("F16")
|
|
# Ollama-style tags are not quants and must not read as a GGUF reference.
|
|
assert not auto_dl.looks_like_quant("latest")
|
|
assert not auto_dl.looks_like_quant("8b")
|
|
assert not auto_dl.looks_like_quant(None)
|
|
|
|
|
|
def test_match_variant_is_case_insensitive_and_exact():
|
|
variants = {"UD-Q4_K_XL": 1, "Q8_0": 2}
|
|
assert auto_dl._match_variant("ud-q4_k_xl", variants) == "UD-Q4_K_XL"
|
|
assert auto_dl._match_variant("Q5_K_M", variants) is None
|
|
# A bare id picks a real local label, never invents one.
|
|
assert auto_dl._match_variant(None, variants) in variants
|
|
|
|
|
|
# --- admission ---------------------------------------------------------------
|
|
|
|
|
|
def test_foreign_id_never_probes(hub):
|
|
assert _run("gpt-4") is None
|
|
assert hub["started"] == []
|
|
|
|
|
|
def test_starts_download_and_asks_for_a_retry(hub):
|
|
refusal = _run("unsloth/x-GGUF:UD-Q5_K_XL")
|
|
assert refusal.status == 503
|
|
assert refusal.code == "model_downloading"
|
|
assert refusal.retry_after and refusal.retry_after > 0
|
|
assert "unsloth/x-GGUF:UD-Q5_K_XL" in refusal.message
|
|
assert hub["started"] == [("unsloth/x-GGUF", "UD-Q5_K_XL", None)]
|
|
|
|
|
|
def test_bare_id_freezes_the_same_quant_a_manual_load_would_pick(hub):
|
|
from utils.models.model_config import _extract_quant_label, _pick_best_gguf
|
|
|
|
refusal = _run("unsloth/x-GGUF")
|
|
assert refusal.status == 503
|
|
repo, variant, _token = hub["started"][0]
|
|
expected = _extract_quant_label(
|
|
_pick_best_gguf([s.rfilename for s in _gguf_repo_info().siblings])
|
|
)
|
|
assert (repo, variant) == ("unsloth/x-GGUF", expected)
|
|
assert variant == "UD-Q4_K_XL"
|
|
|
|
|
|
def test_missing_quant_lists_the_real_ones(hub):
|
|
refusal = _run("unsloth/x-GGUF:Q2_K")
|
|
assert refusal.status == 404 and refusal.code == "model_not_found"
|
|
assert "UD-Q4_K_XL" in refusal.message and "Q8_0" in refusal.message
|
|
assert hub["started"] == []
|
|
|
|
|
|
def test_missing_repo_is_404_without_confirming_existence(hub):
|
|
hub["raise"] = _hub_error(_repo_not_found_error(), 404, "nope")
|
|
# An explicit quant is a deliberate GGUF reference, so a miss is answered.
|
|
refusal = _run("unsloth/not-real:UD-Q4_K_XL")
|
|
assert refusal.status == 404 and refusal.code == "model_not_found"
|
|
assert "not accessible" in refusal.message
|
|
assert hub["started"] == []
|
|
|
|
|
|
def test_an_id_the_hub_does_not_know_falls_through(hub):
|
|
hub["raise"] = _hub_error(_repo_not_found_error(), 404, "nope")
|
|
# "vendor/model" is how LiteLLM names providers, so an unknown id stays a foreign label.
|
|
for foreign in (
|
|
"anthropic/claude-3.5-sonnet",
|
|
"openai/gpt-4o",
|
|
"meta-llama/llama-3-70b-instruct",
|
|
):
|
|
assert _run(foreign) is None
|
|
assert hub["started"] == []
|
|
|
|
|
|
def test_a_foreign_id_is_probed_once_then_cached(hub):
|
|
hub["raise"] = _hub_error(_repo_not_found_error(), 404, "nope")
|
|
assert _run("anthropic/claude-3.5-sonnet") is None
|
|
assert hub["probes"] == 1
|
|
# Every later request would otherwise pay another Hub round trip.
|
|
assert _run("anthropic/claude-3.5-sonnet") is None
|
|
assert hub["probes"] == 1
|
|
|
|
|
|
def test_an_anonymous_404_does_not_silence_an_authorised_caller(hub):
|
|
# The Hub 404s a private repo, so a global verdict would hide it from the token holder.
|
|
hub["raise"] = _hub_error(_repo_not_found_error(), 404, "nope")
|
|
assert _run("myorg/private-GGUF") is None
|
|
assert hub["probes"] == 1
|
|
|
|
hub["raise"] = None
|
|
refusal = _run("myorg/private-GGUF", hf_token = "hf_caller_own")
|
|
assert hub["probes"] == 2
|
|
assert refusal.code == "model_downloading"
|
|
|
|
|
|
def test_the_cache_is_per_token(hub):
|
|
hub["raise"] = _hub_error(_repo_not_found_error(), 404, "nope")
|
|
assert _run("myorg/private-GGUF", hf_token = "hf_a") is None
|
|
assert _run("myorg/private-GGUF", hf_token = "hf_a") is None
|
|
assert hub["probes"] == 1
|
|
# A different credential gets its own verdict.
|
|
assert _run("myorg/private-GGUF", hf_token = "hf_b") is None
|
|
assert hub["probes"] == 2
|
|
|
|
|
|
def test_the_gated_message_names_the_header_that_actually_works(hub):
|
|
# Auto-download never uses the server's token, so a Studio setting would loop the caller.
|
|
hub["info"] = _Info(_gguf_repo_info().siblings, gated = "manual")
|
|
hub["auth_denied"] = True
|
|
refusal = _run("meta-llama/Llama-2-7b-hf")
|
|
assert "X-Unsloth-HF-Token" in refusal.message
|
|
|
|
|
|
def test_gated_repo_is_403(hub):
|
|
hub["raise"] = _hub_error(_gated_error(), 403, "gated")
|
|
refusal = _run("meta-llama/Llama-2-7b-hf")
|
|
assert refusal.status == 403 and refusal.code == "model_access_denied"
|
|
assert hub["started"] == []
|
|
|
|
|
|
def test_a_gated_repo_that_still_returns_metadata_is_403(hub):
|
|
# Metadata for a gated repo is not file access, so report the licence gate, not custom code.
|
|
hub["info"] = _Info(_gguf_repo_info().siblings, gated = "manual")
|
|
hub["auth_denied"] = True
|
|
refusal = _run("meta-llama/Llama-2-7b-hf")
|
|
assert refusal.status == 403 and refusal.code == "model_access_denied"
|
|
assert "licence" in refusal.message
|
|
assert hub["started"] == []
|
|
|
|
|
|
def test_a_gated_repo_this_token_may_read_still_downloads(hub):
|
|
hub["info"] = _Info(_gguf_repo_info().siblings, gated = "manual")
|
|
refusal = _run("meta-llama/Llama-2-7b-hf")
|
|
assert refusal.code == "model_downloading"
|
|
assert len(hub["started"]) == 1
|
|
|
|
|
|
def test_hub_unreachable_is_retryable(hub):
|
|
hub["raise"] = OSError("network down")
|
|
refusal = _run("unsloth/x-GGUF")
|
|
assert refusal.status == 503 and refusal.code == "model_lookup_failed"
|
|
assert refusal.retry_after
|
|
assert hub["started"] == []
|
|
|
|
|
|
def test_non_gguf_repo_is_refused(hub):
|
|
hub["info"] = _Info([_Sibling("model.safetensors", 100), _Sibling("config.json", 10)])
|
|
refusal = _run("unsloth/plain-transformers:Q4_K_M")
|
|
assert refusal.status == 400 and refusal.code == "model_not_supported"
|
|
assert hub["started"] == []
|
|
|
|
|
|
def test_a_bare_non_gguf_id_falls_through(hub):
|
|
# Without a quant this is indistinguishable from a foreign provider label.
|
|
hub["info"] = _Info([_Sibling("model.safetensors", 100)])
|
|
assert _run("unsloth/plain-transformers") is None
|
|
assert hub["started"] == []
|
|
|
|
|
|
def test_remote_code_repo_is_refused(hub):
|
|
hub["auto_map"] = True
|
|
refusal = _run("someone/custom-arch-GGUF")
|
|
assert refusal.status == 403 and refusal.code == "remote_code_consent_required"
|
|
assert "Unsloth Studio" in refusal.message
|
|
assert hub["started"] == []
|
|
|
|
|
|
def test_unreadable_config_fails_closed(hub):
|
|
# _config_has_auto_map returns None when it cannot tell; never assume safe.
|
|
hub["auto_map"] = None
|
|
refusal = _run("someone/unknown-GGUF")
|
|
assert refusal.status == 403 and refusal.code == "remote_code_consent_required"
|
|
assert hub["started"] == []
|
|
|
|
|
|
def test_insufficient_disk_never_downgrades_the_quant(hub, monkeypatch):
|
|
monkeypatch.setattr(auto_dl, "_enough_disk", lambda need: (False, 1024**3))
|
|
refusal = _run("unsloth/x-GGUF:UD-Q5_K_XL")
|
|
assert refusal.status == 507 and refusal.code == "insufficient_disk_space"
|
|
assert hub["started"] == []
|
|
|
|
|
|
def test_second_model_waits_for_the_first(hub):
|
|
assert _run("unsloth/first-GGUF").code == "model_downloading"
|
|
refusal = _run("unsloth/second-GGUF")
|
|
assert refusal.status == 503 and refusal.code == "model_download_busy"
|
|
assert "unsloth/first-GGUF" in refusal.message
|
|
# Only the first was dispatched.
|
|
assert len(hub["started"]) == 1
|
|
|
|
|
|
def test_repeat_request_reports_progress_without_reprobing(hub, monkeypatch):
|
|
assert _run("unsloth/x-GGUF:UD-Q5_K_XL").code == "model_downloading"
|
|
|
|
async def _running(repo, variant):
|
|
return "running", None
|
|
|
|
async def _pct(repo, variant, expected, token):
|
|
return 42.0
|
|
|
|
monkeypatch.setattr(auto_dl, "_job_state", _running)
|
|
monkeypatch.setattr(auto_dl, "_progress_percent", _pct)
|
|
refusal = _run("unsloth/x-GGUF:UD-Q5_K_XL")
|
|
assert refusal.code == "model_downloading" and "42%" in refusal.message
|
|
assert len(hub["started"]) == 1
|
|
|
|
|
|
def test_progress_is_scaled_to_a_percentage(monkeypatch):
|
|
# The hub service reports a 0-1 fraction; a raw 0.492 would render as "0%".
|
|
from hub.services.models import downloads
|
|
|
|
async def _fraction(
|
|
repo_id,
|
|
variant = "",
|
|
expected_bytes = 0,
|
|
hf_token = None,
|
|
):
|
|
return {"progress": 0.492}
|
|
|
|
monkeypatch.setattr(downloads, "get_gguf_download_progress_response", _fraction)
|
|
percent = asyncio.run(auto_dl._progress_percent("org/repo", "Q4_K_M", 0, None))
|
|
assert percent == pytest.approx(49.2)
|
|
|
|
|
|
def test_failed_job_surfaces_once_then_frees_the_slot(hub, monkeypatch):
|
|
assert _run("unsloth/x-GGUF").code == "model_downloading"
|
|
|
|
async def _errored(repo, variant):
|
|
return "error", "disk exploded"
|
|
|
|
monkeypatch.setattr(auto_dl, "_job_state", _errored)
|
|
refusal = _run("unsloth/x-GGUF")
|
|
assert refusal.status == 502 and "disk exploded" in refusal.message
|
|
# Slot released, so a different model can now start.
|
|
assert _run("unsloth/other-GGUF").code == "model_downloading"
|
|
|
|
|
|
def test_hf_token_is_passed_to_the_worker(hub):
|
|
_run("unsloth/x-GGUF", hf_token = "hf_secret")
|
|
assert hub["started"][0][2] == "hf_secret"
|
|
|
|
|
|
# --- the single-flight slot ---------------------------------------------------
|
|
|
|
|
|
def test_a_refused_dispatch_is_not_reported_as_downloading(hub):
|
|
# The hub service can decline without raising (accepted=False), so the caller hears "busy".
|
|
hub["dispatch_result"] = {
|
|
"job_key": "unsloth/x-gguf::ud-q5_k_xl",
|
|
"state": "running", # the blocking job's state, not ours
|
|
"accepted": False,
|
|
"generation": 3,
|
|
}
|
|
refusal = _run("unsloth/x-GGUF:UD-Q5_K_XL")
|
|
assert refusal.status == 503 and refusal.code == "model_download_busy"
|
|
# No watcher installed for a job that is not running.
|
|
assert hub["watched"] == []
|
|
# The slot is free, so an unrelated repo is still admitted.
|
|
assert auto_dl._active is None
|
|
hub["dispatch_result"] = {"job_key": "k", "state": "running", "accepted": True}
|
|
assert _run("unsloth/other-GGUF").code == "model_downloading"
|
|
|
|
|
|
def test_an_adoptable_dispatch_still_tracks_the_existing_job(hub):
|
|
# accepted=True with claimed=False means it is already downloading (Hub UI); attach to it.
|
|
hub["dispatch_result"] = {"job_key": "k", "state": "running", "accepted": True}
|
|
assert _run("unsloth/x-GGUF:UD-Q5_K_XL").code == "model_downloading"
|
|
assert len(hub["watched"]) == 1
|
|
|
|
|
|
def test_a_failed_status_probe_does_not_end_the_watch(hub, monkeypatch):
|
|
# A probe that raised says nothing: reading it as "idle" freed the slot mid-download.
|
|
from hub.services.models import downloads
|
|
|
|
async def _boom(repo_id, gguf_variant = ""):
|
|
raise RuntimeError("registry unavailable")
|
|
|
|
monkeypatch.setattr(downloads, "get_download_status_response", _boom)
|
|
state, error = asyncio.run(auto_dl._job_state("unsloth/x-GGUF", "UD-Q4_K_XL"))
|
|
assert (state, error) == ("unknown", None)
|
|
|
|
|
|
def test_an_unknown_state_still_reports_the_download_to_a_retry(hub, monkeypatch):
|
|
assert _run("unsloth/x-GGUF:UD-Q4_K_XL").code == "model_downloading"
|
|
|
|
async def _unknown(repo, variant):
|
|
return "unknown", None
|
|
|
|
monkeypatch.setattr(auto_dl, "_job_state", _unknown)
|
|
# Still downloading as far as anyone knows, so the slot stays taken.
|
|
assert _run("unsloth/x-GGUF:UD-Q4_K_XL").code == "model_downloading"
|
|
assert _run("unsloth/other-GGUF").code == "model_download_busy"
|
|
|
|
|
|
def test_a_hanging_code_probe_does_not_pin_the_slot(hub, monkeypatch):
|
|
# hf_hub_download and auth_check take no timeout, and both run while the provisional
|
|
# slot is held, so an unresponsive Hub stalled the request far past the metadata
|
|
# budget and reported every other model busy meanwhile. Unchecked is not cleared,
|
|
# so the bounded probe refuses rather than admitting the repo.
|
|
import threading
|
|
|
|
entered, release = threading.Event(), threading.Event()
|
|
|
|
def _hang(repo, token = None):
|
|
entered.set()
|
|
release.wait(30)
|
|
return False
|
|
|
|
monkeypatch.setattr("utils.security.consent._config_has_auto_map", _hang)
|
|
monkeypatch.setattr(auto_dl, "_CODE_PROBE_TIMEOUT_S", 0.2)
|
|
|
|
async def _timed():
|
|
# Time the await, not asyncio.run: the probe thread cannot be cancelled, so
|
|
# loop shutdown waits for it here in a way a long-lived server loop never does.
|
|
started = time.monotonic()
|
|
refusal = await auto_dl.maybe_auto_download("unsloth/x-GGUF:UD-Q4_K_XL")
|
|
waited = time.monotonic() - started
|
|
release.set()
|
|
return refusal, waited
|
|
|
|
refusal, waited = asyncio.run(_timed())
|
|
assert entered.is_set()
|
|
assert refusal.status == 403 and refusal.code == "remote_code_consent_required"
|
|
assert waited < 5
|
|
# The slot was handed back, so the next request is admitted rather than told busy.
|
|
assert auto_dl._active is None
|
|
|
|
|
|
def test_a_hanging_auth_check_falls_through_to_the_download(hub, monkeypatch):
|
|
# Inconclusive, not denied: the download's own auth is the real gate, so a slow
|
|
# gated-repo check must not turn into a refusal.
|
|
import threading
|
|
|
|
hub["info"].gated = True
|
|
release = threading.Event()
|
|
|
|
def _hang(repo, token = None):
|
|
release.wait(30)
|
|
return True
|
|
|
|
monkeypatch.setattr(auto_dl, "_auth_denied", _hang)
|
|
monkeypatch.setattr(auto_dl, "_MODEL_INFO_TIMEOUT_S", 0.2)
|
|
|
|
async def _timed():
|
|
refusal = await auto_dl.maybe_auto_download("unsloth/x-GGUF:UD-Q4_K_XL")
|
|
release.set()
|
|
return refusal
|
|
|
|
assert asyncio.run(_timed()).code == "model_downloading"
|
|
|
|
|
|
def test_a_companion_only_repo_is_not_held_at_busy(hub):
|
|
# mmproj and MTP files are companions, not quants, so admission classifies such a
|
|
# repo as non-servable and lets the label fall through to the resident model. The
|
|
# busy probe accepted any .gguf, which stranded that ordinary traffic behind an
|
|
# unrelated multi-hour download.
|
|
assert _run("unsloth/x-GGUF:UD-Q4_K_XL").code == "model_downloading"
|
|
gb = 1024**3
|
|
hub["info"] = _Info([_Sibling("mmproj-F16.gguf", gb), _Sibling("mtp-model.gguf", gb)])
|
|
assert _run("unsloth/companions-GGUF") is None
|
|
# A repo that does hold a real quant is still a second download.
|
|
hub["info"] = _gguf_repo_info()
|
|
assert _run("unsloth/other-GGUF").code == "model_download_busy"
|
|
|
|
|
|
def test_a_stale_watcher_cannot_release_a_newer_download(hub, monkeypatch):
|
|
# Variant A is downloading; its watcher holds the slot.
|
|
assert _run("unsloth/x-GGUF:UD-Q4_K_XL").code == "model_downloading"
|
|
watcher_a = hub["watched"][-1]
|
|
|
|
# A fails, so an adopting request surfaces the error and frees the slot.
|
|
real_job_state = auto_dl._job_state
|
|
errored = {"on": True}
|
|
|
|
async def _maybe_errored(repo, variant):
|
|
if errored["on"]:
|
|
return "error", "boom"
|
|
return await real_job_state(repo, variant)
|
|
|
|
monkeypatch.setattr(auto_dl, "_job_state", _maybe_errored)
|
|
assert _run("unsloth/x-GGUF:UD-Q4_K_XL").code == "model_download_failed"
|
|
errored["on"] = False
|
|
|
|
# The retry starts variant B of the same repo, which now owns the slot.
|
|
assert _run("unsloth/x-GGUF:UD-Q5_K_XL").code == "model_downloading"
|
|
watcher_b = hub["watched"][-1]
|
|
assert auto_dl._active is watcher_b
|
|
|
|
# Only now does A's watcher clean up. Keyed on repo_id alone, that cleared B.
|
|
errored["on"] = True
|
|
monkeypatch.setattr(auto_dl, "_WATCH_POLL_S", 0.0)
|
|
asyncio.run(hub["real_watch"](watcher_a, None))
|
|
assert auto_dl._active is watcher_b
|
|
assert _run("unsloth/other-GGUF").code == "model_download_busy"
|
|
|
|
|
|
def test_a_cancelled_admission_does_not_wedge_the_slot(hub):
|
|
# CancelledError is a BaseException, so an `except Exception` cleanup would wedge the slot.
|
|
def _cancel():
|
|
raise asyncio.CancelledError()
|
|
|
|
hub["on_probe"] = _cancel
|
|
|
|
async def _cancelled_request():
|
|
with pytest.raises(asyncio.CancelledError):
|
|
await auto_dl.maybe_auto_download("unsloth/x-GGUF")
|
|
|
|
asyncio.run(_cancelled_request())
|
|
assert auto_dl._active is None
|
|
hub["on_probe"] = None
|
|
assert _run("unsloth/other-GGUF").code == "model_downloading"
|
|
|
|
|
|
# --- route wiring ------------------------------------------------------------
|
|
|
|
|
|
class _Url:
|
|
def __init__(self, path):
|
|
self.path = path
|
|
|
|
|
|
class _Req:
|
|
def __init__(
|
|
self,
|
|
path = "/v1/chat/completions",
|
|
headers = None,
|
|
):
|
|
self.url = _Url(path)
|
|
self.headers = headers or {}
|
|
|
|
|
|
def _hook(model, request, enabled):
|
|
import utils.openai_auto_switch_settings as s
|
|
|
|
original = s.get_openai_auto_download_enabled
|
|
s.get_openai_auto_download_enabled = lambda: enabled
|
|
try:
|
|
return asyncio.run(inference_route._maybe_auto_download_model(model, request))
|
|
finally:
|
|
s.get_openai_auto_download_enabled = original
|
|
|
|
|
|
def test_setting_off_does_nothing_at_all(hub):
|
|
# The compatibility invariant: no probe, no dispatch, no raise.
|
|
assert _hook("unsloth/x-GGUF:UD-Q5_K_XL", _Req(), enabled = False) is None
|
|
assert hub["started"] == []
|
|
|
|
|
|
def test_hook_raises_the_openai_envelope_with_retry_after(hub):
|
|
from fastapi import HTTPException
|
|
|
|
with pytest.raises(HTTPException) as excinfo:
|
|
_hook("unsloth/x-GGUF:UD-Q5_K_XL", _Req(), enabled = True)
|
|
exc = excinfo.value
|
|
assert exc.status_code == 503
|
|
assert exc.headers and exc.headers["Retry-After"]
|
|
assert exc.detail["error"]["code"] == "model_downloading"
|
|
assert exc.detail["error"]["param"] == "model"
|
|
assert exc.detail["error"]["type"] == "api_error"
|
|
|
|
|
|
def test_hook_uses_the_anthropic_envelope_on_messages(hub):
|
|
from fastapi import HTTPException
|
|
|
|
with pytest.raises(HTTPException) as excinfo:
|
|
_hook("unsloth/x-GGUF", _Req(path = "/v1/messages"), enabled = True)
|
|
detail = excinfo.value.detail
|
|
assert detail["type"] == "error"
|
|
assert detail["error"]["type"] == "api_error"
|
|
|
|
|
|
def test_hook_swallows_unexpected_failures(hub, monkeypatch):
|
|
# A broken download path must not turn a servable request into a 500.
|
|
async def _boom(model, hf_token = None):
|
|
raise RuntimeError("boom")
|
|
|
|
monkeypatch.setattr(auto_dl, "maybe_auto_download", _boom)
|
|
assert _hook("unsloth/x-GGUF", _Req(), enabled = True) is None
|
|
|
|
|
|
def test_hook_prefers_the_hub_header_token(hub):
|
|
from fastapi import HTTPException
|
|
from hub.dependencies import HUB_HF_TOKEN_HEADER
|
|
|
|
with pytest.raises(HTTPException):
|
|
_hook(
|
|
"unsloth/x-GGUF",
|
|
_Req(headers = {HUB_HF_TOKEN_HEADER: "hf_from_header"}),
|
|
enabled = True,
|
|
)
|
|
assert hub["started"][0][2] == "hf_from_header"
|
|
|
|
|
|
# --- never answer as a different model ----------------------------------------
|
|
|
|
|
|
class _CatalogInfo:
|
|
"""Minimal stand-in for a local model the /v1/models scan listed."""
|
|
|
|
def __init__(self, model_id, path):
|
|
self.model_id = model_id
|
|
self.id = model_id
|
|
self.path = path
|
|
|
|
|
|
class _Loaded:
|
|
"""Minimal stand-in for the GGUF backend with one model resident."""
|
|
|
|
def __init__(
|
|
self,
|
|
identifier,
|
|
variant = None,
|
|
advertised = None,
|
|
):
|
|
self.is_loaded = True
|
|
self.model_identifier = identifier
|
|
self.hf_variant = variant
|
|
self._openai_advertised_id = advertised
|
|
|
|
|
|
def _reject(
|
|
model,
|
|
loaded,
|
|
monkeypatch,
|
|
*,
|
|
downloaded = False,
|
|
auto_switch = False,
|
|
):
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: loaded)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": None})(),
|
|
)
|
|
monkeypatch.setattr(
|
|
"core.inference.local_model_resolver.resolve_local_gguf",
|
|
lambda name, **_kw: ("/p", None, name) if downloaded else None,
|
|
)
|
|
monkeypatch.setattr(
|
|
"utils.openai_auto_switch_settings.get_openai_auto_switch_enabled",
|
|
lambda: auto_switch,
|
|
)
|
|
monkeypatch.setattr(inference_route, "_unavailable_model_message", _fake_unavailable_message)
|
|
return asyncio.run(inference_route._reject_unservable_model(model, _Req()))
|
|
|
|
|
|
async def _fake_unavailable_message(model):
|
|
return f"The model '{model}' is not downloaded on this server."
|
|
|
|
|
|
def test_wrong_quant_is_not_answered_by_the_loaded_one(monkeypatch):
|
|
# The reported bug: asking for UD-Q6_K_XL while UD-Q4_K_XL is resident returned 200.
|
|
loaded = _Loaded("unsloth/gemma-4-E2B-it-GGUF", "UD-Q4_K_XL")
|
|
with pytest.raises(HTTPException) as excinfo:
|
|
_reject("unsloth/gemma-4-E2B-it-GGUF:UD-Q6_K_XL", loaded, monkeypatch)
|
|
assert excinfo.value.status_code == 404
|
|
|
|
|
|
def test_bare_repo_id_is_satisfied_by_any_loaded_quant(monkeypatch):
|
|
# No quant named means "this model", so the resident quant answers it.
|
|
loaded = _Loaded("unsloth/gemma-4-E2B-it-GGUF", "UD-Q4_K_XL")
|
|
assert _reject("unsloth/gemma-4-E2B-it-GGUF", loaded, monkeypatch) is None
|
|
|
|
|
|
def test_matching_quant_is_served(monkeypatch):
|
|
loaded = _Loaded("unsloth/gemma-4-E2B-it-GGUF", "UD-Q4_K_XL")
|
|
assert _reject("unsloth/gemma-4-E2B-it-GGUF:ud-q4_k_xl", loaded, monkeypatch) is None
|
|
|
|
|
|
def test_advertised_alias_counts_as_serving(monkeypatch):
|
|
# Loaded by path, requested by the repo id auto-switch advertised for it.
|
|
loaded = _Loaded("/cache/snap/abc", "UD-Q4_K_XL", "unsloth/gemma-4-E2B-it-GGUF")
|
|
assert _reject("unsloth/gemma-4-E2B-it-GGUF", loaded, monkeypatch) is None
|
|
|
|
|
|
@pytest.mark.parametrize("foreign", ["gpt-4", "gpt-4o-mini", "claude-3-5-sonnet", "default"])
|
|
def test_foreign_ids_still_fall_through(monkeypatch, foreign):
|
|
# Drop-in compatibility: an id with no namespace is a label, not a reference.
|
|
loaded = _Loaded("unsloth/gemma-4-E2B-it-GGUF", "UD-Q4_K_XL")
|
|
assert _reject(foreign, loaded, monkeypatch) is None
|
|
|
|
|
|
def test_downloaded_but_auto_switch_off_says_so(monkeypatch):
|
|
loaded = _Loaded("unsloth/A-GGUF", "UD-Q4_K_XL")
|
|
with pytest.raises(HTTPException) as excinfo:
|
|
_reject("unsloth/B-GGUF", loaded, monkeypatch, downloaded = True)
|
|
assert "Switch model by request" in str(excinfo.value.detail)
|
|
|
|
|
|
def test_a_failed_switch_is_reported_not_answered_by_the_resident_model(monkeypatch):
|
|
# On disk and switching allowed means the swap failed; the resident model is wrong weights.
|
|
loaded = _Loaded("unsloth/A-GGUF", "UD-Q4_K_XL")
|
|
with pytest.raises(HTTPException) as excinfo:
|
|
_reject("unsloth/B-GGUF", loaded, monkeypatch, downloaded = True, auto_switch = True)
|
|
assert excinfo.value.status_code == 503
|
|
assert excinfo.value.detail["error"]["code"] == "model_switch_failed"
|
|
assert excinfo.value.headers["Retry-After"] == "5"
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"foreign",
|
|
[
|
|
"anthropic/claude-3.5-sonnet",
|
|
"openai/gpt-4o",
|
|
"meta-llama/llama-3-70b-instruct",
|
|
"mistralai/Mistral-7B-Instruct-v0.2",
|
|
],
|
|
)
|
|
def test_a_provider_prefixed_label_still_reaches_the_resident_model(foreign, monkeypatch):
|
|
# A namespace is how LiteLLM addresses providers, so reading it as a reference 404s them.
|
|
loaded = _Loaded("unsloth/A-GGUF", "UD-Q4_K_XL")
|
|
assert _reject(foreign, loaded, monkeypatch) is None
|
|
|
|
|
|
def test_an_explicit_quant_is_still_refused(monkeypatch):
|
|
# A quant is the signal: no LiteLLM or OpenRouter id carries one.
|
|
loaded = _Loaded("unsloth/A-GGUF", "UD-Q4_K_XL")
|
|
with pytest.raises(HTTPException) as excinfo:
|
|
_reject("unsloth/B-GGUF:UD-Q6_K_XL", loaded, monkeypatch)
|
|
assert excinfo.value.status_code == 404
|
|
|
|
|
|
def test_a_repo_that_is_here_is_refused_without_a_quant(monkeypatch):
|
|
# The other half of the evidence test: a repo this server has is a reference to it.
|
|
loaded = _Loaded("unsloth/A-GGUF", "UD-Q4_K_XL")
|
|
with pytest.raises(HTTPException) as excinfo:
|
|
_reject("unsloth/B-GGUF", loaded, monkeypatch, downloaded = True)
|
|
assert excinfo.value.status_code == 404
|
|
|
|
|
|
def test_a_diagnosis_failure_does_not_serve_the_wrong_model(monkeypatch):
|
|
# The mismatch is already established, so falling through would answer as another model.
|
|
loaded = _Loaded("unsloth/A-GGUF", "UD-Q4_K_XL")
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: loaded)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": None})(),
|
|
)
|
|
|
|
def _boom(name, **_kw):
|
|
raise OSError("cache scan unavailable")
|
|
|
|
monkeypatch.setattr("core.inference.local_model_resolver.resolve_local_gguf", _boom)
|
|
with pytest.raises(HTTPException) as excinfo:
|
|
asyncio.run(inference_route._reject_unservable_model("unsloth/B-GGUF:UD-Q6_K_XL", _Req()))
|
|
assert excinfo.value.status_code == 404
|
|
|
|
|
|
def test_nothing_loaded_leaves_the_existing_error_alone(monkeypatch):
|
|
# The handler's own no-model-loaded error is already correct; don't preempt it.
|
|
idle = type("B", (), {"is_loaded": False, "model_identifier": None, "hf_variant": None})()
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: idle)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": None})(),
|
|
)
|
|
assert asyncio.run(inference_route._reject_unservable_model("unsloth/B-GGUF", _Req())) is None
|
|
|
|
|
|
def test_reload_only_sentinel_is_ignored(monkeypatch):
|
|
loaded = _Loaded("unsloth/A-GGUF", "UD-Q4_K_XL")
|
|
assert _reject(inference_route._RELOAD_ONLY_MODEL, loaded, monkeypatch) is None
|
|
|
|
|
|
def test_diagnosis_failure_never_breaks_a_servable_request(monkeypatch):
|
|
loaded = _Loaded("unsloth/A-GGUF", "UD-Q4_K_XL")
|
|
|
|
def _boom(_name, **_kw):
|
|
raise RuntimeError("scan exploded")
|
|
|
|
monkeypatch.setattr("core.inference.local_model_resolver.resolve_local_gguf", _boom)
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: loaded)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": None})(),
|
|
)
|
|
assert asyncio.run(inference_route._reject_unservable_model("unsloth/B-GGUF", _Req())) is None
|
|
|
|
|
|
def test_anthropic_surface_gets_its_own_envelope(monkeypatch):
|
|
loaded = _Loaded("unsloth/A-GGUF", "UD-Q4_K_XL")
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: loaded)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": None})(),
|
|
)
|
|
monkeypatch.setattr(
|
|
"core.inference.local_model_resolver.resolve_local_gguf", lambda name, **_kw: None
|
|
)
|
|
monkeypatch.setattr(inference_route, "_unavailable_model_message", _fake_unavailable_message)
|
|
with pytest.raises(HTTPException) as excinfo:
|
|
asyncio.run(
|
|
inference_route._reject_unservable_model(
|
|
"unsloth/B-GGUF:UD-Q6_K_XL", _Req(path = "/v1/messages")
|
|
)
|
|
)
|
|
assert excinfo.value.detail["type"] == "error"
|
|
|
|
|
|
# --- settings ----------------------------------------------------------------
|
|
|
|
|
|
def test_auto_download_defaults_off_and_is_gated_on_auto_switch(monkeypatch):
|
|
store = {}
|
|
monkeypatch.setattr(settings, "_cached_setting", lambda k, d = None: store.get(k, d))
|
|
assert settings.get_stored_openai_auto_download_enabled() is False
|
|
assert settings.get_openai_auto_download_enabled() is False
|
|
|
|
store[settings.OPENAI_AUTO_DOWNLOAD_SETTING_KEY] = True
|
|
# Stored on, but auto-switch off: nothing would load the result, so it is off.
|
|
assert settings.get_stored_openai_auto_download_enabled() is True
|
|
assert settings.get_openai_auto_download_enabled() is False
|
|
|
|
store[settings.OPENAI_AUTO_SWITCH_SETTING_KEY] = True
|
|
assert settings.get_openai_auto_download_enabled() is True
|
|
|
|
|
|
def test_setter_round_trips_auto_download_in_one_transaction(monkeypatch):
|
|
import storage.studio_db as db
|
|
|
|
calls = []
|
|
store = {}
|
|
|
|
def _upsert(mapping):
|
|
calls.append(dict(mapping))
|
|
store.update(mapping)
|
|
|
|
monkeypatch.setattr(db, "upsert_app_settings", _upsert)
|
|
monkeypatch.setattr(settings, "_cached_setting", lambda k, d = None: store.get(k, d))
|
|
|
|
result = settings.set_openai_auto_switch(True, 120, None, True)
|
|
assert result == (True, 120, True, True)
|
|
assert len(calls) == 1
|
|
assert calls[0][settings.OPENAI_AUTO_DOWNLOAD_SETTING_KEY] is True
|
|
|
|
|
|
def test_setter_rejects_a_non_boolean_auto_download(monkeypatch):
|
|
monkeypatch.setattr(settings, "_cached_setting", lambda k, d = None: None)
|
|
with pytest.raises(ValueError, match = "true or false"):
|
|
settings.set_openai_auto_switch(True, None, None, "garbage")
|
|
|
|
|
|
def test_settings_route_exposes_auto_download(monkeypatch):
|
|
import routes.settings as settings_route
|
|
|
|
monkeypatch.setattr(settings_route, "get_openai_auto_switch_enabled", lambda: True)
|
|
monkeypatch.setattr(settings_route, "get_stored_auto_unload_idle_seconds", lambda: 0)
|
|
monkeypatch.setattr(settings_route, "get_auto_unload_idle_seconds", lambda: 0)
|
|
monkeypatch.setattr(settings_route, "get_auto_unload_keep_kv", lambda: True)
|
|
monkeypatch.setattr(settings_route, "get_stored_openai_auto_download_enabled", lambda: True)
|
|
assert settings_route.get_openai_auto_switch("tester").auto_download_model is True
|
|
|
|
|
|
# --- the placeholder API key -------------------------------------------------
|
|
|
|
|
|
def test_placeholder_api_key_gets_a_specific_message():
|
|
from auth.authentication import API_KEY_PLACEHOLDER, _invalid_api_key_detail
|
|
|
|
detail = _invalid_api_key_detail(API_KEY_PLACEHOLDER)
|
|
assert "placeholder" in detail
|
|
assert "Settings > API" in detail
|
|
|
|
|
|
def test_every_other_bad_key_stays_indistinguishable():
|
|
from auth.authentication import _invalid_api_key_detail
|
|
|
|
generic = "Invalid or expired API key"
|
|
assert _invalid_api_key_detail("sk-unsloth-revoked") == generic
|
|
assert _invalid_api_key_detail("sk-unsloth-YOUR_KEY ") == generic
|
|
assert _invalid_api_key_detail("sk-unsloth-your_key") == generic
|
|
|
|
|
|
def test_the_servers_own_hf_token_is_never_borrowed(monkeypatch):
|
|
# The repo is named by an API key holder, so the owner's Hub identity must not be used.
|
|
import routes.settings as settings_route
|
|
|
|
monkeypatch.setattr(settings_route, "_ambient_hf_token", lambda: "hf_owner_secret")
|
|
assert inference_route._auto_download_hf_token(_Req()) is None
|
|
caller = _Req(headers = {"X-Unsloth-HF-Token": "hf_caller_own"})
|
|
assert inference_route._auto_download_hf_token(caller) == "hf_caller_own"
|
|
|
|
|
|
def test_a_quant_cannot_be_satisfied_by_a_non_gguf_backend(monkeypatch):
|
|
# llama.cpp matches :QUANT against hf_variant; Transformers has no quant identity.
|
|
idle = type("B", (), {"is_loaded": False, "model_identifier": None, "hf_variant": None})()
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: idle)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": "org/model"})(),
|
|
)
|
|
assert inference_route._loaded_satisfies("org/model") is True
|
|
assert inference_route._loaded_satisfies("org/model:Q4_K_M") is False
|
|
# An Ollama-style tag is not a claim about the weights, so it still matches.
|
|
assert inference_route._loaded_satisfies("org/model:latest") is True
|
|
|
|
|
|
def test_the_worker_is_never_given_the_servers_own_token(hub):
|
|
# A falsy token would make the worker fall back to the server owner's HF_TOKEN.
|
|
assert _run("unsloth/x-GGUF").code == "model_downloading"
|
|
assert hub["started"][0][2] is None
|
|
assert hub["allow_ambient"] is False
|
|
|
|
|
|
def test_the_metadata_probe_is_explicitly_anonymous(hub):
|
|
# token=None means "use the cached login" to huggingface_hub; only False is anonymous.
|
|
_run("unsloth/x-GGUF")
|
|
assert hub["token"] is False
|
|
auto_dl.reset_for_tests()
|
|
_run("unsloth/y-GGUF", hf_token = "hf_caller_own")
|
|
assert hub["token"] == "hf_caller_own"
|
|
|
|
|
|
def test_an_ollama_tag_still_matches_the_resident_gguf(monkeypatch):
|
|
# looks_like_quant() calls these foreign, so they must not be checked against hf_variant.
|
|
loaded = _Loaded("unsloth/A-GGUF", "UD-Q4_K_XL")
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: loaded)
|
|
assert inference_route._loaded_satisfies("unsloth/A-GGUF:latest") is True
|
|
assert inference_route._loaded_satisfies("unsloth/A-GGUF:8b") is True
|
|
assert inference_route._loaded_satisfies("unsloth/A-GGUF:UD-Q4_K_XL") is True
|
|
assert inference_route._loaded_satisfies("unsloth/A-GGUF:Q8_0") is False
|
|
|
|
|
|
def test_a_probing_adoption_never_releases_the_slot(hub, monkeypatch):
|
|
# The whole-repo job key can hold a stale error that would free the probe's slot.
|
|
hub["on_probe"] = lambda: _run_nested()
|
|
seen = {}
|
|
|
|
def _run_nested():
|
|
async def _stale(repo, variant):
|
|
seen["queried"] = True
|
|
return "error", "an older failure"
|
|
|
|
monkeypatch.setattr(auto_dl, "_job_state", _stale)
|
|
seen["refusal"] = _run("unsloth/x-GGUF")
|
|
|
|
assert _run("unsloth/x-GGUF").code == "model_downloading"
|
|
assert seen["refusal"].code == "model_downloading"
|
|
assert "queried" not in seen # the stale job key was never consulted
|
|
|
|
|
|
def test_a_bpw_qualified_quant_is_a_quant_request():
|
|
# _extract_quant_label emits these for repos shipping several files at one base quant.
|
|
assert auto_dl.looks_like_quant("IQ4_XS-3.53bpw")
|
|
assert auto_dl.looks_like_quant("UD-Q4_K_XL-4.19BPW")
|
|
assert not auto_dl.looks_like_quant("3.53bpw")
|
|
|
|
|
|
def test_the_default_pick_survives_lowercase_quant_labels():
|
|
# Preference tokens match case-sensitively, so a lower-case repo would take F16.
|
|
lowered = {"f16": 20, "ud-q4_k_xl": 4, "q8_0": 9}
|
|
assert auto_dl._match_variant(None, lowered) == "ud-q4_k_xl"
|
|
assert auto_dl._match_variant(None, {"F16": 20, "UD-Q4_K_XL": 4}) == "UD-Q4_K_XL"
|
|
|
|
|
|
def test_a_slashless_local_model_is_still_a_concrete_reference(monkeypatch):
|
|
# /v1/models advertises these without a namespace, so a namespace decides nothing.
|
|
loaded = _Loaded("unsloth/A-GGUF", "UD-Q4_K_XL")
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: loaded)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": None})(),
|
|
)
|
|
monkeypatch.setattr(inference_route, "_unavailable_model_message", _fake_unavailable_message)
|
|
monkeypatch.setattr(
|
|
"utils.openai_auto_switch_settings.get_openai_auto_switch_enabled", lambda: False
|
|
)
|
|
|
|
monkeypatch.setattr(
|
|
"core.inference.local_model_resolver.resolve_local_gguf",
|
|
lambda name, **_kw: ("/p", None, name) if name.startswith("standalone-Q4_K_M") else None,
|
|
)
|
|
with pytest.raises(HTTPException) as excinfo:
|
|
asyncio.run(inference_route._reject_unservable_model("standalone-Q4_K_M", _Req()))
|
|
assert excinfo.value.status_code == 404
|
|
|
|
# A slashless name that is not here stays a foreign label.
|
|
monkeypatch.setattr(
|
|
"core.inference.local_model_resolver.resolve_local_gguf", lambda name, **_kw: None
|
|
)
|
|
assert asyncio.run(inference_route._reject_unservable_model("gpt-4", _Req())) is None
|
|
assert asyncio.run(inference_route._reject_unservable_model("default", _Req())) is None
|
|
|
|
|
|
def test_a_cancelled_download_is_not_reported_as_failed(hub, monkeypatch):
|
|
# fail_open rendered a deliberate cancel as "Model download failed".
|
|
from core.inference import api_monitor as monitor_module
|
|
|
|
assert _run("unsloth/x-GGUF:UD-Q4_K_XL").code == "model_downloading"
|
|
active = hub["watched"][-1]
|
|
|
|
async def _cancelled(repo, variant):
|
|
return "cancelled", None
|
|
|
|
monkeypatch.setattr(auto_dl, "_job_state", _cancelled)
|
|
monkeypatch.setattr(auto_dl, "_WATCH_POLL_S", 0)
|
|
asyncio.run(hub["real_watch"](active, None))
|
|
[row] = [e for e in monitor_module.api_monitor.snapshot() if e["id"] == active.monitor_id]
|
|
assert row["status"] == "cancelled"
|
|
assert row.get("error") is None
|
|
|
|
|
|
def test_disk_admission_counts_only_what_is_left_to_fetch(hub, monkeypatch):
|
|
# Charging again for bytes already on disk 507s a download that fits.
|
|
seen = {}
|
|
|
|
def _enough(need):
|
|
seen["need"] = need
|
|
return True, 10 * 1024**4
|
|
|
|
gb = 1024**3
|
|
hub["info"] = _Info(
|
|
[
|
|
_Sibling("model-UD-Q4_K_XL.gguf", 4 * gb, blob_id = "sha-main"),
|
|
_Sibling("mmproj-F16.gguf", 1 * gb, blob_id = "sha-mmproj"),
|
|
_Sibling("mtp-model.gguf", 1 * gb, blob_id = "sha-mtp"),
|
|
]
|
|
)
|
|
monkeypatch.setattr(auto_dl, "_enough_disk", _enough)
|
|
monkeypatch.setattr(
|
|
"hub.utils.download_registry.existing_blob_bytes",
|
|
lambda repo_type, repo_id, hashes: 3 * gb,
|
|
)
|
|
assert _run("unsloth/x-GGUF:UD-Q4_K_XL").code == "model_downloading"
|
|
# 4 GB quant + 2 GB companions, 3 GB of which is already cached.
|
|
assert seen["need"] == 3 * gb
|
|
|
|
|
|
def test_a_resolver_alias_for_the_resident_model_is_not_refused(monkeypatch):
|
|
# A manual load stores the on-disk path /v1/models aliases as publisher/model.
|
|
loaded = _Loaded("/models/publisher/model/weights.gguf", None)
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: loaded)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": None})(),
|
|
)
|
|
monkeypatch.setattr(
|
|
"core.inference.local_model_resolver.resolve_local_gguf",
|
|
lambda name, **_kw: ("/models/publisher/model/weights.gguf", None, "publisher/model"),
|
|
)
|
|
monkeypatch.setattr(
|
|
"utils.openai_auto_switch_settings.get_openai_auto_switch_enabled", lambda: False
|
|
)
|
|
assert asyncio.run(inference_route._reject_unservable_model("publisher/model", _Req())) is None
|
|
|
|
|
|
def test_the_request_path_never_triggers_a_model_index_rescan(monkeypatch):
|
|
# The scan takes seconds under a lock, so this hook must answer from the last built index.
|
|
from core.inference import local_model_resolver as resolver
|
|
|
|
scans = []
|
|
warmed = []
|
|
monkeypatch.setattr(resolver, "_build_index", lambda: scans.append(1) or {})
|
|
monkeypatch.setattr(resolver, "_scan", (1.0, {}))
|
|
# Stub the warm: it is allowed to scan, just not on the thread serving the request.
|
|
monkeypatch.setattr(resolver, "warm_index_soon", lambda: warmed.append(1))
|
|
loaded = _Loaded("unsloth/A-GGUF", "UD-Q4_K_XL")
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: loaded)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": None})(),
|
|
)
|
|
monkeypatch.setattr(inference_route, "_unavailable_model_message", _fake_unavailable_message)
|
|
monkeypatch.setattr(
|
|
"utils.openai_auto_switch_settings.get_openai_auto_switch_enabled", lambda: False
|
|
)
|
|
for model in ("gpt-4", "anthropic/claude-3.5-sonnet", "unsloth/B-GGUF:UD-Q6_K_XL"):
|
|
try:
|
|
asyncio.run(inference_route._reject_unservable_model(model, _Req()))
|
|
except HTTPException:
|
|
pass
|
|
assert scans == []
|
|
assert warmed == [1, 1, 1]
|
|
|
|
|
|
def test_a_cold_index_is_scanned_rather_than_read_as_nothing_here(monkeypatch):
|
|
# Before the first scan there is no cached evidence, and treating that as "not
|
|
# downloaded" answers a named local model with the resident one. The scan is paid
|
|
# once, off the loop; every later request reads the built index instead.
|
|
from core.inference import local_model_resolver as resolver
|
|
|
|
entry = resolver._LocalGgufEntry("org/other", "/srv/models/org--other", ("Q4_K_M",))
|
|
scans = []
|
|
|
|
def _build():
|
|
scans.append(1)
|
|
return {"org/other": entry}
|
|
|
|
monkeypatch.setattr(resolver, "_scan", (0.0, {}))
|
|
monkeypatch.setattr(resolver, "_build_index", _build)
|
|
loaded = _Loaded("unsloth/A-GGUF", "UD-Q4_K_XL")
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: loaded)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": None})(),
|
|
)
|
|
monkeypatch.setattr(inference_route, "_unavailable_model_message", _fake_unavailable_message)
|
|
monkeypatch.setattr(
|
|
"utils.openai_auto_switch_settings.get_openai_auto_switch_enabled", lambda: False
|
|
)
|
|
|
|
# The bug: a bare name that IS on disk used to fall through to the resident model.
|
|
with pytest.raises(HTTPException) as excinfo:
|
|
asyncio.run(inference_route._reject_unservable_model("org/other", _Req()))
|
|
assert excinfo.value.status_code == 404
|
|
assert scans == [1], "the cold index was not scanned"
|
|
|
|
# Built now, so the request path reads the cache and never scans again.
|
|
assert asyncio.run(inference_route._reject_unservable_model("gpt-4", _Req())) is None
|
|
assert scans == [1]
|
|
|
|
|
|
def test_a_cold_scan_that_never_finishes_says_so_instead_of_guessing(monkeypatch):
|
|
# The scan is bounded so a pathological install cannot hold the request open, but
|
|
# an unfinished scan knows nothing about the name, and falling through would put
|
|
# the resident model behind it. Answer "not yet", with a Retry-After.
|
|
import threading
|
|
|
|
from core.inference import local_model_resolver as resolver
|
|
|
|
monkeypatch.setattr(resolver, "_scan", (0.0, {}))
|
|
monkeypatch.setattr(inference_route, "_COLD_INDEX_WAIT_S", 0.05)
|
|
released = threading.Event()
|
|
monkeypatch.setattr(resolver, "_build_index", lambda: (released.wait(5), {})[1])
|
|
warmed = []
|
|
monkeypatch.setattr(resolver, "warm_index_soon", lambda: warmed.append(1))
|
|
loaded = _Loaded("unsloth/A-GGUF", "UD-Q4_K_XL")
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: loaded)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": None})(),
|
|
)
|
|
try:
|
|
with pytest.raises(HTTPException) as excinfo:
|
|
asyncio.run(inference_route._reject_unservable_model("gpt-4", _Req()))
|
|
assert excinfo.value.status_code == 503
|
|
assert excinfo.value.headers.get("Retry-After")
|
|
assert warmed == [1], "the scan was not left to finish in the background"
|
|
finally:
|
|
released.set()
|
|
|
|
|
|
def test_a_refusal_is_never_swallowed_by_the_cannot_verify_handler(monkeypatch):
|
|
# The checks run inside a broad `except Exception` that turns a failure to decide
|
|
# into a fallthrough. An HTTPException raised in there is a decision, and was
|
|
# being logged as a failure and answered by the resident model instead.
|
|
loaded = _Loaded("unsloth/A-GGUF", "UD-Q4_K_XL")
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: loaded)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": None})(),
|
|
)
|
|
|
|
def _boom(*_a, **_k):
|
|
raise HTTPException(status_code = 418, detail = "decided")
|
|
|
|
monkeypatch.setattr(inference_route, "_resolves_to_resident", _boom)
|
|
monkeypatch.setattr(
|
|
"core.inference.local_model_resolver.resolve_local_gguf",
|
|
lambda *_a, **_k: ("/srv/models/x", "Q4_K_M", "x"),
|
|
)
|
|
with pytest.raises(HTTPException) as excinfo:
|
|
asyncio.run(inference_route._reject_unservable_model("org/x", _Req()))
|
|
assert excinfo.value.status_code == 418
|
|
|
|
|
|
def test_warming_the_index_never_waits_on_the_scan_lock(monkeypatch):
|
|
# _lock is held for the whole scan, so contending for it would park every later request.
|
|
import threading
|
|
import time as _time
|
|
|
|
from core.inference import local_model_resolver as resolver
|
|
|
|
monkeypatch.setattr(resolver, "_scan", (0.0, {}))
|
|
released = threading.Event()
|
|
monkeypatch.setattr(resolver, "_build_index", lambda: (released.wait(5), {})[1])
|
|
_real_warm_index_soon()
|
|
try:
|
|
started = _time.perf_counter()
|
|
_real_warm_index_soon()
|
|
resolver.resolve_local_gguf("unsloth/A-GGUF", allow_scan = False)
|
|
elapsed = _time.perf_counter() - started
|
|
finally:
|
|
released.set()
|
|
# Join before the monkeypatches unwind, or the scan publishes its stub result over them.
|
|
for _ in range(500):
|
|
if not resolver._warming:
|
|
break
|
|
_time.sleep(0.01)
|
|
assert elapsed < 0.5, f"request path blocked on the warm scan for {elapsed:.2f}s"
|
|
|
|
|
|
def test_a_stale_index_is_refreshed_so_a_hub_download_becomes_visible(monkeypatch):
|
|
# Only the auto-download watcher calls invalidate_index, so a Hub UI download is seen
|
|
# only if the warm can run again.
|
|
from core.inference import local_model_resolver as resolver
|
|
|
|
scans = []
|
|
monkeypatch.setattr(resolver, "_build_index", lambda: scans.append(1) or {})
|
|
monkeypatch.setattr(resolver, "_scan", (time.monotonic() - resolver._CACHE_TTL_S - 1, {}))
|
|
monkeypatch.setattr(resolver, "_last_scan_s", 0.0)
|
|
_real_warm_index_soon()
|
|
for _ in range(500):
|
|
if scans and not resolver._warming:
|
|
break
|
|
time.sleep(0.01)
|
|
assert scans == [1]
|
|
|
|
|
|
def test_an_id_v1_models_advertised_is_refused_before_the_resolver_warms(monkeypatch):
|
|
# /v1/models can advertise an unloaded local GGUF while the resolver index is cold. A bare
|
|
# id has no quant to refuse on, so without that evidence the resident model would answer.
|
|
from core.inference import local_model_resolver as resolver
|
|
|
|
monkeypatch.setattr(resolver, "_scan", (0.0, {}))
|
|
# Stub the walk: a real multi-root scan inside the cold-wait budget makes this
|
|
# test time out into a 503 under load instead of asserting what it is here for.
|
|
monkeypatch.setattr(resolver, "_build_index", lambda: {})
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"_CATALOG_CACHE",
|
|
{"at": 1.0, "models": [_CatalogInfo("org/Other", "/srv/models/org--Other")]},
|
|
)
|
|
monkeypatch.setattr(inference_route, "_ADVERTISED_CACHE", {"at": None, "paths": {}})
|
|
loaded = _Loaded("unsloth/A-GGUF", "UD-Q4_K_XL")
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: loaded)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": None})(),
|
|
)
|
|
monkeypatch.setattr(inference_route, "_unavailable_model_message", _fake_unavailable_message)
|
|
with pytest.raises(HTTPException) as excinfo:
|
|
asyncio.run(inference_route._reject_unservable_model("org/Other", _Req()))
|
|
assert excinfo.value.status_code == 404
|
|
# An id the catalog never listed still proves nothing, so it falls through.
|
|
assert asyncio.run(inference_route._reject_unservable_model("org/Unlisted", _Req())) is None
|
|
|
|
|
|
def test_an_advertised_alias_for_the_resident_weights_is_still_served(monkeypatch):
|
|
# The flip side: the catalog can list the resident weights under an alias, which is not
|
|
# evidence of a different model.
|
|
from core.inference import local_model_resolver as resolver
|
|
|
|
monkeypatch.setattr(resolver, "_scan", (0.0, {}))
|
|
# Stub the walk: a real multi-root scan inside the cold-wait budget makes this
|
|
# test time out into a 503 under load instead of asserting what it is here for.
|
|
monkeypatch.setattr(resolver, "_build_index", lambda: {})
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"_CATALOG_CACHE",
|
|
{"at": 2.0, "models": [_CatalogInfo("publisher/Qwen3", "/srv/models")]},
|
|
)
|
|
monkeypatch.setattr(inference_route, "_ADVERTISED_CACHE", {"at": None, "paths": {}})
|
|
loaded = _Loaded("/srv/models/Qwen3-Q4.gguf", "Q4_K_M")
|
|
loaded.gguf_path = "/srv/models/Qwen3-Q4.gguf"
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: loaded)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": None})(),
|
|
)
|
|
assert asyncio.run(inference_route._reject_unservable_model("publisher/Qwen3", _Req())) is None
|
|
|
|
|
|
def test_a_rejected_token_says_so_instead_of_asking_for_a_retry(hub):
|
|
# Hugging Face answers an expired or invalid X-Unsloth-HF-Token with 401. Only
|
|
# 403 and 404 were handled, so it fell through to "could not reach Hugging Face"
|
|
# with a 503, telling the caller to retry something that cannot start working.
|
|
from huggingface_hub.utils import HfHubHTTPError
|
|
|
|
hub["raise"] = _hub_error(HfHubHTTPError, 401, "unauthorized")
|
|
refusal = _run("unsloth/x-GGUF:UD-Q5_K_XL", hf_token = "hf_expired")
|
|
assert refusal.status == 401 and refusal.code == "model_access_denied"
|
|
assert "token" in refusal.message.lower()
|
|
assert hub["started"] == []
|
|
|
|
|
|
def test_an_image_request_does_not_download_a_text_only_model(hub):
|
|
# The capability guard only ever sees an already-local target, so without this
|
|
# an image request would spend gigabytes on weights that cannot answer it and
|
|
# then 400 on every retry.
|
|
gb = 1024**3
|
|
hub["info"] = _Info([_Sibling("model-UD-Q5_K_XL.gguf", 5 * gb)])
|
|
refusal = asyncio.run(
|
|
auto_dl.maybe_auto_download("unsloth/text-GGUF:UD-Q5_K_XL", require_vision = True)
|
|
)
|
|
assert refusal.status == 400 and refusal.code == "invalid_value"
|
|
assert "mmproj" in refusal.message
|
|
assert hub["started"] == []
|
|
# The stock fixture repo ships mmproj-F16.gguf, so that one is allowed to start.
|
|
hub["info"] = _gguf_repo_info()
|
|
assert (
|
|
asyncio.run(
|
|
auto_dl.maybe_auto_download("unsloth/x-GGUF:UD-Q5_K_XL", require_vision = True)
|
|
).code
|
|
== "model_downloading"
|
|
)
|
|
assert len(hub["started"]) == 1
|
|
|
|
|
|
def test_two_models_differing_only_in_case_are_not_the_same_weights(monkeypatch):
|
|
# Lowercasing paths made /srv/models/Foo and /srv/models/foo compare equal, so
|
|
# on a case-sensitive filesystem a request for one was answered by the other.
|
|
import os
|
|
|
|
loaded = _Loaded("/srv/models/Foo/model.gguf")
|
|
loaded.gguf_path = "/srv/models/Foo/model.gguf"
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: loaded)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": None})(),
|
|
)
|
|
assert inference_route._resolves_to_resident("/srv/models/Foo") is True
|
|
same = os.path.normcase("A") == os.path.normcase("a")
|
|
assert inference_route._resolves_to_resident("/srv/models/foo") is same
|
|
|
|
|
|
def test_a_quant_request_is_not_satisfied_by_transformers_weights(monkeypatch):
|
|
# A Transformers model active from a directory that also holds GGUF exports
|
|
# resolves to that same directory, and the path match let admission answer an
|
|
# explicit quant with the safetensors weights. Only llama.cpp has a quant
|
|
# identity, which is why _loaded_satisfies already refuses this by name.
|
|
from core.inference import local_model_resolver as resolver
|
|
|
|
entry = resolver._LocalGgufEntry("alias", "/srv/models/tuned", ("Q4_K_M",))
|
|
monkeypatch.setattr(resolver, "_scan", (time.monotonic(), {"alias": entry}))
|
|
monkeypatch.setattr(
|
|
inference_route, "get_llama_cpp_backend", lambda: type("L", (), {"is_loaded": False})()
|
|
)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": "/srv/models/tuned"})(),
|
|
)
|
|
monkeypatch.setattr(inference_route, "_unavailable_model_message", _fake_unavailable_message)
|
|
with pytest.raises(HTTPException) as excinfo:
|
|
asyncio.run(inference_route._reject_unservable_model("alias:Q4_K_M", _Req()))
|
|
assert excinfo.value.status_code == 404
|
|
# A bare name claims nothing about the weights, so the active model still answers.
|
|
assert asyncio.run(inference_route._reject_unservable_model("alias", _Req())) is None
|
|
|
|
|
|
def test_a_timed_out_download_keeps_the_slot_while_it_is_still_running(monkeypatch):
|
|
# The watch window only bounds progress reporting. Releasing on the clock while
|
|
# the worker is alive would admit a second multi-GB download beside it.
|
|
monkeypatch.setattr(auto_dl, "_MAX_WATCH_S", 0.0)
|
|
monkeypatch.setattr(auto_dl, "_WATCH_POLL_S", 0.001)
|
|
monkeypatch.setattr(auto_dl, "_TIMED_OUT_POLL_S", 0.001)
|
|
active = auto_dl._Active(repo_id = "org/big-GGUF", variant = "Q4_K_M")
|
|
|
|
async def _drive():
|
|
finished = asyncio.Event()
|
|
|
|
async def _state(repo, variant):
|
|
return ("complete" if finished.is_set() else "running"), None
|
|
|
|
monkeypatch.setattr(auto_dl, "_job_state", _state)
|
|
auto_dl._active = active
|
|
watcher = asyncio.create_task(auto_dl._watch(active, None))
|
|
# Long past the deadline, and still running: the slot must not come back.
|
|
await asyncio.sleep(0.05)
|
|
held = auto_dl._active is active
|
|
finished.set()
|
|
await watcher
|
|
return held, auto_dl._active
|
|
|
|
held, after = asyncio.run(_drive())
|
|
assert held, "the slot was released while the worker was still running"
|
|
assert after is None, "the slot was not released once the job finished"
|
|
|
|
|
|
def test_a_timed_out_download_stops_holding_the_slot_once_unprobeable(monkeypatch):
|
|
# The other direction: a probe that can no longer confirm the worker is alive
|
|
# must not wedge auto-download for the life of the process.
|
|
monkeypatch.setattr(auto_dl, "_MAX_WATCH_S", 0.0)
|
|
monkeypatch.setattr(auto_dl, "_WATCH_POLL_S", 0.001)
|
|
monkeypatch.setattr(auto_dl, "_TIMED_OUT_POLL_S", 0.001)
|
|
active = auto_dl._Active(repo_id = "org/big-GGUF", variant = "Q4_K_M")
|
|
|
|
async def _unknown(repo, variant):
|
|
return "unknown", None
|
|
|
|
monkeypatch.setattr(auto_dl, "_job_state", _unknown)
|
|
|
|
async def _drive():
|
|
auto_dl._active = active
|
|
await auto_dl._watch(active, None)
|
|
return auto_dl._active
|
|
|
|
assert asyncio.run(_drive()) is None
|
|
|
|
|
|
def test_a_sibling_quant_in_the_same_directory_is_not_the_resident_one(monkeypatch):
|
|
# Quants of one repo share a directory, so the path match alone cannot tell
|
|
# them apart, and an explicit :Q8_0 was answered by a resident Q4_K_M that
|
|
# _loaded_satisfies had already refused by name.
|
|
from core.inference import local_model_resolver as resolver
|
|
|
|
entry = resolver._LocalGgufEntry("org/model", "/hf/org--model/snap", ("Q4_K_M", "Q8_0"))
|
|
monkeypatch.setattr(resolver, "_scan", (time.monotonic(), {"org/model": entry}))
|
|
loaded = _Loaded("org/model", "Q4_K_M")
|
|
loaded.gguf_path = "/hf/org--model/snap/model-Q4_K_M.gguf"
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: loaded)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": None})(),
|
|
)
|
|
monkeypatch.setattr(inference_route, "_unavailable_model_message", _fake_unavailable_message)
|
|
with pytest.raises(HTTPException):
|
|
asyncio.run(inference_route._reject_unservable_model("org/model:Q8_0", _Req()))
|
|
# The quant that is actually resident still answers.
|
|
assert asyncio.run(inference_route._reject_unservable_model("org/model:Q4_K_M", _Req())) is None
|
|
|
|
|
|
def test_a_remote_tag_that_names_no_quant_picks_the_preferred_one(hub):
|
|
# ":latest" and ":8b" name no quant, so remote admission must default-select
|
|
# like a bare repo id instead of 404ing on a quant that never existed. Matches
|
|
# what the local resolver now does with the same tag.
|
|
assert _run("unsloth/x-GGUF").code == "model_downloading"
|
|
bare_repo, bare_variant, _ = hub["started"][0]
|
|
for tag in (":latest", ":8b"):
|
|
auto_dl.reset_for_tests()
|
|
hub["started"].clear()
|
|
assert _run(f"unsloth/x-GGUF{tag}").code == "model_downloading"
|
|
assert hub["started"][0][0] == bare_repo
|
|
assert hub["started"][0][1] == bare_variant, f"{tag} did not default-select"
|
|
|
|
# A real quant the repo does not have is still a 404, never a substitution.
|
|
auto_dl.reset_for_tests()
|
|
hub["started"].clear()
|
|
refusal = _run("unsloth/x-GGUF:Q2_K")
|
|
assert refusal.status == 404 and "no quant" in refusal.message
|
|
assert hub["started"] == []
|
|
|
|
|
|
def test_a_generic_gguf_advertises_the_label_the_worker_resolves(hub):
|
|
# With no recognized quant token the label extractors part ways: one takes the
|
|
# last hyphenated segment, the plan and the worker key the whole stem. Dispatching
|
|
# ours made the worker exit with "No GGUF shards matching variant".
|
|
from hub.utils.gguf import extract_quant_label as canonical
|
|
from hub.utils.gguf_plan import build_gguf_variant_plans
|
|
|
|
sibling = _Sibling("llama-7b.gguf", 4 * 1024**3)
|
|
hub["info"] = _Info([sibling])
|
|
assert _run("unsloth/generic-GGUF").code == "model_downloading"
|
|
dispatched = hub["started"][0][1]
|
|
assert dispatched == canonical("llama-7b.gguf")
|
|
# The key the worker will look up has to contain it, which is the whole point.
|
|
assert dispatched.lower() in build_gguf_variant_plans([sibling])
|
|
|
|
|
|
def test_windows_style_paths_still_match_their_own_directory(monkeypatch):
|
|
# normcase folds case and rewrites the separator to a backslash on Windows, so
|
|
# normalizing to "/" before it left the descendant checks comparing a "/" against
|
|
# a path that had none, and a resident model read as a different one.
|
|
import ntpath
|
|
|
|
monkeypatch.setattr(inference_route.os.path, "normcase", ntpath.normcase)
|
|
# A manual load records the file, so only the descendant check can match the
|
|
# directory the resolver returns; an equality match would prove nothing here.
|
|
loaded = _Loaded("C:\\models\\repo\\model.gguf")
|
|
loaded.gguf_path = "C:\\models\\repo\\model.gguf"
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: loaded)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": None})(),
|
|
)
|
|
assert inference_route._resolves_to_resident("C:\\models\\repo") is True
|
|
assert inference_route._resolves_to_resident("C:\\Models\\Repo") is True
|
|
assert inference_route._resolves_to_resident("C:\\models\\other") is False
|
|
|
|
|
|
def test_a_bare_request_for_a_just_downloaded_model_is_refused(monkeypatch):
|
|
# End of the same chain: the note has to reach admission, or a bare request in
|
|
# the window between the download landing and the scan is served by the resident
|
|
# model, which is the whole failure this hook exists to stop.
|
|
from core.inference import local_model_resolver as resolver
|
|
|
|
monkeypatch.setattr(resolver, "_scan", (time.monotonic(), {}))
|
|
monkeypatch.setattr(resolver, "_just_downloaded", {"org/fresh"})
|
|
loaded = _Loaded("unsloth/A-GGUF", "UD-Q4_K_XL")
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: loaded)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": None})(),
|
|
)
|
|
monkeypatch.setattr(inference_route, "_unavailable_model_message", _fake_unavailable_message)
|
|
with pytest.raises(HTTPException) as excinfo:
|
|
asyncio.run(inference_route._reject_unservable_model("org/fresh", _Req()))
|
|
assert excinfo.value.status_code == 404
|
|
assert asyncio.run(inference_route._reject_unservable_model("org/never", _Req())) is None
|
|
|
|
|
|
def test_a_non_quant_tag_does_not_tear_down_a_serving_quant(monkeypatch):
|
|
# _already_serving split on ":" rather than on whether the suffix names a quant,
|
|
# so org/model:latest against a serving Q8_0 counted as a quant mismatch and
|
|
# swapped in the preferred Q4_K_M, for a request either one satisfies.
|
|
from core.inference import local_model_resolver as resolver
|
|
|
|
entry = resolver._LocalGgufEntry("org/model", "/hf/org--model/snap", ("Q4_K_M", "Q8_0"))
|
|
monkeypatch.setattr(resolver, "_scan", (time.monotonic(), {"org/model": entry}))
|
|
loaded = _Loaded("org/model", "Q8_0")
|
|
loaded.gguf_path = "/hf/org--model/snap/model-Q8_0.gguf"
|
|
monkeypatch.setattr(inference_route, "get_llama_cpp_backend", lambda: loaded)
|
|
monkeypatch.setattr(
|
|
inference_route,
|
|
"get_inference_backend",
|
|
lambda: type("B", (), {"active_model_name": None})(),
|
|
)
|
|
loads: list = []
|
|
|
|
async def _record_load(request, *a, **k):
|
|
loads.append(getattr(request, "gguf_variant", None))
|
|
|
|
monkeypatch.setattr(inference_route, "_load_model_impl", _record_load)
|
|
monkeypatch.setattr(
|
|
"utils.openai_auto_switch_settings.get_openai_auto_switch_enabled", lambda: True
|
|
)
|
|
for tag in ("org/model:latest", "org/model:8b", "org/model"):
|
|
asyncio.run(inference_route._maybe_auto_switch_model(tag, _Req(), "tester"))
|
|
assert loads == [], "a tag naming no quant swapped the serving model out"
|
|
|
|
|
|
def test_the_trust_probe_never_falls_back_to_the_server_identity(hub, monkeypatch):
|
|
# huggingface_hub treats None as "use the cached login", so only an explicit
|
|
# False is anonymous. The metadata probe and the worker already pass one; this
|
|
# probe did not, so a caller-named repo was read with the server's identity.
|
|
seen: list = []
|
|
|
|
def _probe(model_name, hf_token = None):
|
|
seen.append(hf_token)
|
|
return False
|
|
|
|
monkeypatch.setattr("utils.security.consent._config_has_auto_map", _probe)
|
|
_run("unsloth/x-GGUF:UD-Q5_K_XL")
|
|
assert seen == [False], f"trust probe ran with {seen!r}, not an explicit anonymous token"
|
|
|
|
seen.clear()
|
|
auto_dl.reset_for_tests()
|
|
_run("unsloth/x-GGUF:UD-Q5_K_XL", hf_token = "hf_caller")
|
|
assert seen == ["hf_caller"], "the caller's own token must still be used"
|
|
|
|
|
|
def test_a_foreign_label_is_not_told_to_wait_for_someone_elses_download(hub):
|
|
# The busy refusal fired before the probe, so any namespaced label a drop-in
|
|
# client sends (LiteLLM/OpenRouter style) was told to wait out a download that
|
|
# has nothing to do with it, for as long as that download runs.
|
|
assert _run("unsloth/first-GGUF").code == "model_downloading"
|
|
|
|
hub["info"] = _Info([_Sibling("README.md", 1024)]) # real repo, no GGUF
|
|
assert _run("anthropic/claude-3.5-sonnet") is None, "a foreign label was refused as busy"
|
|
|
|
# A label that really is another downloadable model still gets the busy refusal.
|
|
hub["info"] = _gguf_repo_info()
|
|
refusal = _run("unsloth/second-GGUF")
|
|
assert refusal.status == 503 and refusal.code == "model_download_busy"
|
|
|
|
|
|
def test_a_failed_download_keeps_the_slot_until_someone_is_told(monkeypatch):
|
|
# The watcher freed the slot the moment it saw the error, but Retry-After is 30s
|
|
# and the poll is 2s, so the client came back to an empty slot and started the
|
|
# identical failing download again instead of being told it had failed.
|
|
monkeypatch.setattr(auto_dl, "_MAX_WATCH_S", 60.0)
|
|
monkeypatch.setattr(auto_dl, "_WATCH_POLL_S", 0.001)
|
|
|
|
async def _errored(repo, variant):
|
|
return "error", "disk exploded"
|
|
|
|
monkeypatch.setattr(auto_dl, "_job_state", _errored)
|
|
active = auto_dl._Active(repo_id = "org/x-GGUF", variant = "Q4_K_M")
|
|
auto_dl._active = active
|
|
asyncio.run(auto_dl._watch(active, None))
|
|
assert auto_dl._active is active, "the slot was freed before anyone was told"
|
|
assert active.error == "disk exploded"
|
|
|
|
|
|
def test_the_retry_after_a_failure_is_told_instead_of_restarting_it(hub, monkeypatch):
|
|
# End of the same chain: the held failure has to reach the caller.
|
|
active = auto_dl._Active(
|
|
repo_id = "unsloth/x-GGUF",
|
|
variant = "UD-Q5_K_XL",
|
|
error = "disk exploded",
|
|
failed_at = 1.0,
|
|
)
|
|
auto_dl._active = active
|
|
|
|
async def _idle(repo, variant):
|
|
return "idle", None
|
|
|
|
monkeypatch.setattr(auto_dl, "_job_state", _idle)
|
|
refusal = _run("unsloth/x-GGUF:UD-Q5_K_XL")
|
|
assert refusal.status == 502 and "disk exploded" in refusal.message
|
|
assert hub["started"] == [], "the retry restarted the failing download"
|
|
# Told once, so the slot is free again for a fresh attempt.
|
|
assert auto_dl._active is None
|
|
|
|
|
|
def test_a_completed_download_does_not_restage_the_scan_it_just_warmed(monkeypatch):
|
|
# finalize_worker_exit invalidates and warms. A second invalidation here marks
|
|
# that fresh scan stale and pushes a synchronous rescan onto the client's retry.
|
|
import inspect
|
|
|
|
src = inspect.getsource(auto_dl._watch)
|
|
complete_branch = src[src.index('if state == "complete"') :]
|
|
assert "invalidate_index" not in complete_branch
|
|
|
|
|
|
def test_an_exact_generic_variant_beats_the_default_pick(hub):
|
|
# Canonicalizing generic labels made them real worker keys, but the matcher still
|
|
# read anything non-quant-shaped as a tag, so repo:llama-13b default-selected and
|
|
# fetched llama-7b instead of the model that was actually asked for.
|
|
gb = 1024**3
|
|
hub["info"] = _Info([_Sibling("llama-7b.gguf", 4 * gb), _Sibling("llama-13b.gguf", 8 * gb)])
|
|
assert _run("unsloth/generic-GGUF:llama-13b").code == "model_downloading"
|
|
assert hub["started"][0][1] == "llama-13b"
|
|
|
|
# A quant-shaped suffix that matches nothing is still a miss, never a swap.
|
|
auto_dl.reset_for_tests()
|
|
hub["started"].clear()
|
|
hub["info"] = _gguf_repo_info()
|
|
assert _run("unsloth/x-GGUF:Q2_K").status == 404
|
|
assert hub["started"] == []
|