16 commits

Author	SHA1	Message	Date
Daniel Han	06829c2627	Studio: tighten the comments added by the OpenAI model-admission work (#7501) ... Comment-only follow-up to #7454. That change carried 523 comment lines, many of them three and four line preambles where one line says the same thing. This collapses them and drops the ones restating what the code already says, for a net 77 lines. Scope is limited to comments #7454 itself introduced. The files it touched hold about 3,761 comments in total; the rest predate it and are untouched, verified by checking that every removed line is one that commit added. Nothing that records why a non-obvious decision was made was dropped, only compressed. Still stated: the normcase-before-versus-after Windows separator trap, the innermost-indexed-model rule for nested directories, an HTTPException being a decision rather than a failure to decide, that only an explicit False is anonymous to huggingface_hub while None borrows the server owner's login, the fail-closed tri-state custom-code gate, and the regressions each test was written for. Code is provably unchanged: comment_tools.py check reports 17/17 files comments-only. Backend CI command 10337 passed, 0 failed. tsc -b clean.	2026-07-27 05:59:03 -07:00
Daniel Han	da447d47ba	Studio: fix the "No model loaded" error, and optionally auto-download a model named in an API request (#7454) ... * Studio: say which model is missing instead of "No model loaded" A /v1 request naming a model that is not downloaded returned the generic "No model loaded. Call POST /inference/load first.", which cannot fix it. Return 404 model_not_found naming the model and listing what can serve, and make the API usage examples name a model the server actually has. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: page the API monitor, show model load/unload, pin the example quant The monitor rendered all 50 retained entries in one scroller: page it 5 at a time, freezing history while paged back so live traffic cannot reorder it. Add model load/unload rows so the feed shows what the server is doing, and stop the header rendering the loaded model as a raw host path. Advertise each model's GGUF quant on /v1/models so the example pins repo:QUANT, and move the auto-switch section above the monitor with shorter copy. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: optionally download a model named in an OpenAI API request Auto-switch only ever loaded models already on disk, so naming one this server does not have either 404s or, when something else is loaded, gets quietly answered by the resident model. Add openai_api_auto_download_model (off by default, gated on auto-switch). When on, a /v1 request naming a GGUF repo that is not downloaded starts a background fetch and returns 503 with Retry-After and a typed model_downloading code. The resident model keeps serving in the meantime, and the retry after the download completes is served by the new model through the existing auto-switch path. The download reuses the Hub manager's service layer, which already does repo-id validation, casing, claim bookkeeping, disk preflight, resume and cancel. The in-loader download is deliberately not used: it silently falls back to a smaller quant under low disk, which is wrong when the caller named an exact one. Admission is narrow, since a request only needs an API key: - namespace/name only, so gpt-4 and other foreign ids fall through to the resident model exactly as before - GGUF only, decided from the remote file list rather than the repo name - anything declaring auto_map is refused, so trust_remote_code stays a deliberate opt-in in the UI and can never be granted over the API - a single download at a time, plus a free-disk reserve - one model_info call answers existence, gating and the quant list, so a missing repo, a gated repo and a wrong quant each get their own error With the setting off every one of these paths is byte-identical to before. Also: - monitor rows for downloads, with a live percentage - public_model_id resolves an HF cache snapshot to its repo id, so a cache-loaded model is no longer labelled with a commit sha; this drops the duplicate helper added for the monitor and fixes the same leak in the inference status response - the unedited sk-unsloth-YOUR_KEY from the copyable examples now says so instead of "Invalid or expired API key"; every other bad key keeps the generic message * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add an Unload button to the API monitor The monitor names the loaded model but offered no way to free it. Idle auto-unload is the only existing release path, and it needs a TTL and a wait. The button sits next to Refresh, appears only while a model is loaded and is disabled mid-unload. /unload matches on the internal identifier, which this response deliberately omits because it would be a host path, so the click reads it from /api/inference/status the same way the chat runtime does rather than widening the monitor payload. Also stamp the manual unload row with the quant, read before the teardown clears it, so it reads repo:QUANT like the load row it pairs with. * Studio: keep the API monitor Unload button visible when idle It only rendered while a model was loaded, which hid the one manual release path at exactly the moment someone goes looking for it. Render it always, disabled with a "No model is loaded" tooltip when there is nothing to free. * Studio: never answer a named model with a different one Asking for a model this server is not serving returned 200 from whatever was resident. Requesting gemma-4-E2B-it-GGUF:UD-Q6_K_XL while UD-Q4_K_XL was loaded got a confident answer from the wrong quant, with nothing in the response saying so. A name carrying a namespace (org/model, optionally :QUANT) is a concrete reference, so 404 instead, with the reason: - wrong quant -> names the quants that are actually downloaded - not on disk -> lists what is available - on disk but auto-switch off -> says to turn it on Ids without a namespace (gpt-4, claude-3, default) are foreign labels rather than references, so they still fall through to the resident model and drop-in clients are unaffected. A bare org/model is still satisfied by any loaded quant of that repo; only an explicit :QUANT must match. The check runs whatever the auto-switch and auto-download toggles are, since serving the wrong weights is wrong in every configuration. It is skipped when nothing is loaded, where the existing no-model-loaded error already says the right thing, and when the model is on disk with auto-switch on, where a failed swap should still fall back. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: use a simpler prompt in the API usage examples "What is Unsloth Studio?" rather than "Can Unsloth Studio do API calling?". One constant feeds all nine snippet tabs. * Studio: only refuse a model reference meant for this server A namespace alone was treated as a concrete model reference, so a /v1 request naming anthropic/claude-3.5-sonnet, openai/gpt-4o or any other LiteLLM or OpenRouter style vendor/model id started returning 404 instead of being answered by the resident model. Refuse only on evidence the caller meant this server: an explicit GGUF quant label, or a repo that is actually on disk here. gpt-4 and vendor/model alike fall through again, while the wrong-quant and wrong-repo cases this PR exists for still refuse. Also from review: - Release the single download slot by object identity, not repo id. A stale watcher could clear a newer download of the same repo and let a second multi-GB fetch start alongside it. - Catch BaseException around admission: CancelledError is not an Exception, so a cancelled request stranded the slot for the process lifetime. - Honour the download service's accepted=False, which it returns without raising for a cross-variant conflict, instead of promising a download that was never dispatched. - Treat a failed status probe as unknown rather than idle, so a transient read cannot fail the monitor row and free the slot under a live worker. - Check gated repos with auth_check. The Hub serves metadata for a gated repo without granting its files, so the licence gate was being reported as the unrelated custom-code refusal. - Size the disk reserve from the download plan, which includes the mmproj and MTP companions the worker fetches with every quant. - Never fetch under the server's own HF token. The repo is named by whoever holds an API key, so the ambient token let that key pull the owner's private repos. - Refuse an explicit quant on a backend with no quant identity, gated on the suffix really being a quant so Ollama style :latest tags still match. - Raise instead of falling through when the diagnosis fails: the mismatch is already established by then, only the wording is uncertain. - Report a failed switch as 503 model_switch_failed rather than answering as the resident model. - Fail an open monitor row under the same lock as the check, so a finish landing in between cannot stamp an error onto a completed row. - Usage examples never emit a hardcoded model id: the catalog is tri-state and the panel asks for a model to be loaded instead of printing one the server cannot serve. It also refreshes when the loaded model changes. - Keep the monitor pager reachable while frozen entries expire. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: scope the auto-download 404 cache to the caller's credentials The Hub answers 404 for a private repo the caller cannot see, so caching that verdict per repo alone let one anonymous request mark a private repo unservable for everyone for the whole TTL. A later caller sending a valid X-Unsloth-HF-Token skipped the probe and fell through to the resident model instead of downloading what it asked for. Keyed on the repo id plus a digest of the token now, so the token itself is never held. Two more from the same review: - Clear the chat runtime checkpoint after unloading from the API monitor, as the chat eject flow already does. The store went on treating the freed checkpoint as loaded and the usage examples kept naming it. - Point gated and not-found callers at the X-Unsloth-HF-Token header. Automatic download deliberately ignores the server's own Hugging Face identity, so telling the user to add a token in Studio sent them round the same 403 forever. * Studio: tighten the comments added by this branch * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep API auto-download off the server's Hugging Face identity Passing None for the caller's token was not anonymous. spawn_worker substitutes the backend's HF_TOKEN for a falsy one, and HfApi(token=None) falls back to a cached login, so a repo named by an API-key holder could still be fetched under the owner's Hub identity and land in the shared catalog. The metadata probe and auth_check now pass an explicit False, and dispatch threads allow_ambient_token=False so the worker stays anonymous too. The flag defaults to True, so the UI download path keeps the ambient fallback that private repos rely on. Three more from the same review: - Require an exact hf_variant match only when the suffix is really a quant. The llama.cpp branch still compared Ollama style :latest and :8b against the loaded quant and refused the resident model, which is the opposite of what looks_like_quant classifies them as. - Decode an HF cache repo id only when the models-- component is followed by snapshots. An ordinary directory whose name merely starts with models-- was being read as an encoded repo id. - Return the probing response before consulting the job registry when an adopted claim has no variant yet. A stale error on the whole-repo key could otherwise release the slot the first request's probe still holds, letting a second large download start beside it. * Studio: stop treating a namespace as what decides model intent The rule refused a reference only when it carried a namespace, which was wrong in both directions. vendor/model is how LiteLLM and OpenRouter name every provider, and a standalone or custom-folder GGUF is advertised without one, so asking for a path-free local id such as model-Q4_K_M was answered by whatever else happened to be resident. The slashless early return is gone and the same evidence test now applies to every id: an explicit quant, or a model that actually resolves here. gpt-4 and default still fall through because they are not local, not because of their shape. Also: - Recognise bits-per-weight quant labels. _extract_quant_label emits IQ4_XS-3.53bpw and the resolver and downloader both accept it, but _GGUF_KNOWN_QUANT_RE has no bpw group, so looks_like_quant rejected a reference the rest of the machinery understands. - Upper-case the synthetic names handed to _pick_best_gguf. Its preference tokens are upper case and matched case-sensitively, so a repo with lower-case filenames skipped the preference and took the first entry, which can be F16. - Only offer a downloaded but unloaded model as a runnable example when auto-switch is on. It is off by default, so the copied snippet hit the no-model-loaded error, which is the failure this branch exists to fix. The tool-passthrough cancel test stubbed asyncio.to_thread module-wide, so it cancelled at the first thread hop rather than the generation hop it means to test. Model resolution runs off the loop before the monitor row opens, so that stub now passes the resolver through. * Studio: tighten the comments added since the last pass * Studio: match a resident model through its resolver alias A manual load stores the model by its on-disk path while the resolver and /v1/models advertise it as publisher/model, so _loaded_satisfies could not recognise the alias. Reducing the resolution to a boolean then threw away the load path that would have proved the match, and the request was refused with 404 for a model the server was serving at that moment. Common for LM Studio models and custom-folder aliases. The resolved path is compared against the resident backend before anything is refused. Also: - Size disk admission on what is left to fetch. expected_bytes is the whole plan, so a resumed quant or a companion already pulled in by another quant was charged for twice and could 507 a download that fits. Cached blobs are subtracted through existing_blob_bytes, the same accounting the worker's own preflight does, and it falls open to the full size when no blob hashes are available. - Report a cancelled download as cancelled. The catch-all sent every state other than complete or idle through fail_open, so a deliberate cancel rendered as a download failure rather than the monitor's cancelled state. - Keep polling the servable ids while nothing is loaded. The poll settled as soon as auto-switch was on, so turning it back off left the examples naming an unloaded model until something else remounted the panel. * Studio: shorten the comments added in the last pass * Studio: keep the FLA fast-path tests hermetic across transformers versions _discover_fla_model_types scans the *installed* transformers for modeling files importing `from fla.`, so `models/qwen3_5/` only exists from transformers 5.x. The backend supports transformers>=4.51, and on a 4.x install the Qwen3.5 gate returns False, so 14 tests in test_training_worker_flash_attn.py silently exercised a no-op instead of the install path and failed their call-count assertions. Pin the discovered model_type set in those 14 tests, the same way test_hook_does_not_install_tilelang_for_model_outside_allowlist already pins it against newly added FLA model_types. Test-only change: the production gate and the _discover_fla_model_types unit tests are untouched. * Studio: keep the /v1 admission check off the model-scanning path The admission check added here runs on every /v1 request, including with auto-switch off, where the route used to return straight away. It called resolve_local_gguf, whose index is cached for 5s and otherwise rebuilt by walking ./models and every HF cache root, under a lock the next caller waits on. On an install with a large cache that scan measured 6.1s, longer than the TTL that is meant to amortise it, so steady traffic would keep rebuilding it. Answer from the last built index instead and never rebuild from the request path: a stale answer is fine here, since what is on disk barely moves and a finished download already invalidates the index. The first request, before any scan has completed, warms the index on a background thread and skips the check rather than blocking on it. That also makes the lookup a dict read, so it no longer needs handing to a thread. Cold resolve on this box goes from 6152495us to 0.4us, and the whole hook now costs the same for a foreign label as for the resident model. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: fix the admission hook's cold, stale and contended index paths Five review items, four of them on the admission hook added here. Skipping the check until the first scan lands also skipped explicit quant mismatches, so the first request after startup could ask for :Q8_0 while Q4_K_M was resident and be answered by it. The early return was redundant as well: with an empty index resolved is None and here is False, so the gate below already lets a bare name through and refuses an explicit quant, which is what the except branch has always concluded. Dropped it and index_is_built with it. index_is_built took _lock, which _index holds for the whole scan, so once a warm was running every later request blocked on the event loop for exactly as long as the scan it was there to avoid. The warm now has its own lock and reads the timestamp unlocked, which is safe because _scan is only ever rebound. Warming only when the index had never been built left a model fetched in the Hub UI, or dropped into a scan folder, invisible for the life of the process, since only the auto-download watcher calls invalidate_index. Warm on staleness too, and unconditionally, so it refreshes within a TTL without a scan on the request path. Rescanning is capped at a tenth of the scan's own duration: a big install takes longer to scan than the TTL, and warming on the TTL alone would keep a thread scanning continuously. An Ollama-style tag names no quant, so the resolver misses it and auto-download saw a model the resident one already answers to, then 404'd it for having no such quant. Return early when the loaded model satisfies the reference. Frontend: a cancelled download said "Model download failed", because the label collapsed everything non-completed into failure. The backend tests get an autouse fixture that stops the warm from walking the developer's real HF caches; that scan starved the loop under the timing sensitive streaming tests. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make /v1/models and the admission hook agree on what is local Three review items, all on the seam between the catalog scan and the resolver index, which run on separate schedules. /v1/models can advertise a local GGUF the resolver has not indexed yet. A bare id carries no quant to refuse on, so a client asking for one it had just been handed was answered by the resident model instead. The hook now reads the catalog cache as evidence too, never scanning it. It takes the path rather than a yes/no because the converse also happens: the catalog can list the resident weights under an alias the loaded entry does not answer to, and those must stay served. That alias was also emitted twice by /v1/models, once as the loaded basename a manual load records and once as publisher/model marked unloaded, because the dedup only compared ids. Compare the path as well. A directly loaded standalone .gguf takes its quant from the filename, but the resolver stores such files with no quants, so the advertised <stem>:<quant> stopped resolving as soon as anything else loaded. Advertise a quant only when that reference resolves, and downgrade only on a definite answer so a cold index leaves the metadata alone. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: tighten the comments this branch adds Collapse the multi-line notes in the auto-download path, the /v1 admission hook and their tests to one line each, keeping the reason and dropping the restatement. No behaviour change. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: four admission and catalog fixes from review Lowercasing paths in _resolves_to_resident made /srv/models/Foo and /srv/models/foo the same weights on any case-sensitive filesystem, so a request for one could be answered by the other and /v1/models could mark the wrong entry loaded. That helper now backs residency as well as admission, so use os.path.normcase, which folds case only where the filesystem does. Advertising a quant whenever the resolver could not disprove it kept the bug it was meant to fix: a standalone .gguf loaded before the first scan still got <stem>:<quant> published, and the usage examples persist that. No proof is not proof, so omit it and warm the index instead. A 401 from an expired or invalid X-Unsloth-HF-Token skipped the 403 and 404 branches and surfaced as "could not reach Hugging Face, retry shortly". It now says to replace the token, kept apart from the gated refusal since a rejected credential is not an unaccepted licence. An image request naming an undownloaded text-only GGUF started the whole download and only then hit the capability guard, which never sees a remote target, so every retry 400d and the bytes were wasted. Thread require_vision into admission and check it against the mmproj companions the disk preflight already asks build_gguf_variant_plans for. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make the Hub error fixture carry a status on both hub majors The 401 test built HfHubHTTPError directly, which works on 0.x and fails on 1.x where response is required and keyword-only, so all four Python jobs failed while the same test passed locally. _hub_error already handled both constructors, but the 0.x branch left the exception with no response at all, and hf_error_status reads the status off it for the types that do not encode it in their name. So it could only produce a usable error on 1.x, which is why the test bypassed it. Attach the status when the constructed exception lacks it, and use the helper. Cover the helper itself against stand-ins for both constructor shapes, since whichever hub is installed only ever exercises one of them. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: invalidate on every download, resolve bare tags, keep polling Three review items. Only the API auto-download watcher dropped the resolver cache, so a GGUF fetched in the Hub UI stayed absent to the cache-only request path and the request was answered by whatever was resident. finalize_worker_exit is the one point every download worker exits through, so invalidate there. That closes the window without leaning on the TTL, which the scan-duration throttle can stretch past 5s on an install where the scan itself takes longer than that. A downloaded but unloaded GGUF asked for as org/model:latest missed the resolver, since the suffix was always treated as an exact quant. With auto-download on that probed the Hub and returned a 404 for a quant that was never a quant; with it off it refused without switching. Fall back to the base entry when the suffix is not quant-shaped, and keep exact matching for real quants so a swap can never serve the wrong weights under the right name. The usage examples stopped polling once a model was resident, but idle unload frees one without touching the store, so nothing re-ran the effect and the examples kept naming a model that could no longer be reloaded. Slow the poll to 60s instead of stopping it. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: hold the download slot while it is in use, and keep quants to llama.cpp _loaded_satisfies refuses a quant reference against the Transformers backend by name, but the path match did not carry that rule. A Transformers model active from a directory that also holds GGUF exports therefore matched a request for one of those quants and answered it with the safetensors weights. Only llama.cpp has a quant identity, so admission now passes llama_only whenever the reference is quant-qualified. A bare name still matches either backend, and /v1/models residency keeps the default so a loaded Transformers model is still reported loaded. The 24 hour watch window was bounding ownership of the single-flight slot when it should only have been bounding progress reporting, so a legitimately slow download had its slot handed back while the worker was still writing, admitting a second multi-gigabyte download beside it. Resolve the row on the clock, but keep the slot on a slower poll until the job is actually terminal. Past the deadline an unknown state does release it, since it means the worker cannot be probed and holding it on that forever would wedge auto-download. * Studio: keep what the resolver already knew when a download lands Invalidating cleared the index to empty. The request path reads that cache without scanning, so from a completed download until the rebuild landed it had no evidence about any local model, not just the new one, and a bare request for any of them was answered by whatever was resident. Wiring the hook into the shared completion path in the last commit widened that from auto-download to every download. Mark the scan stale and keep the entries instead. Both _index and warm_index_soon rebuild on a zero stamp, while the request path still sees everything it knew a moment ago. Only a completed download invalidates, and that only ever adds models, so nothing retained goes false. Warm from the completion hook too, so the rebuild starts when the download lands rather than when the next request happens to need it. * Studio: match the quant, not just the directory, and default-select bare tags Two quants of one repo share a directory, so the path match could not tell them apart and an explicit :Q8_0 was answered by a resident Q4_K_M that _loaded_satisfies had already refused by name. The llama_only fix in the last commit only ruled out the wrong backend, not the wrong quant on the right one. Both path matches now require the resident hf_variant to equal the requested quant whenever the reference is quantified; a bare name still matches on the path alone, since it claims nothing about the weights. The local resolver already treated a tag that names no quant as meaning the repo, but remote admission still looked for a quant literally called "latest", so the same reference resolved locally and 404d remotely. Branch on looks_like_quant there too. A real quant the repo does not have is still a 404 and never a substitution, which is what separates this from the loader's low-disk fallback. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: one quant preference, and stop trusting a stale checkpoint list_local_gguf_variants sorts by descending size, so the head of variants was the biggest quant, often F16, while remote admission and a plain load both rank through _pick_best_gguf. A bare id therefore meant a different quant depending on which side answered it, and the local answer was the one that could evict a working model and then fail or OOM starting an F16 next to a usable Q4. /v1/models advertised that same head for pinning. Pull the ranking into one preferred_quant helper and have both sides use it. The usage examples returned a stored checkpoint without ever consulting /v1/models, and the polling added last round was gated on not having one, so for a stored checkpoint it never ran. An idle unload then left the panel showing a snippet that could not run. Poll whenever mounted, and prefer the checkpoint only while the catalog still backs it or switching can reload it. A catalog that has not answered yet is not evidence against it. The static contract pinned the old dependency array, so it now asserts the intent it documents: a finished load re-runs the fetch, and the effect is not gated on having no checkpoint. * Studio: fix the Windows path compare, and advertise a label the worker knows The case fix normalized the separator to "/" and then called os.path.normcase, which on Windows folds case and rewrites the separator back to a backslash, so the descendant checks compared against a "/" the path no longer had. A manually loaded GGUF reached through an alias then read as a different model, giving a false 404 and an alias marked unloaded. Run normcase first and normalize the separator after it. There are two quant-label extractors and they only agree while a recognized quant token is present. With none, _extract_quant_label takes the last hyphenated segment, "7b" of llama-7b, while build_gguf_variant_plans and the worker key the whole stem: the plan lookup missed and the job exited on a variant it had no shards for. Use the canonical extractor for the unrecognized case only. Checked across real filenames first, the two match on every recognized quant and part on bpw-qualified labels, which _extract_quant_label keeps apart on purpose so byteshape's IQ4_XS at 3.53, 3.97 and 4.19 bpw stay separate variants. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: a stored checkpoint needs catalog evidence, not just the switch setting Preferring it whenever switching was on short-circuited the catalog check, so a checkpoint the store still held after the model was deleted or moved kept being named even though /v1/models had already proved it absent, and the snippets 404d instead of falling back to a model that is actually there. A lookup rather than a disjunction, which settles the whole matrix in one place: no answer yet keeps the checkpoint, since that is not evidence against it; listed and resident keeps it; listed but unloaded keeps it only when switching can reload it; absent falls back whatever the setting says. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: normalize the quote style pre-commit would have rewritten * Studio: cover the model that just landed, and pin the quant the catalog has Retaining the index on invalidation protects what was already scanned and by construction cannot contain the model that just finished downloading, so a bare request for it in the window before the rebuild was still answered by the resident model. Record the repo at the completion hook and treat that as admission evidence alongside the resolver and the catalog; the next completed scan clears the notes, since the index then covers them. Publishing a rebuilt index before completion becomes observable would have closed it too, but that blocks the download worker for the length of the scan. Catalog membership proves the repo, not the saved quant, and the examples then pinned the stored one. A quant deleted while another quant of the same repo remained produced repo:deleted-quant, a missing-quant 404 with a runnable alternative listed right beside it. Pin what the catalog advertises: for a resident entry that is the resident quant, for an unloaded one it is a quant actually on disk. The store is only consulted before /v1/models has answered. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: apply three rules everywhere they belong, not only where reported The trust probe was the last credential handoff still passing a raw token. huggingface_hub reads None as "use the cached login", so a caller-named repo was read with this server's Hugging Face identity whenever the caller sent none, which is exactly the isolation the metadata probe and the worker already keep. It takes _hub_token now. Enumerated the rest of that path while there: auth_check, model_info and spawn_worker were already correct. finalize_worker_exit is shared with dataset downloads, so the resolver hook fired for every completed dataset, scanning the model directories for nothing and recording the dataset id as local-model evidence, which turns a bare /v1 request naming that id into a refusal instead of a foreign-id fallthrough. Gated on repo_type. _already_serving decided "bare" on the presence of a colon while _loaded_satisfies and the resolver decide it on whether the suffix names a quant, so org/model:latest against a serving Q8_0 read as a mismatch and swapped in the preferred Q4_K_M for a request either one answers. That rule now lives in four places, each fixed in its own round, so this time I looked for the rest and found a fifth: describe_local_miss splits on the bare colon and its docstring claims it splits like the resolver. It no longer did, and would report a missing quant named "latest". Fixed here too, unreported. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: probe before refusing busy, and scan once when the index is cold The busy refusal fired before anything established the requested label was a model at all, so any namespaced id a drop-in client sends was told to wait out an unrelated download for as long as it ran. Probe first and refuse only a label the Hub actually serves as GGUF; anything else falls through to the resident model as before. A probe failure answers "not downloadable", since stranding ordinary traffic costs more than missing a busy refusal. Treating an unbuilt index as "nothing here" let the first request after startup be answered by the resident model under another model's name. That was a deliberate trade to keep the scan off the request path, and it was the wrong one. Cold, the scan now runs once on a thread, bounded so a pathological install falls through rather than hanging the request. Built, the request path still never scans, so the latency fix stands. The watcher freed the slot the moment it saw an error, while Retry-After is thirty times the poll interval, so the client came back to an empty slot and restarted the same failing download instead of being told. Hold the failure on the slot until a retry surfaces it, and let another repo take it after three retry intervals so a client that never returns cannot keep it. The watcher also invalidated on completion, which now lands after finalize_worker_exit's warm and marks that fresh scan stale, pushing a synchronous rescan onto the retry. Removed. _loaded_satisfies lowercased paths as well as aliases, so it returned satisfied before the case-preserving compare below could run. Both now go through one helper: paths compare with normcase, aliases stay case-insensitive. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: an unfinished scan is not absence, and a decided refusal is not a failure Bounding the cold scan then reading the bound as "not here" left the same hole one branch over. A timeout now answers 503 model_indexing with a Retry-After and leaves the warm running. A foreign label sent inside that window is asked to retry rather than falling through, which is a real cost, but the window is one request on an install whose scan exceeds ten seconds and it clears itself, where answering with the wrong weights does not. That uncovered a worse one. Every check here runs inside a broad except whose job is "could not verify, so fall through", so an HTTPException raised in the block was logged as a verification failure and the request was answered by the resident model. Any refusal decided in there was being swallowed. Re-raise it ahead of that handler. Canonicalizing generic labels made them real variant keys, but the matcher still decided on shape, so repo:llama-13b fell past an exact match and fetched llama-7b. Match exactly first, whatever the shape; a quant-shaped suffix that matches nothing is still a miss and never a swap. Marking a catalog alias loaded while publishing the preferred on-disk quant claimed alias:Q4 was loaded while Q8 was serving, and requiring the resident quant to match then made pinning it a 404. Advertise the resident variant when the entry resolves to the resident model. * Studio: keep the asyncio.timeout fallback tests runnable on Python 3.10 Both tests deleted asyncio.timeout to force _wall_clock_timeout down its pre-3.11 branch, but monkeypatch.delattr raises when the attribute is already absent. On Python 3.10, the one version the fallback exists for, there is nothing to delete, so the two tests errored with AttributeError before reaching the code they cover. Passing raising=False makes the deletion a no-op there and leaves the assertions running against the same branch on every version. Every other delattr in the repo already passes raising=False for exactly this reason. Verified with asyncio.timeout removed from the interpreter: the two tests fail with the CI AttributeError before this change and pass after, and the file still runs 89 passed on 3.13 where the deletion is real. * Studio: decide GGUF residency, servability and variant keys by one rule each Four admission and catalog fixes, each closing a gap between two places that were answering the same question differently. The /v1/models catalog asked _resolves_to_resident without llama_only, so a Transformers model live from a directory that also holds GGUF exports marked a GGUF alias loaded and gave it a GGUF quant. The usage examples then pinned alias:quant that nothing could serve with switching off. Every entry in that loop is advertised as GGUF, so residency there is llama.cpp residency. The busy probe accepted any .gguf sibling while admission excludes mmproj, MTP drafters and big-endian builds. A repo holding only companions is not downloadable, so it was held at model_download_busy for the length of an unrelated download instead of falling through to the resident model as it does when no download is running. It now reuses _gguf_variants, the same filter. split_model_ref refused any slash-bearing suffix, but an unrecognized GGUF below a subdirectory keys on its path (build/llama-13b), which is_valid_gguf_variant allows and the catalog advertises. Pinning such a variant could not parse, so only the default-ranked one was reachable. A slash-bearing suffix is now a variant exactly when a real Hub repo precedes it, which still leaves C:/models/x.gguf a path rather than a quant. The usage examples treated a downloaded-but-unloaded model as runnable only under auto-switch, but a standalone UNSLOTH_MODEL_IDLE_TTL reloads exactly what it freed on the next request. The panel hid runnable examples after an idle unload. Tracked apart from auto-switch, because the stash restores the stored checkpoint only and never an arbitrary catalog entry. Also stub the index walk in the three cold-index tests that missed it: a real multi-root scan inside the cold-wait budget made them time out into a 503 under load rather than assert what they are there for. One of them flaked locally. Verified each fix is load-bearing by reverting it and watching its test fail. Backend CI command: 10195 passed, 0 failed. tsc -b and the frontend build clean. * Studio: bound the Hub admission probes and stop guessing at nested model paths Three review fixes plus a test-isolation one. _resolves_to_resident matched on a path prefix, so two separately indexed models that nest (/models/A alongside /models/A/sub/B) both satisfied it: loading B made a request for A resident and answered it with B's weights, and the catalog marked A loaded. A prefix match now counts only when no catalog entry sits deeper, which is the innermost indexed model that actually owns the file. With nothing indexed there is no nesting to tell apart, so the directory-to-weights match this exists for is unchanged. auth_check and hf_hub_download take no timeout of their own, and both ran while the provisional single-flight slot was held, so an unresponsive Hub stalled the request far past the metadata budget and reported every other model busy for the duration. Both are bounded now. Each default errs the safe way: an unchecked repo is not a cleared one, so the custom-code probe refuses on timeout, while a slow gated-repo check stays inconclusive because the download's own auth is the real gate. The usage examples caught a failed refresh into an empty catalog and a disabled auto-switch, which made a transient error authoritative and blanked every example while the model was still servable. The catalog is deliberately tri-state; a failure now keeps the last answer and retries. Also start the backend tests from a built, empty model index. Stubbing only the background warm still left the cold path walking real caches synchronously inside the admission wait, so on a large install a test asserted against a 503 "still indexing" instead of its subject. _build_index is untouched, so the tests that call it directly still exercise the real walk. Verified each fix is load-bearing by reverting it and watching its test fail. tsc -b clean. Backend CI command green apart from two failures reproduced only on this box (a real model-dir scan and an orphan-process cleanup), neither touched by this PR; staging CI is the gate for those. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-07-27 05:02:06 -07:00
Michael Han	6d8c18cd1a	Replace standalone Studio wording with Unsloth (#7221) ... * Replace standalone Studio wording with Unsloth Replace the single word Studio with Unsloth wherever it is used as shorthand for Unsloth Studio in docs, CLI output, UI strings, i18n locales, workflow display names, comments and docstrings. Kept unchanged: the full name Unsloth Studio, third party product names (LM Studio, Visual Studio, Mac Studio), feature names (Recipe Studio, Fine-tuning Studio and its translations), and all identifiers such as env vars, commands, paths and filenames. * Address review feedback on the Studio wording rename Use "an" before Unsloth where the rename left the article as "a". Restore the split brand where Unsloth and Studio render as two halves of the full product name: the onboarding sidebar subtitle and the IPv6 localhost warning. Scope two messages to the full name Unsloth Studio where plain Unsloth was misleading: the AMD README bullet and the CLI studio setup error.	2026-07-19 00:47:04 -07:00
Daniel Han	5211b506e1	Studio: opt-in OpenAI /v1 model auto-switch and idle keep-warm (#6392) ... * Studio: opt-in OpenAI /v1 model auto-switch and idle keep-warm The OpenAI-compatible endpoints serve whichever GGUF is loaded and ignore the request model field, so an OpenAI client that changes model never reloads. Add an opt-in setting that, when a /v1 request names a downloaded local GGUF different from the loaded one, loads it before serving by reusing the existing /load path (its dedup, tensor fallback, and threading apply). Unknown names still serve the loaded model, so drop-in compatibility is preserved and no remote download is triggered. Also add an optional idle auto-unload (TTL keep-warm): a pure-ASGI middleware tracks in-flight inference requests so a stream is never unloaded mid-response, and a lifespan loop unloads the model after the configured idle seconds. Both settings default off and live in the app_settings store, exposed via GET/PUT /api/settings/openai-auto-switch. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: variant-aware auto-switch, /v1/responses coverage, keep-warm load stamp Follow-ups from review of the opt-in OpenAI auto-switch path: 1. Variant-aware dedup. _maybe_auto_switch_model compared only the repo id, so requesting another quant of the loaded repo (e.g. Q4_K_M loaded, Q8_0 asked) was served by the old quant. Compare hf_variant too, matching /load dedup. 2. Streaming /v1/responses now calls the auto-switch hook. It went straight into _responses_stream and only checked is_loaded, so stream=True could serve the old model or 400. Non-streaming already routed through chat completions; the hook is idempotent once loaded. 3. resolve_local_gguf tries an exact id match before splitting a trailing :VARIANT, so local ids that contain a colon (e.g. a Windows path) resolve instead of being cut at the drive letter. 4. Idle keep-warm stamps activity on a load/swap transition. _last_active was only refreshed by inference requests, so a model loaded after the server sat idle past the TTL could be unloaded before its first request. Tests cover each case. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make the /v1/responses auto-switch test order-independent The new streaming-responses test passed in isolation but failed under the CI's randomized collection order with "object has no attribute 'state'": it passed a bare object() as the request and stubbed only one dispatcher, so an ordering where the real dispatcher ran hit request.state. Give the request a state and stub both dispatchers; the test still asserts the hook fires before dispatch. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: assert /v1/responses auto-switch wiring on source, not at runtime The behavioral version executed openai_responses and relied on stubbing its callees, which a randomized collection order in CI could defeat (the real dispatcher ran and hit request attributes). Assert on the function source that the hook precedes both dispatchers instead; the hook's runtime behavior is already covered by the direct _maybe_auto_switch_model tests. * Studio: auto-switch on /v1/embeddings, GGUF-only targets, idle-unload race gate Second-pass review follow-ups on the opt-in auto-switch path: 1. /v1/embeddings now calls the auto-switch hook before the loaded-state check, matching the other model-bearing OpenAI endpoints (the keep-warm middleware already treats embeddings as inference). 2. The resolver index is now GGUF-only. The local-model scanners also surface Transformers/safetensors repos; without a filter, auto-switch could unload the GGUF and route a request into the non-GGUF loader. _has_local_gguf checks a direct .gguf, a models-dir folder, and the HF-cache snapshots layout. 3. Idle keep-warm now holds an asyncio gate across the idle check and the unload, and a request bumps inflight under the same gate, so the loop can no longer unload in the window between "looks idle" and the kill. Tests cover each. Broader local-model source parity (LM Studio, Ollama, legacy caches, custom scan folders) is a follow-up; missing one of those today just falls through to the loaded model. * Studio: variant-aware local resolver, count_tokens + audio auto-switch coverage Third-pass review follow-ups on the opt-in auto-switch path: 1. The resolver is now variant-aware via list_local_gguf_variants. It indexes only the quants actually on disk, recursing snapshots and quant subdirs such as the nested per-quant folders, so a requested repo:VARIANT resolves only when that quant is local and a bare repo resolves to a concrete local quant. This fixes two gaps: the previous shallow glob rejected nested-variant GGUF repos, and a request for an uncached quant could send /load down the remote download path, breaking the local-only contract. 2. /v1/messages/count_tokens now auto-switches like its sibling /v1/messages, so a count uses the requested model's tokenizer. 3. /api/inference/audio/generate (direct GGUF TTS) is now tracked as in-flight inference, so the idle loop cannot unload the model mid-generation. Tests cover each. Two reviewer items are left as follow-ups: indexing the remaining local sources (LM Studio, Ollama, legacy/default caches, custom scan folders), which fails safe today by falling through to the loaded model; and fully serializing concurrent different-model requests, an inherent limit of the single-slot llama backend that the opt-in feature is not designed around. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: make local GGUF resolver fail-safe so a bad model name cannot 500 The auto-switch hook calls resolve_local_gguf without its own guard, and /v1/completions and /v1/embeddings pass body.get("model") through unchanged. A non-string model (e.g. {"model": 123}) or any internal scan failure would then raise out of the resolver and turn a request that would otherwise be served by the loaded model into a 500, breaking the drop-in compatibility the feature is built on. Guard the resolver at its boundary: reject non-string input up front and wrap the lookup so any failure returns None (fall through to the loaded model). Add regression tests for both paths. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: per-model launch flags for auto-switched GGUF models * Studio: list switch-eligible GGUFs in /v1/models when auto-switch is on * Studio: settings UI for OpenAI model auto-switch and idle auto-unload * Studio: show save error over the disabled-idle hint in auto-switch settings * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: address gemini review (case-insensitive /v1/models retrieve, idle-input empty guard) * Studio: address codex review (deterministic override args, exclude probe/embedding models from discovery) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep-warm count_tokens, gate idle on auto-switch, drop hidden models Three hardening fixes to the opt-in auto-switch path surfaced while reviewing the work that builds on it: 1. count_tokens keep-warm. /v1/messages/count_tokens counts via the loaded tokenizer and already auto-switches, but the keep-warm middleware did not track it, so idle auto-unload could free the model mid-count. It is now a tracked in-flight path. 2. "Off means unchanged" for idle unload. get_auto_unload_idle_seconds now reports 0 while auto-switch is disabled. Idle unload only makes sense with auto-switch on (an unloaded model returns only via the next request's swap), so a stray TTL can no longer trigger a destructive unload while the feature is off, keeping the disabled state identical to pre-feature behavior. 3. Hidden models are not switch targets. The resolver index now skips what Studio hides from its own pickers (the llama.cpp validation probe, RAG embedding weights) via _is_hidden_model, so they can never be auto-switched to by name. Tests added for each. * Studio: bare-id reuse, responses validation order, in-flight tracking Review follow-ups after folding in the per-model overrides and discovery work: 1. A bare model id (no :VARIANT) is now satisfied by any loaded quant of that repo. Previously a bare name resolved to the largest local quant, so it could force a slow reload when a different quant of the same repo was already serving. An explicit repo:VARIANT request still honors the quant. 2. /v1/responses now runs the auto-switch hook after the empty-input validation so a request that 400s can no longer trigger a multi-minute model load before being rejected. The hook still precedes both dispatchers, so streaming requests switch. 3. The keep-warm middleware now tracks in-flight requests whenever auto-switch is enabled rather than only when the idle TTL is already positive, so a stream that starts with the TTL at 0 is still protected if idle-unload is enabled mid-stream. Off still passes straight through. Tests added for each. * Studio: tighten auto-switch code comments Comment/docstring-only pass over the OpenAI auto-switch feature: collapse multi-line blocks, drop a comment that restated the gate it sits next to, and trim verbose docstrings on internal helpers while keeping the load-bearing rationale (concurrency, API behavior, drop-in compat, gotchas). No logic change: verified comment-only with the AST/printer signature check. * Studio: bind auto-switch locks per running loop Review follow-up. The auto-switch swap lock and the keep-warm unload gate were module-level asyncio.Lock objects. That is safe under the single uvicorn loop and on Python 3.10+ (the Lock resolves the running loop lazily on acquire), but a module-level Lock binds to one loop on pre-3.10, which can raise a loop mismatch in multi-loop runners. Resolve each lock through a per-loop accessor backed by a WeakKeyDictionary so every running loop gets its own Lock and stale loops are collected. No behavior change under the server's single loop. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: auto-switch re-review fixes (body codes, coverage, swap, alias, tracking) Follow-ups from a second review pass over the opt-in OpenAI auto-switch feature: 1. OFF-state status codes: /v1/completions and /v1/embeddings moved the body read ahead of the loaded-state check, so a malformed/empty body with no model loaded returned 500 instead of the prior 503. A shared helper reads the body defensively (an unparseable/non-dict body yields no model), and the handler re-reads after the 503 gate to surface the original parse error exactly as before. OFF behavior is unchanged. 2. Local-model coverage: the resolver index only scanned ./models and the active HF cache, while the model picker also lists the legacy/default HF caches, LM Studio dirs, and user scan folders. A request for one of those named models silently served the loaded model instead. _build_index now scans the same roots (Ollama's symlink-creating scanner is skipped on the request path), and resolution is offloaded with asyncio.to_thread so the wider scan never blocks the event loop. 3. Swap vs in-flight stream: a cross-model swap killed the llama-server while another client was still streaming from it. The hook now tracks how many requests are streaming on the loaded model (in-flight minus those still inside the hook) and returns 409 instead of swapping while one is active. Concurrent same-model requests never reach this path, so they are unaffected. 4. Idle-unload + alias: after idle-unload freed the model, an unknown/alias name resolved to nothing and 503'd, though it served the active model before the TTL. Idle-unload now remembers the freed id and an alias request reloads it (only an already-local model, so no remote download), cleared once a model is loaded again. 5. In-flight tracking: the keep-warm middleware tracked in-flight only while the feature was on, so a stream started while off could be unloaded if idle-unload was enabled mid-stream. It now tracks on every inference path; counting is cheap and invisible to clients. Tests added for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove stray async_task_outputs files committed by mistake * Studio: auto-switch review round 3 (revert swap guard, hardening) Addressing a third review pass: - Revert the cross-model swap guard. It counted keep-warm in-flight (which includes external-provider calls that never touch the local model) and so could 409 a local swap spuriously, and it still left a same-model request able to start streaming on the model a concurrent swap was unloading. A correct fix needs a request-lifetime reader/writer barrier; a partial guard was worse than the honest single-slot behavior, so concurrent different-model use is back to being serialized (documented), like llama-swap's single slot. - Non-string request model (e.g. {"model": 123} on a raw-body endpoint) is now treated as absent, so it falls through instead of raising in the membership checks once an idle-unload stash exists. - Idle-unload now stashes and replays the freed quant: an alias reload restores the exact (id, variant) that was freed rather than the largest local quant. - Anthropic /v1/messages validates max_tokens before the auto-switch hook, so a request that 400s never triggers a model load. - Keep-warm tracks a pending count for requests waiting on the unload gate, so the idle loop cannot unload the model out from under a request that is blocked on the gate but not yet counted as in-flight. - The idle-unload task is awaited after cancel on shutdown to avoid pending-task warnings. - The resolver's HF cache scan is None-safe and logs at debug instead of letting a bad root abort the whole index build. - upsert_app_setting_map_entry rolls back explicitly on error. Tests updated/added for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: keep saved idle-unload seconds when auto-switch is toggled off * Studio: auto-switch hardening (thread-safe lock maps, body validation) Defensive fixes from review: - Guard the per-loop WeakKeyDictionary get-or-create for both the unload gate and the auto-switch lock with a threading lock, since WeakKeyDictionary mutation is not thread-safe when two event loops run on different threads. - Build the resolver index under the cache lock so concurrent callers with an expired cache don't all run the multi-dir scan at once. - /v1/completions and /v1/embeddings return a clean 400 for a valid JSON body that is not an object (e.g. a list), instead of a 500 from body.get(...). - The keep-warm middleware only tracks POST requests (inference is always POST), so CORS preflight (OPTIONS) is not counted, and tolerates a None path. Tests added for the list-body 400 and the non-POST skip. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: auto-switch review round 4 (local-path load, swap guard, idle fixes) From a 10-reviewer pass: - HF-cache entries now load by a concrete local path, not the bare repo id. The resolver records a load_path (the snapshot dir for a models--* cache repo, the file/dir otherwise) so /load takes the local branch and can never trigger a download to satisfy a partial cache. The advertised loader_id (repo id) is kept as the launch-override key. resolve_local_gguf now returns (load_path, variant, loader_id). - Re-add a single-slot swap guard: a cross-model swap returns 409 model_switch_busy while another inference request is active rather than killing its stream (the caller is excluded from the count), and holds the keep-warm gate across the load so no new inference starts mid-swap. Concurrent same-model requests never reach this path. A residual spurious 409 is possible while a concurrent or external- provider request is active; that is the documented single-slot tradeoff. - Idle keep-warm tracks (model_identifier, hf_variant): reloading the same repo at a different quant counts as a fresh model, so it is not unloaded before one TTL. - Track Studio's own /api/inference/generate/stream so the idle loop can't unload the model mid-stream on that route. - A successful manual /load clears the idle-unload reload stash synchronously, not only on the next idle poll. Also merged origin/main (the branch had fallen behind, which would have reverted unrelated files on merge). Tests added/updated for each. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: auto-switch review round 5 (concurrency, identity, load gate) From a 10-reviewer pass (9 request-changes, 1 approve): - Concurrent same-target requests load once instead of each returning 409. The count-based busy guard could not tell "another request wants the same model" (safe, load once) from "another request is using the loaded model" (refuse). Track in-flight auto-switch requests per (target, variant) and subtract same-target waiters from the busy count; a cross-model swap still 409s while a genuinely different request is active. - Fix the identity confusion introduced when round 4 began loading by concrete local path: the backend identifier became a filesystem path. Record the advertised repo id on the backend after an auto-switch load and use it so (a) a model loaded manually by repo id is recognized as already serving (no spurious reswap/409), (b) /v1/models reports the repo id, never a host path or a duplicate, and (c) the idle-unload stash keeps the override keyed by the repo id, so an alias reload after TTL keeps the user's saved launch flags. - Gate the manual /load route with the keep-warm lifecycle gate so idle auto-unload can't unload a model mid-load. load_model now wraps _load_model_impl in the gate; auto-switch calls _load_model_impl directly since it already holds the gate. - Restore default-off parity on Anthropic /v1/messages: an unloaded backend with auto-switch disabled 503s before the max_tokens 400 check, as it did pre-feature. When the feature is on, request-shape validation still runs before any load. Tests added for each; full backend suite diff vs baseline is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: auto-switch review round 6 (concurrency ordering, leaks, unload gate) From a second 10-reviewer pass (8 request-changes, 2 approve): - Same-target concurrency: register a waiter by the raw requested model before the (slow) resolve, and exclude pending requests from the swap busy count. The middleware counts a concurrent same-model request as in-flight before it resolves and joins the resolved-target waiter map, so the prior fix could still 409 it. The guard now subtracts max(same resolved-target, same raw-request) waiters and ignores pending (a pending request is blocked in the middleware, not generating, so a swap can't interrupt it). - External-provider requests no longer block a local swap. The keep-warm middleware counts every inference-path POST, but external-provider chat returns before the auto-switch hook and never touches the local GGUF. The chat handler now untracks itself before proxying, so its in-flight stream can't trip model_switch_busy on a concurrent local auto-switch. The middleware skips its own end-decrement for an untracked request. - Manual /unload is gated like load and idle-unload: it holds the lifecycle gate and returns 409 rather than tearing down llama-server while an inference request is in flight. - Response model id no longer leaks the load path. /v1/models already advertised the repo id; chat, completions, embeddings, Anthropic messages, and audio response bodies now use the same _llama_public_model_id helper instead of the concrete on-disk model_identifier. - Chat completions validates the non-system-message requirement before the auto-switch hook (as /responses and /messages already do), so an invalid request can't swap the resident model before returning 400. Tests added for each; full backend suite diff vs baseline is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: auto-switch review round 7 (teardown policy, Unsloth-active swap, training) From a third 10-reviewer pass (9 request-changes, 1 approve), all on the same asymmetric-teardown theme. Resolved per the intended policy that only automatic paths defer to an active stream; deliberate user actions stay interrupting: - Revert the manual /unload in-flight guard added last round. A manual /load or /unload is a deliberate action and tears down immediately, as before; only the automatic idle-unload loop and auto-switch defer to an active request. This removes the asymmetry the reviewers flagged (manual /load, the /unload Unsloth branch, and the opposite-backend swaps inside _load_model_impl) by not extending the guard to deliberate paths, rather than spreading it. - Auto-switch now refuses a swap whenever another inference request is in flight, not only when a GGUF is already loaded. _load_model_impl also unloads an active Unsloth/transformers backend before loading a GGUF, so the busy guard must cover that case too; otherwise an Unsloth stream could be killed by an auto-switch. - Refuse API-initiated training while inference is active. When Studio is driven as an inference API (sk-unsloth key auth), POST /api/training/start returns 409 if a request is in flight, since training frees VRAM by unloading the chat model and would kill the stream. The Studio UI (session auth) still starts training and coexists/frees VRAM as before. A mixed UI+API session is not yet special-cased. Adds auth.authentication.authenticated_via_api_key. Tests added/updated for each; full backend suite diff vs baseline is unchanged. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add UNSLOTH_MODEL_IDLE_TTL env override for idle-unload Borrowed from PR 6517: a startup env var that sets the idle-unload TTL without the settings UI. Unlike the stored setting (gated on auto-switch), the env value is a standalone default that enables idle-unload even with auto-switch off, for headless/container deploys. An explicit UI/API value still overrides it and stays gated. The settings GET reflects the env default when nothing is stored. * Studio: auto-switch fixes from review (paths, embeddings input, env idle reload) - /v1/models advertises a client-facing alias instead of a filesystem path: the ./models and LM Studio scanners report the on-disk path as the model id, so the index now prefers model_id/display_name as the advertised/override id and keeps the concrete path internal as load_path, still resolvable by path. - /v1/embeddings validates input before auto-switch: a request with a model but no input now 400s before the hook (like chat/responses/messages), so an invalid embeddings request cannot unload or swap the resident model. - Standalone UNSLOTH_MODEL_IDLE_TTL reloads the freed model: the hook now runs when auto-switch or idle-unload is active, and with auto-switch off it skips the resolver and only restores the idle-unloaded model, so the first idle timeout no longer leaves later /v1 requests with nothing loaded. - Do not resurrect a stale GGUF over an active Unsloth model: the reload-stash path bails when a non-GGUF backend is loaded, so an unknown /v1 name cannot tear down a live Transformers/Unsloth model. - Defensive HF cache scan: each cache root's resolve/dedup is wrapped so a missing or malformed root skips that root rather than aborting the index. - Single-model retrieve checks the id is a string before lowercasing. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: fix automatic-load asymmetry, audio reload, preview, idle timer The standalone UNSLOTH_MODEL_IDLE_TTL reload is a second automatic-load trigger, but several validate-before-switch guards and reload hooks only checked the auto-switch toggle. Add a shared _automatic_model_load_may_run() (auto-switch on, or idle TTL > 0) and route every guard through it. - /v1/completions validates prompt before any automatic load (it was the one model-bearing route with no pre-check). - /v1/chat/completions and /v1/embeddings pre-checks gate on the shared predicate so a standalone idle TTL cannot reload then reject. - /v1/messages no longer 503s before the reload hook can restore an idle-freed model when auto-switch is off. - Raw completions/embeddings with no model field pass a non-empty sentinel so the idle-stash reload runs, restoring the legacy "omit model, use loaded" path. - /api/inference/audio/generate gains the reload hook (after message validation) so an idle-freed audio GGUF is restored. - Public preview opts out of auto-switch via a request-scope flag, so a caller's model field cannot swap away from the pinned checkpoint; preview chat streams are now matched by _is_inference_path so idle-unload cannot kill them. - Keep-warm no longer stamps activity on request start, and external-provider untracking decrements without restamping, so periodic external traffic can no longer keep the local GGUF warm forever. Merges origin/main (the branch had fallen behind, which also brought in the preview route the review flagged). * Studio: surface model auto-switch in the API tab and demo it in examples The OpenAI auto-switch toggle previously lived only in Settings -> General. Add the same toggle to the API tab's usage-examples panel (it shares the settings cache), and make the examples reflect it: when on, the Python examples append a second call naming a different downloaded GGUF (so the model field visibly selects which model serves), and the curl examples gain a one-line note. Reuses the existing settings API client and i18n keys. * Studio: harden OpenAI auto-switch reload-only path and Anthropic tool validation - Omitted-model raw-body requests pass a reload-only sentinel so the idle-stash reload still restores an idle-freed model, but the resolver never matches a downloaded GGUF literally named "default". - Reject malformed Anthropic client tools before _maybe_auto_switch_model so an invalid request can no longer evict the loaded model. * Studio: extend auto-switch reload-only and tool validation to schema endpoints - Schema-backed endpoints (chat completions, responses, count_tokens, messages, audio) defaulted an omitted model to "default" and passed it to the switch hook, so a downloaded GGUF named "default" could be swapped to. Route the hook through a helper that switches only on an explicitly set model, else reload-only. - Propagate the explicit-set status when building the chat request from a Responses request, so the non-streaming chat re-check stays reload-only too. - Validate Responses function tools before the switch hook so a malformed tool returns 400 without evicting the loaded model. * Studio: serialize auto-switch swaps across event loops with a process-wide gate The auto-switch lock is a per-event-loop asyncio.Lock, so two /v1 swaps on different loops in one process could both pass it and race the single model slot (the backend and _load_model_impl are process-wide). Add a process-wide threading gate around the swap, acquired off the loop so a cross-loop wait never blocks it, layered with the existing per-loop lock. Add a cross-loop test that fails without the gate (two slow loads overlap) and passes with it. * Studio: make the auto-switch swap gate wait cancellation-safe _acquire_swap_gate awaited asyncio.to_thread(lock.acquire) when another loop held the process-wide gate. to_thread cancellation doesn't stop the worker thread, so a /v1 request cancelled mid-wait (client disconnect during a cross-loop swap) would have its thread acquire the gate after the fact, while the finally that releases it never runs -- permanently deadlocking later auto-switch swaps. Poll a non-blocking acquire off a short asyncio.sleep instead: it still keeps the wait off the loop and serializes across loops, but a cancel now lands during the sleep, when the gate is not held, so nothing leaks. Add a test that deadlocks the to_thread variant (it times out) and passes with the poll. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: validate modality and tool-confirmation before auto-switch Two more request shapes could load a named GGUF and only then 400, evicting the resident model: - An image request naming a different text-only GGUF. The switch hook now takes require_vision and rejects a swap to a non-vision target before loading it; a GGUF's vision capability is its companion mmproj, knowable without a load, and matches the post-load guard. Only the resolver branch is checked, never the reload-stash restore. - confirm_tool_calls=true with stream=false and local tools. /v1/chat/completions now rejects that shape before the hook, mirroring the local tool path's bypass_permissions exemption and intent signal. The vision probe threads the ambient HF token to keep the capability-probe invariant. Reload-only and idle-reload paths are unaffected. * Studio: extend validate-before-switch and make the lifecycle gate process-wide - /v1/messages/count_tokens now rejects malformed client tools before the switch hook, like /messages (shared _validate_anthropic_client_tools helper), so a count request can't evict the loaded model. - /v1/chat/completions rejects a malformed tool_choice forcing object (a {"type":"function","function":{}} with no name) before the switch hook. - The inference lifecycle gate that blocks new inference during a swap is now process-wide (a poll-acquired threading lock, cancellation-safe), not a per-loop asyncio lock, so a request on another event loop can't start inference while a swap tears the single backend down. - Usage examples no longer hard-code a switch-demo repo most users lack; the model is an explicit placeholder the user replaces. * Studio: extend the auto-switch modality guard to /v1/responses and /v1/messages The pre-load vision check that guards /v1/chat/completions now also runs on /v1/responses and /v1/messages, so an image request naming a text-only GGUF is rejected before the swap and never evicts the resident vision model. Run the vision capability probe off the event loop. Make the /v1/models retrieve loaded fast-path case-insensitive, and never advertise a host path from the resolver. Remove the dead list_switch_eligible_ids helper, superseded by the /v1/models catalog. * Studio: filter /v1/models to GGUF, per-loop catalog lock, reject system-only Responses Address review findings on the auto-switch path: - /v1/models advertises only GGUF models the API can actually switch to; a safetensors/LoRA entry would be selectable but never loadable via llama.cpp. - The /v1/models catalog cache uses a per-loop lock (like the auto-switch path) so a second event loop awaiting it can't hang in a multi-loop process. - /v1/responses rejects system/developer-only input before the switch, mirroring chat, so an invalid request can't evict the resident model. - _build_index guards each scan source on its own so one bad root drops only that source; the vision probe logs a real detection failure instead of swallowing it. * Studio: list cached GGUFs in /v1/models by inspecting files, not model_format The HF-cache scanner leaves model_format unset for GGUF snapshots, so the previous model_format == "gguf" filter dropped every downloaded HF-cache GGUF from /v1/models and the retrieve fallback. Decide GGUF-ness from the on-disk files via the resolver (info_has_local_gguf) instead, run off the event loop, so the catalog advertises exactly what /v1 can serve. * Studio: fix /v1/messages/count_tokens route binding plus auto-switch review fixes The @router.post decorator for /messages/count_tokens had been separated from anthropic_count_tokens by the _validate_anthropic_client_tools helper, so the route bound to the validator and dropped its auth dependency. Move the decorator back onto the handler. Add route-binding tests asserting each /v1 endpoint maps to its handler with the auth dependency, so a decorator/handler split is caught at the route level (the direct-call tests missed it). Also from review: - update_openai_auto_switch writes both settings keys in one transaction so a PUT can't leave one updated and the other stale (drop the now-unused single setters). - max_seq_length override rejects 0 at the boundary (ge=1) instead of accepting then silently dropping it. - Document that embeddings auto-switch is best-effort: GGUF pooling has no cheap pre-load probe like vision's mmproj, so a guard would false-reject GGUF embedders. - Add a positive idle-unload test (loop frees the model and stashes it for reload). * Studio: validate Responses tool_choice + Anthropic mixed tools before switch, filter Ollama from catalog More auto-switch review findings: - /v1/responses rejects a forcing-function tool_choice with no name before the switch, mirroring chat, so a malformed request can't evict the resident model. - /v1/messages rejects mixing Anthropic server tools with custom client tools before the switch (the check depends only on the payload, so it moves up cleanly). - /v1/models no longer advertises Ollama-link models: info_has_local_gguf excludes .studio_links / ollama_links entries, which the resolver skips and can't switch to, so an advertised id never silently falls through. * Studio: guard chat audio input before switch; surface env-backed idle unload in settings UI A chat request carrying audio_base64 rides the same companion mmproj projector as a vision request, so a text-only target cannot serve it either. Flag require_vision for audio input as well so the multimodal probe runs before the switch and a rejected request never evicts the working model. Generalize the reject message to cover image and audio. The settings response now reports idle_unload_active (effective TTL > 0) so the UI can distinguish idle-unload that is active via the UNSLOTH_MODEL_IDLE_TTL env var from the case where it needs the toggle enabled. * Studio: harden auto-switch eviction guards (count_tokens vision, TTS reload-only, mmproj/stash) Four eviction/correctness fixes on the opt-in /v1 auto-switch path: - /v1/messages/count_tokens now carries the same require_vision guard as /messages, so an image count naming a text-only GGUF can't evict a loaded vision model for a swap that can't serve the request. - /audio/generate is now reload-only. A local GGUF's audio-input capability is not a cheap pre-load probe (the companion mmproj signal can't tell an audio projector from a vision one, and codec TTS ships no projector), so resolving the client model could load a text/vision-only target and evict the working audio model before the audio check fails. Only the idle-stash restore runs here; switching TTS models is an explicit /load. - The resolver no longer treats a standalone mmproj .gguf as a servable model. _scan_models_dir's standalone-file pass does not filter mmproj the way its directory scan does, so /v1/models could advertise a projector and a switch could load it over the real weights. - A non-GGUF (Transformers/Unsloth) load and a deliberate /unload now clear the idle reload stash, so a manual load/unload is never superseded by a stale idle-freed GGUF that the next /v1 request resurrects. * Studio: report advertised repo id consistently after an auto-switch Two model-id reporting fixes so an auto-switched cached HF GGUF is named by its repo id everywhere, not its snapshot path: - Streamed /v1/responses envelopes now derive the model id from _llama_public_model_id (which prefers _openai_advertised_id) instead of the raw model_identifier. After an auto-switch the identifier is the snapshot path while the repo id lives in _openai_advertised_id, so the stream used to report a snapshot basename while /v1/models, chat completions, and non-streaming Responses all reported the repo id. - When an advertised alias already resolves to the loaded model (a model loaded by local path, requested by its repo or LM Studio id), the already-serving early return now records the alias as the advertised id, so /v1/models and responses report the alias and mark it loaded instead of the path-derived basename. Resolver branch only; safe lock-free because an in-flight request blocks any concurrent swap via the single-slot busy guard. * Studio: validate request shapes before auto-switch (prompt/input/audio/mcp confirm) Four more validate-before-switch guards so a deterministic client error never evicts the resident model on the opt-in /v1 auto-switch path: - /v1/completions rejects an object/number prompt (only a string or array is valid) before the switch, instead of loading the named GGUF and letting llama-server reject the shape afterward. - /v1/embeddings rejects an object/number input the same way. - Chat rejects an oversized audio_base64 upload (413) before the switch. The size cap is a cheap, target-independent length check; the decode itself stays post-switch to avoid decoding a valid upload twice. - The chat confirm-without-stream pre-switch guard now mirrors the tool loop's actual enablement: _effective_enable_tools (honoring a CLI --enable-tools policy) and mcp_enabled (which opens the tool loop on its own but defers to a CLI --disable-tools policy). Previously a confirm+no-stream request with only mcp_enabled slipped past and 400'd after the swap. * Studio: fix model-id retrieval, streaming n>1, resolver cache TTL, keep-warm auth Four fixes from review: - GET /v1/models/{id} legacy raw-path fallback now maps the raw identifier to the same public id its /v1/models entry uses. After an auto-switch load the identifier is the snapshot path while the entry is keyed by the advertised repo id, so a client that cached the old absolute path no longer 404s on a model that is in fact loaded. - stream=true with n>1 is now rejected before the switch. Only the non-streaming GGUF path returns multiple choices, so streaming n>1 is invalid on every local serving path; both fields are known pre-switch, so it must not load model B only to 400 and evict model A. Non-streaming n>1 stays post-switch where the serving path decides. - The resolver index cache is stamped after _build_index, not with the pre-scan timestamp. On installs with enough local models for the multi-root scan to exceed the 5s TTL, the cache was stored already expired and every request rebuilt it. - The keep-warm middleware no longer stamps model activity for 401/403 responses. It runs before FastAPI auth, so unauthenticated probes used to refresh the idle timer without touching llama.cpp; they now decrement the in-flight count without keeping the model warm. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>	2026-07-01 06:42:23 -07:00
Daniel Han	187144d4e7	Reduce and tighten code comments and docstrings repo-wide (#6095) ... Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison.	2026-06-08 23:09:51 -07:00
Daniel Han	8292e699e4	Studio: make code comments and docstrings more succinct (#6029) ... Trim and tighten code comments and docstrings across studio/ Python. Comment-only: every changed file verified code-identical to main via AST/token comparison.	2026-06-08 23:07:28 -07:00
Daniel Han	3ce187da02	Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) ... Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.	2026-06-08 04:24:13 -07:00
Wasim Yousef Said	a5eb2e3d50	Add tauri (#5144) ... * add unsloth studio desktop app * Fix review findings - studio/src-tauri/tauri.conf.json: retarget updater to staging repo (danielhanchen/unsloth-staging-2); switch to unslothai/unsloth on upstream merge. - studio/src-tauri/linux/postremove.sh: drop the interactive read loop and the /home/* iteration. Package maintainer scripts must stay non-interactive and must not touch other users' data. - studio/frontend/src/app/auth-guards.ts: honor tauriAutoAuth() boolean. Failed auto-auth now redirects to /login; requireGuest/requirePasswordChangeFlow only redirect to /chat when auth succeeds. The new early-return on failed auth is intentional so the login / change-password flows remain reachable when desktop auth is not yet established. - studio/frontend/src/config/env.ts: keep fetched=false on health failure so later calls retry instead of caching the client-side platform guess. - studio/src-tauri/src/install.rs: pick the available system package manager (apt-get, dnf, zypper, pacman); AppImage bundles run on non-Debian distros. - studio/frontend/src/lib/open-link.ts + markdown-text/sources callers: return boolean from openLink so callers only preventDefault on handled URLs; relative hrefs now navigate natively. - studio/frontend/src/features/settings/tabs/about-tab.tsx: fetch(apiUrl(...)) so the version request targets the backend port in desktop mode. The bare /api/health predates the Tauri webview (blame: the earlier onboarding commit, which ran with same-origin frontend/backend); in desktop mode the webview origin is tauri://localhost so the bare path fails. - install.ps1: gate the install_python_stack.py hotfix on a sentinel comment instead of a content regex; append the sentinel after applying so reruns are unambiguous. - unsloth_cli/commands/studio.py _write_auth_secret: use the atomic mkstemp + os.replace path on Windows too; chmod calls are wrapped in try/except OSError. - studio/src-tauri/src/preflight.rs probe_existing_backends: fan out the health probes concurrently; desktop-auth status still runs sequentially per candidate. reqwest::Client is internally Arc-wrapped so the in-loop .clone() is a refcount bump, not a deep clone; annotated inline. - studio/src-tauri/src/preflight.rs run_cli_probe: wait() after kill() to reap the child, matching probe_cli_capability. - studio/src-tauri/src/process.rs + main.rs: add stop_backend_detached and use it from the tray quit handler so the 5s graceful-wait does not block the Tauri main loop. RunEvent::Exit keeps the synchronous safety-net call. - studio/backend/main.py: drop the permissive localhost CORS regex in api-only mode; the explicit allow_origins list is sufficient. - .github/workflows/release-desktop.yml: drop max-parallel: 1 so platform builds run in parallel, and lift releaseBody to an env var so the three tauri-action invocations share one source of truth. * Fix review findings (loop 2) - studio/backend/auth/storage.py update_password: clear_desktop_secret() alongside clear_bootstrap_password() so rotating the admin password also revokes any previously provisioned .desktop_secret. Without this, an old local desktop credential keeps minting fresh admin tokens via /api/auth/desktop-login after a password rotation. - studio/src-tauri/src/desktop_auth.rs provision_desktop_auth: wrap cmd.output().await in tokio::time::timeout(30s). DESKTOP_AUTH_LOCK is held across the whole desktop_auth flow, and previously a hanging `unsloth studio provision-desktop-auth` subprocess would pin the lock indefinitely and freeze every subsequent desktop_auth call. * Add review tests * Consolidate review tests Merge review-added tests into the existing studio/backend/tests/test_desktop_auth.py (the PR's authoritative desktop-auth test file). Drops three scaffolding files under tests/python/ in favor of five focused tests next to the tests they extend: - test_update_password_clears_desktop_secret (runtime) - test_update_password_on_unknown_user_leaves_desktop_secret_intact (runtime) - test_cli_provisioning_delegates_to_storage_create_desktop_secret (source-level) - test_cli_connect_auth_db_reads_storage_db_path (source-level) - test_desktop_auth_provision_has_bounded_timeout (Rust source-level) * Revert auth-guards.ts Tauri branches to unconditional form The review loop on PR 5144 introduced a regression: the isTauri branch of requireAuth redirected to /login when tauriAutoAuth() returned false, and requireGuest / requirePasswordChangeFlow silently fell through on the same condition. The Tauri desktop app authenticates via a local auto-generated secret; it must never surface /login or /change-password to the user. A failed auto-auth should let the startup layer retry, not expose a password form. Restore the three Tauri branches to the author's original unconditional form (requireAuth: return; requireGuest / requirePasswordChangeFlow: throw redirect({to: '/chat'})). Keep the rest of the review fixes -- the apiUrl() fetch wrapping, authRedirect helper, and fetchAuthStatus refactor are all legitimate improvements and are preserved. * Revert release-desktop.yml to author's version The review loop's workflow-file tweaks (drop max-parallel: 1, lift releaseBody to an env var) are cosmetic. OAuth tokens cannot push workflow-file changes, and fine-grained PATs cannot honor maintainerCanModify on a third-party fork. Reverting the workflow file to wasimysaid's version lets the push go through without needing a classic PAT with both repo and workflow scopes. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: Daniel Han <unslothai@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-04-23 04:50:10 -07:00
Daniel Han	9a261aec5f	Studio: Expose openai and anthropic compatible external API end points (#4956) ... * Studio: add API key authentication for programmatic access External users want to hit the Studio API (chat completions with tool calling, training, export, etc.) without going through the browser login flow. This adds sk-unsloth- prefixed API keys that work as a drop-in replacement for JWTs in the Authorization: Bearer header. Backend: - New api_keys table in SQLite (storage.py) - create/list/revoke/validate functions with SHA-256 hashed storage - API key detection in _get_current_subject before the JWT path - POST/GET/DELETE /api/auth/api-keys endpoints on the auth router Frontend: - /api-keys page with create form, one-time key reveal, keys table - API Keys link in desktop and mobile navbar - Route registered with requireAuth guard Zero changes to any existing route handler -- every endpoint that uses Depends(get_current_subject) automatically works with API keys. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Use actual origin in API key usage examples The examples on /api-keys were hardcoded to localhost:8888 which is wrong for remote users. Use window.location.origin so the examples show the correct URL regardless of where the user is connecting from. * Add `unsloth studio run` CLI command for one-liner model serving Adds a `run` subcommand that starts Studio, loads a model, creates an API key, and prints a ready-to-use curl command -- similar to `ollama run` or `vllm serve`. Usage: unsloth studio run -m unsloth/Qwen3-1.7B-GGUF --gguf-variant UD-Q4_K_XL * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add end-to-end tests for `unsloth studio run` and API key usage Tests the 4 usage examples from the API Keys page: 1. curl basic (non-streaming) chat completions 2. curl streaming (SSE) chat completions 3. OpenAI Python SDK streaming completions 4. curl with tools (web_search + python) Also tests --help output, invalid key rejection, and no-key rejection. All 7 tests pass against Qwen3-1.7B-GGUF. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add /v1/completions, /v1/embeddings, /v1/responses endpoints and --parallel support - llama_cpp.py: accept n_parallel param, pass to llama-server --parallel - run.py: plumb llama_parallel_slots through to app.state - inference.py: add /completions and /embeddings as transparent proxies to llama-server, add /responses as application-level endpoint that converts to ChatCompletionRequest; thread n_parallel through load_model - studio.py: set llama_parallel_slots=4 for `unsloth studio run` path * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make /v1/responses endpoint match OpenAI Responses API format The existing /v1/responses shim returned Chat Completions format, which broke OpenAI SDK clients using openai.responses.create(). This commit replaces the endpoint with a proper implementation that: - Returns `output` array with `output_text` content parts instead of `choices` with `message` - Uses `input_tokens`/`output_tokens` instead of `prompt_tokens`/ `completion_tokens` in usage - Sets `object: "response"` and `id: "resp_..."` - Emits named SSE events for streaming (response.created, response.output_text.delta, response.completed, etc.) - Accepts all OpenAI Responses API fields (tools, store, metadata, previous_response_id) without erroring -- silently ignored - Maps `developer` role to `system` and `input_text`/`input_image` content parts to the internal Chat format Adds Pydantic schemas for request/response models and 23 unit tests covering schema validation, input normalisation, and response format. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: add Anthropic-compatible /v1/messages endpoint (#4981) * Add Anthropic-compatible /v1/messages endpoint with tool support Translate Anthropic Messages API format to/from internal OpenAI format and reuse the existing server-side agentic tool loop. Supports streaming SSE (message_start, content_block_delta, etc.) and non-streaming JSON. Includes offline unit tests and e2e tests in test_studio_run.py. * Add enable_tools, enabled_tools, session_id to /v1/messages endpoint Support the same shorthand as /v1/chat/completions: enable_tools=true with an optional enabled_tools list uses built-in server tools without requiring full Anthropic tool definitions. session_id is passed through for sandbox isolation. max_tokens is now optional. * Strip leaked tool-call XML from Anthropic endpoint content Apply _TOOL_XML_RE to content events in both streaming and non-streaming tool paths, matching the OpenAI endpoint behavior. * Emit custom tool_result SSE event in Anthropic stream Adds a non-standard tool_result event between the tool_use block close and the next text block, so clients can see server-side tool execution results. Anthropic SDKs ignore unknown event types. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Split /v1/messages into server-side and client-side tool paths enable_tools=true runs the existing server-side agentic loop with built-in tools (web_search/python/terminal). A bare tools=[...] field now triggers a client-side pass-through: client-provided tools are forwarded to llama-server and any tool_use output is returned to the caller with stop_reason=tool_use for client execution. This fixes Claude Code (and any Anthropic SDK client) which sends tools=[...] expecting client-side execution but was previously routed through execute_tool() and failing with 'Unknown tool'. Adds AnthropicPassthroughEmitter to convert llama-server OpenAI SSE chunks into Anthropic SSE events, plus unit tests covering text blocks, tool_use blocks, mixed, stop reasons, and usage. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix httpcore GeneratorExit in /v1/messages passthrough stream Explicitly aclose aiter_lines() before the surrounding async with blocks unwind, mirroring the prior fix in external_provider.py (a41160d3) and cc757b78's RuntimeError suppression. * Wire stop_sequences through /v1/messages; warn on tool_choice Plumb payload.stop_sequences to all three code paths (server-side tool loop, no-tool plain, client-side passthrough) so Anthropic SDK clients setting stop_sequences get the behavior they expect. The llama_cpp backend already accepted `stop` on both generate_chat_ completion and generate_chat_completion_with_tools; the Anthropic handler simply wasn't passing it. tool_choice remains declared on the request model for Anthropic SDK compatibility (the SDK often sets it by default) but is not yet honored. Log a structured warning on each request carrying a non- null tool_choice so the silent drop is visible to operators. * Wire min_p / repetition_penalty / presence_penalty through /v1/messages Align the Anthropic endpoint's sampling surface with /v1/chat/completions. Adds the three fields as x-unsloth extensions on AnthropicMessagesRequest and threads them through all three code paths: server-side tool loop, no-tool plain, and client-side passthrough. The passthrough builder emits "repeat_penalty" (not "repetition_penalty") because that is llama-server's field name; the backend methods already apply the same rename internally. * Fix block ordering and prev_text reset in non-streaming tool path _anthropic_tool_non_streaming was building the response by appending all tool_use blocks first, then a single concatenated text block at the end — losing generation order and merging pre-tool and post-tool text into one block. It also never reset prev_text between synthesis turns, so the first N characters of each post-tool turn were dropped (where N = length of the prior turn's final cumulative text). Rewrite to build content_blocks incrementally in generation order, matching the streaming emitter's behavior: deltas within a turn are merged into the trailing text block, tool_use blocks interrupt the text sequence, and prev_text is reset on tool_end so turn N+1 diffs against an empty baseline. Caught by gemini-code-assist[bot] review on #4981. * Make test_studio_run.py e2e tests pytest-compatible Add a hybrid session-scoped studio_server fixture in conftest.py that feeds base_url / api_key into the existing e2e test functions. Three invocation modes are now supported: 1. Script mode (unchanged) — python tests/test_studio_run.py 2. Pytest + external server — point at a running instance via UNSLOTH_E2E_BASE_URL / UNSLOTH_E2E_API_KEY env vars, no per-run GGUF load cost 3. Pytest + fixture-managed server — pytest drives _start_server / _kill_server itself via --unsloth-model / --unsloth-gguf-variant, CI-friendly The existing _start_server / _kill_server helpers and main() stay untouched so the script entry point keeps working exactly as before. Test function signatures are unchanged — the (base_url, api_key) parameters now resolve via the new fixtures when running under pytest. * Rename test_studio_run.py -> test_studio_api.py The file is entirely about HTTP API endpoint testing (OpenAI-compatible /v1/chat/completions, Anthropic-compatible /v1/messages, API key auth, plus a CLI --help sanity check on the command that runs the API). None of its tests cover training, export, chat-UI, or internal-Python-API concerns. The old name misleadingly suggested "tests for the unsloth studio run CLI subcommand" — the new name reflects the actual scope. Updates: - git mv the file (rename tracked, history preserved) - Rewrite opening docstring to state the API surface focus and call out what is explicitly out of scope - Update all 4 Usage-block path references to the new filename - LOG_FILE renamed to test_studio_api.log - conftest.py fixture import rewritten from test_studio_run to test_studio_api, plus 7 docstring/comment references updated No functional changes to test logic, signatures, or main(). --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Fix httpcore asyncgen cleanup in /v1/messages and /v1/completions The earlier fix in 985e92a9 was incomplete: it closed aiter_lines() explicitly but still used `async with httpx.AsyncClient()` / `async with client.stream()` inside the generator. When the generator is orphaned (e.g. client disconnects mid-stream and Starlette drops the StreamingResponse iterator without explicitly calling aclose()), Python's asyncgen finalizer runs the cleanup in a DIFFERENT task than the one that originally entered the httpx context managers. The `async with` exits then trigger httpcore's HTTP11ConnectionByteStream .aclose(), which enters anyio.CancelScope.__exit__ with a mismatched task and raises RuntimeError("Attempted to exit cancel scope in a different task"). That error escapes any user-owned try/except because it happens during GC finalization. Replace `async with` with manual client/response lifecycle in both /v1/messages passthrough and /v1/completions proxy. Close the response and client in a finally block wrapped in `try: ... except Exception: pass`. This suppresses RuntimeError (and other Exception subclasses) from the anyio cleanup noise while letting GeneratorExit (a BaseException, not Exception) propagate cleanly so the generator terminates as Python expects. Traceback observed in user report: File ".../httpcore/_async/connection_pool.py", line 404, in __aiter__ yield part RuntimeError: async generator ignored GeneratorExit ... File ".../anyio/_backends/_asyncio.py", line 455, in __exit__ raise RuntimeError( RuntimeError: Attempted to exit cancel scope in a different task * Expand unsloth studio run banner with SDK base URL and more curl examples Add an explicit "OpenAI / Anthropic SDK base URL" line inside the info box so SDK users don't accidentally copy the bare server URL (without /v1) into their OpenAI/Anthropic SDK constructors and hit 404s. Replace the single /v1/chat/completions curl example with three labeled blocks: chat/completions, Anthropic /messages, and OpenAI Responses. The Anthropic example includes max_tokens (Anthropic SDKs require it even though Studio accepts None). All examples derived from a computed sdk_base_url so the /v1 prefix stays in sync if the public path ever changes. * Hash API keys with HMAC-SHA256 + persistent server secret Stores the HMAC secret in a new app_secrets singleton table. Fixes CodeQL py/weak-sensitive-data-hashing alert on storage.py:74-76, 394-395. Refresh tokens stay on plain SHA-256 (unchanged _hash_token) so existing user sessions survive upgrade — API keys are new on this branch so there is no migration. * Use PBKDF2 for API key hashing per CodeQL recommendation HMAC-SHA256 was still flagged by py/weak-sensitive-data-hashing. Switch to hashlib.pbkdf2_hmac, which is in CodeQL's recommended allowlist (Argon2/scrypt/bcrypt/PBKDF2). Persistent server-side salt stays in app_secrets for defense-in-depth. 100k iterations to match auth/hashing.py's password hasher. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com> Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>	2026-04-13 21:08:11 +04:00
Wasim Yousef Said	629199e3a6	fix: remove old comments (#4292) ... * fix: quotation marks * diceware passphrase generation * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2026-03-14 16:50:13 +04:00
Roland Tannous	47654cb91c	Final cleanup	2026-03-12 18:28:04 +00:00
Roland Tannous	a2baf80511	Update license headers	2026-03-12 17:23:10 +00:00
Roland Tannous	d882678fe4	Add AGPL-3.0 SPDX headers to all source files	2026-03-09 20:17:45 +00:00
Leo Borcherding	a3daae1c40	fix: replace datetime.UTC with timezone.utc for Python 3.9+ compatibility ... - Replace datetime.UTC with datetime.timezone.utc in authentication.py and storage.py - Fixes ImportError on Python versions < 3.11 - timezone.utc works on Python 3.9+ Resolves #237	2026-02-24 14:37:00 -06:00
Roland Tannous	01fcb4f713	authentication refactor - added setup token and token refresh mechanism	2026-02-11 12:09:47 +00:00
sshah229	0082627801	fixed the errors- renamed jwt to authentication, used raw jwt, and removed search route	2026-02-07 03:14:30 -07:00

Renamed from studio/backend/auth/jwt.py (Browse further)