* fix: add XPU device support and update hardcoded CUDA selections
* fix: add XPU device support for pytest CUDA skipped tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix device handling for PR #7401
- perplexity_eval.py: use DEVICE_TYPE_TORCH, not DEVICE_TYPE. The latter can
be "hip" or "mlx", which .to() rejects, so this regressed ROCm.
- test_batched_leftpad_generation_gpu.py: XPU diverges here today, so mark it
non-strict xfail on XPU instead of reverting to a CUDA-only guard. Keeps the
real XPU gap visible and turns green once it is fixed.
- Guard torch.xpu.is_available() with hasattr, matching device_type.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Re-enable the flash varlen attention test in CI for PR #7401
attention_dispatch.py now predefines flash_attn_func / flash_attn_varlen_func
as None, so test_run_attention_flash_varlen_receives_window_and_softcap no
longer needs flash_attn importable to be monkeypatched. Verified on a runner
shaped like the CPU-only one: the test fails against main's attention_dispatch
and passes at this head, so the deselect is now dead weight.
* Tighten comments for PR #7401
Drop the hasattr rationale: torch.xpu has existed since torch 2.3 and the
dependency floor is 2.4, so no supported build predates the namespace. The
guard stays as cheap defence, but the comment claimed something untrue.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
The manifest-removal guard added in #7492 exits with a bare 'exit 1', so in
Tauri mode the installer never emits the [TAURI:ERROR] line and the desktop
UI falls back to a generic failure instead of naming the cause. Every other
failure path in studio/setup.ps1 goes through Exit-SetupFailure, and
tests/sh/test_tauri_retry_failure_context.sh asserts that invariant, so
'Repo tests (CPU)' has been red on main since that merge.
Co-authored-by: danielhanchen <unslothai@gmail.com>
* Bound how many approvals may park, against the executor
#7455 landed parking, which is the right shape and supersedes what this branch
was carrying. It is unbounded, though, and the thing it is unbounded against is
not the GPU.
A run stopped on an approval prompt is blocked inside the to_thread(next, gen)
call that drives it, so it holds one of asyncio's default min(32, cpu + 4)
executor threads until the user answers. The slot cap used to bound that.
Parking hands the slot back, which admits another run that can park too, so the
ceiling became the wait line: 64 deep on a 1-slot backend. Long before that, the
executor is full and nothing else in the backend runs, including generation
steps for chats that already hold slots and the stream teardown that would clean
up after a disconnect.
The pool already permits `capacity` pending prompts, and each park adds one
more, so the budget is what the executor has left after the cap and a reserve of
4. On this machine (32 workers) --parallel 4 gets 8 parks and 20 free threads,
--parallel 24 gets 4 and 4, and --parallel 28 or higher gets none: there the
prompt keeps its slot and behaves exactly as it did before parking existed.
Counted process-wide rather than per queue. There is one executor, but a
per-queue budget is the same allowance again for every backend, and base_url
carries a fresh port on every model load, so a reload would mint a queue that
knows nothing about the approvals still parked on the old one. A reset clears it
too, or a leaked claim shrinks the budget for the life of the process.
park() reports whether it took the budget, and a refusal costs nothing to undo
because the slot never left its holder. The stream reads that answer rather than
recording a refused park as parked, which would make it skip the park for every
later approval in the same run even once the budget freed up.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Size the park budget from the executor's own CPU count
Two review findings, both real.
The budget read os.cpu_count(). 3.13 sizes ThreadPoolExecutor from
os.process_cpu_count(), which honours CPU affinity and cgroup quotas, and
asyncio's default executor is a plain ThreadPoolExecutor(), so a container
pinned to one core on a 64-core host got a 5-thread executor and a budget
computed from 64. The bound was then looser than no bound at all in exactly the
environment that can least afford it. It asks the same source the executor does,
and the test compares against a real ThreadPoolExecutor rather than restating
the formula, so it stays right on 3.12 as well.
The reserve was a flat 4, which on that same 5-thread executor left nothing to
budget and turned parking off entirely. Small hosts are where a chat most needs
to keep moving while another sits on a prompt. It scales now, and the ceiling
has a floor of two: a quarter of five is one, and one park cannot cover two
chats on prompts at once, which is what #7455's own two-approvals test needs.
Without that floor, that test fails on a one or two CPU runner. `spare` still
takes the budget to zero when the pool already fills the executor, so nothing
about a 32-worker machine changes: --parallel 4 still gets 8 parks, 24 gets 4,
28 gets none.
The two behavioural budget tests pin the worker count rather than reading it off
the runner, and the property test sweeps executor sizes from one CPU to 64
instead of asserting against whatever the host happens to have. The whole suite
passes with the CPU count faked to 1, 2 and 4, which is how both of these were
reproduced.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Size the park budget from every live backend, and free it on the answer
Two review findings, both real.
The budget was global but sized from one queue's capacity. A reload mints a
queue on a new port while the old one drains, so both are live, and prompts on
both park executor threads. Eight parks on an old 1-slot queue plus a new
24-slot backend is 32 threads on a 32-thread executor, with the new backend's
prompts refused and holding their slots, which is the state the reserve exists
to prevent. It sums the capacity of every backend still serving instead. Idle
queues are skipped: those are the ones the registry is about to evict, and they
are holding nothing.
The budget also outlived the wait it was paying for. unpark_async only dropped
it after reacquiring a slot, but the generator yields its post-approval event
first, so the executor thread is already back in the pool while the resume
queues. An approved chat waiting on a slot would refuse a different chat's park,
and that chat then keeps the slot the resumer is waiting for, so an unanswered
prompt strands chats that were already approved. The budget is released when the
prompt wait ends now, and the queue's parked count still runs until the slot is
back, which is what guards idle eviction and the resume ordering.
Both are separate counters on the lease as a result, and every exit from a park
drops the budget: unpark, unpark_async and release. That last one was the mutant
that came back missed, since a client disconnecting on a prompt releases
straight out of parked and would otherwise lose a budget slot for good.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten the comments on the park budget
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Keep `import unsloth` working when bitsandbytes is absent
device_type.py already prints "bitsandbytes is not installed - 4bit QLoRA
unallowed, but 16bit and full finetuning works" and clears
ALLOW_BITSANDBYTES / ALLOW_PREQUANTIZED_MODELS, but the import chain then
hard-required the module anyway, so `import unsloth` raised instead.
#7354 made this reachable: the gfx906 install path uninstalls the generic
bitsandbytes wheel (no gfx906 kernels in it), which leaves an MI50 / Radeon VII
host unable to import unsloth at all, not on the 16bit path the message
promises.
- kernels/utils.py: guard the bnb import; bind get_ptr and the five 4bit ctypes
handles to a stub that raises a clear message if a 4bit path is entered.
HAS_CUDA_STREAM stays False, which is the correct route.
- save.py, models/granite.py: guard Bnb_Linear4bit and peft's Linear4bit
(peft exports it only when bnb imported cleanly) with placeholder classes.
Both names only feed isinstance checks, so nothing matching is exact.
- _gpu_init.py: same degradation on the xpu branch as the cuda branch above.
Verified on a Strix Halo (gfx1151, DEVICE_TYPE=hip, torch 2.11.0+rocm7.13.0)
by blocking bitsandbytes with sys.modules["bitsandbytes"] = None, so
find_spec returns None and the import raises exactly as when the package is
absent. Before: ModuleNotFoundError at kernels/utils.py:136. After: import
succeeds, FastLanguageModel/FastModel import, ALLOW_BITSANDBYTES=False,
ALLOW_PREQUANTIZED=False, and the 4bit stub raises with the real cause. With
bitsandbytes present, every binding is unchanged.
New test walks the `import unsloth` module graph with ast and fails on any
unguarded bitsandbytes (or peft Linear4bit) import; verified it catches the
old code. Targeted suites: 702 passed, 18 skipped.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address the review on #7502: zoo coupling, non-hip flags, py3.9 collection
Three findings, each reproduced first and negative-controlled after.
1. The fix still needed an unreleased unsloth_zoo (P1). save.py imported
unsloth_zoo.saving_utils at module scope, and any zoo without the companion
#953 fix imports bitsandbytes there, so `import unsloth` kept failing for a
dependency set pyproject.toml allows. Raising the floor was not an option:
PyPI's newest zoo is 2026.7.6 and #953 is merged but unreleased, so a bump
would break every install today. Both names it pulled in are used only inside
functions, so the import is now lazy at those two call sites, matching what
determine_base_model_source in the same file already does. Verified against a
real pre-#953 zoo checkout with bitsandbytes blocked: import succeeds, and
restoring the eager import reproduces the failure at saving_utils.py:70.
This PR no longer depends on a zoo release.
2. Capability flags were only cleared on hip (P2). device_type.py probed
bitsandbytes inside its DEVICE_TYPE == "hip" branch, so a cuda or xpu host
without bnb imported fine but still reported ALLOW_BITSANDBYTES=True, and the
default load_in_4bit=True path in models/loader.py would select a 4bit
checkpoint before failing. Clear both flags whenever the module is absent, on
every backend, via find_spec so a working install pays nothing. A cuda host
with bnb blocked now reports False/False; with bnb present nothing changes.
3. The new test could not be collected on Python 3.9 (P2). `Path | None` is a
PEP 604 union and requires-python still allows 3.9, so pytest raised
TypeError at import. Added `from __future__ import annotations`. Checked in
real uv venvs on 3.9, 3.10 and 3.13: 2 passed each; removing the future
import reproduces "unsupported operand type(s) for |" on 3.9 only.
The xpu branch in _gpu_init.py needs no separate flag handling now that the
probe is backend-independent.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address the second review on #7502: guarded probe, and 8bit in the same guard
1. The capability probe used find_spec while the fallbacks in kernels/utils.py
and _gpu_init.py treat any import failure as unavailable, so an installed but
unusable wheel would leave ALLOW_BITSANDBYTES true while the kernels had
already bound the stub. Probe with the same guarded import instead, so all
three agree by construction. No new cost on any path: _gpu_init.py already
imports bnb before device_type is reached on cuda, and device_type's own hip
block imports it a few lines later.
Worth recording that the state this prevents is currently unreachable for an
unrelated reason: a broken wheel takes `import unsloth` down earlier, in
transformers/integrations/bitsandbytes.py:20 via
unsloth_zoo/patching_utils.py:680, whichever exception it raises (OSError also
escapes the zoo moe_utils `except ImportError`). So this is correctness for
when those imports get guarded, not an observable fix today.
2. Both loader guards printed for load_in_4bit or load_in_8bit but only cleared
load_in_4bit, so an explicit load_in_8bit=True survived and reached
Transformers, which builds the bnb quantizer and fails there. Clear both. The
message no longer says AMD either: the flag now goes false whenever bnb is
unusable on any backend.
Tests: the probe must not use find_spec, and an ast walk requires every
ALLOW_BITSANDBYTES guard in loader.py to clear both flags, so a third guard
cannot be added with the same omission. Dropping either fix reddens them (1 and
2 failures respectively). 4 passed on 3.9, 3.13 and the ROCm venv; absent and
healthy bnb both stay consistent across hip and cuda.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop the importlib import left over from the find_spec probe on #7502
* Address the third review on #7502: exact-name bypass and a forwarded bnb config
Both findings hold up, so both are fixed.
1. use_exact_model_name=True skipped the guard entirely. load_in_4bit defaults
to True, so on a host without bitsandbytes
FastLanguageModel.from_pretrained(name, use_exact_model_name=True) kept 4bit
set and failed downstream. That option suppresses repo-name remapping and
cannot make bitsandbytes available, so it has no business gating a capability
check. Ungated at both sites.
2. A user-supplied quantization_config survived the fallback. It sets
load_in_4bit/8bit at the top of from_pretrained and stays in kwargs, so
clearing the local flags still let Transformers rebuild the bnb quantizer.
Now dropped as part of the fallback.
One correction to the second suggestion: it cannot be dropped whenever the
fallback runs. quantization_config also carries GPTQ, AWQ, fp8 and torchao
configs, which have nothing to do with bitsandbytes and must reach the loader
untouched. The pop is gated on the config actually requesting load_in_4bit or
load_in_8bit, reusing the same dict/attr probe from the top of the function.
Behaviour, exercising the real guard block against synthetic inputs with
use_exact_model_name=True and bnb unusable:
default 4bit, no cfg 4bit=False 8bit=False
explicit 8bit, no cfg 4bit=False 8bit=False
BitsAndBytesConfig(4bit/8bit) 4bit=False 8bit=False config dropped
dict bnb config 4bit=False 8bit=False config dropped
GPTQ config 4bit=False 8bit=False config SURVIVES
fp8 dict 4bit=False 8bit=False config SURVIVES
Nothing changes when bitsandbytes works: the whole block is inside
`if not ALLOW_BITSANDBYTES`.
Tests: an ast walk requires neither guard to reference use_exact_model_name in
its test, and requires each to pop quantization_config behind a _wants_bnb
check, so an unconditional pop fails too. Re-gating one guard or removing one
pop reddens a test each. 6 passed on 3.9, 3.13 and the ROCm venv.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address the fourth review on #7502: FastModel never reached the 16bit path
Both findings are real, and the second one meant this PR did not actually
deliver what it advertises for FastModel or vision loads. Reproduced first.
1. patch_compiling_bitsandbytes() ran unguarded at the top of
FastModel.from_pretrained, and unsloth_zoo's copy imports bitsandbytes
unconditionally (patching_utils.py:40). So every FastModel call on a
bnb-less host died there, whatever the arguments:
FastModel(load_in_16bit=True) -> ModuleNotFoundError at patching_utils.py:40
FastModel(full_finetuning=True) -> ModuleNotFoundError at patching_utils.py:40
The FastLanguageModel path already wraps this call in try/except with a
warning, and its comment even says "Mirror FastModel" - FastModel was the
unwrapped one. Wrapped it the same way, so behaviour is unchanged wherever
bitsandbytes imports.
2. The mode-exclusivity check ran before the capability fallback. load_in_4bit
defaults to True, so load_in_16bit=True made
int(load_in_4bit) + int(load_in_16bit) == 2 and raised "Can only load in 4bit
or 8bit or 16bit" before the fallback could clear the unavailable 4bit
request. Moved the fallback ahead of that check.
After both, the same three calls get past every bitsandbytes gate and reach
model resolution, failing only on the deliberately fake repo name used by the
probe. Nothing changes when bitsandbytes works: the fallback is still inside
`if not ALLOW_BITSANDBYTES`, and the wrapper only swallows an import that
previously crashed the load.
Tests: the mode check must be preceded by an ALLOW_BITSANDBYTES fallback in the
same function, and no call to patch_compiling_bitsandbytes may sit outside a
try. The ordering assertion is scoped to the enclosing function on purpose - my
first version compared line numbers file-wide, so the other loader's guard
satisfied it and the negative control passed when it should have failed. With
the scoping fixed, moving the fallback back after the mode check reddens it, as
does unwrapping the patch call. 8 passed on 3.9, 3.13 and the ROCm venv.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add Unsloth desktop deep links
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address deep-link review feedback
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
studio-tauri-smoke.yml only ever built the crate, so none of its ~100 unit
tests executed. Running them surfaced two that were broken on platforms CI
never exercised:
- non_utf8_import_name_preserves_csv_extension built a filename containing a
raw 0xFF byte. Linux stores that fine, macOS enforces UTF-8 on APFS/HFS+ and
refuses to create it, so the test panicked on the unwrap. Skip when the
filesystem rejects the name; the branch under test is only reachable where
such a file can exist.
- losing_a_studio_package_changes_the_fingerprint created the posix venv
layout unconditionally, but site_packages_dirs() only walks
lib/<pyver>/site-packages on unix and looks at Lib/site-packages on Windows.
The dist-info was therefore invisible to the fingerprint there, removing it
changed nothing and the assert_ne could never hold. Build the layout the
code actually reads for the target platform.
Add the cargo test step to the existing Tauri job, where the toolchain and
WebKit dev packages are already installed.
* fix(studio): reuse cached tokens while highlighting streaming code blocks
A streaming fence re-enters highlight() every animation frame with the whole
block, so Shiki re-tokenizes it from scratch each time: O(length) per frame and
O(length^2) over the message. One generation made 808 highlight() calls and
tokenized 5.5MB of text to render a 13.5KB block, putting ~50% of the renderer
main thread in the TextMate tokenizer.
Blocks under 2000 chars are unchanged. Above that, a growing fence reuses the
tokens from the last real tokenization and appends the new tail unstyled, with
a full re-tokenize at most every 250ms.
* fix(studio): render the streamed tail unstyled and always converge
Two defects found while property-testing the reuse path:
- plainLine() spread the template token, so newly streamed lines inherited the
first token's colour instead of the default foreground. Emit a bare token.
- A reused result could be the final one if the caller stopped re-rendering,
leaving the tail permanently unstyled. Schedule a trailing re-tokenize so a
reused run always converges.
* fix(studio): key the highlight cache per fence and keep tokens paired with code
Review found four real defects in the previous approach:
- entry.code advanced at dispatch time while entry.result still held the older
tokens, so a reuse could slice one against the other and drop text from the
cached run's final line.
- A finished fence re-rendered with identical code re-dispatched every frame,
keeping the per-frame cost for the rest of the stream.
- All fences of one language shared a single entry, so sibling fences evicted
each other and both were fully tokenized on every render.
- An overdue trailing timer could dispatch stale code after a newer dispatch.
Cache is now one slot per fence, matched by longest prefix. code and result
only ever move together, an exact match is served straight from cache, and a
direct dispatch or a slot eviction cancels any pending trailing refresh.
* Studio: adopt synchronous highlight results and use a monotonic throttle
@streamdown/code answers out of its own cache synchronously and never invokes
the callback in that case. dispatch() ignored that return value, so the slot
kept pointing at the older tokens. On the trailing refresh, where nothing else
consumes the return, that left the fence showing its unstyled tail until an
unrelated remount. Adopt the synchronous result on both paths and hand it to
the pending callback.
Drive the throttle off performance.now(). Date.now() is wall clock, so a
backward step from an NTP correction or a resume from sleep makes elapsed
negative, which pins the reuse branch on and schedules the trailing refresh by
the size of the step.
* Tighten code-plugin comments
---------
Co-authored-by: shimmyshimmer <info@unsloth.ai>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* fix(rocm): prefer system LLVM runtime on native Linux
* Fix/adjust the nested LLVM probe for PR #7448: lib64 hosts and non-directories
Two gaps found while simulating the fix against real ROCm layouts.
1. lib64 hosts got no LLVM dir. The candidate was built from the HSA dir, so a
host with libhsa-runtime64 under lib64 probed <root>/lib64/llvm/lib. ROCm
installs LLVM under <root>/lib/llvm regardless, so that host kept binding
system libamd_comgr to the bundle's libLLVM: exactly the bug #7446 reports.
Probe both spellings, the HSA dir's own first so a genuine lib64 layout still
wins. When lib_sub is already "lib" the seen set collapses them.
2. os.path.exists accepted a non-directory. The serve-time caller joins these
straight into LD_LIBRARY_PATH with no is-dir filter, so a file named
llvm/lib reached the loader. os.path.isdir instead.
Verified on a 27-case matrix built from real directory trees (not mocks), run on
both Windows and Linux against three revisions: main, this PR as-is, and this
commit. Zero regressions and zero reorderings of the pre-existing entries in
every case, and the installer and launcher copies never disagree. The lib64 case
goes [lib64] -> [lib64, lib/llvm/lib]; the file case drops the bogus entry; a
symlinked llvm/lib resolves correctly on Linux.
End-to-end loader check: built real ELF objects mirroring the shipped bundle
(RUNPATH=$ORIGIN, an incomplete libLLVM.so.23.0git next to llama-server, system
comgr from /opt/rocm/lib) and reproduced the reported failure verbatim, then
confirmed the prepend clears it:
before undefined symbol: LLVMInitializeSPIRVTarget -> after exit 0
Test helper now patches os.path.isdir alongside os.path.exists, else every fake
host reports its nested llvm dir as missing. New cases: lib64 finding llvm under
lib, lib64 preferring its own when both exist, and a real-filesystem check that a
non-directory is not prepended. Removing the lib fallback from one copy reddens
three tests including the two-copy parity guard.
tests/studio/install: 1361 passed on Linux, 4 pre-existing environmental
failures unchanged (3 managed-node-runtime under root, 1 the real /opt/rocm case
already covered by #7397). 30/30 on the helper suite on Windows and Linux.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: add durable Deep Research workflows
* Studio: preserve research integration after upstream updates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep research worker compatible with Python 3.11
* Studio: address Deep Research lifecycle review
* Studio: preserve durable research recovery
* Studio: preserve research stream and context
* Studio: harden research sources and limits
* Studio: align research with shared chats
* Studio: guard durable research actions
* Studio: protect durable research turns
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: deepen durable research decisions
* Studio: protect research prompts and queries
* Studio: slim research stream deltas
* Studio: preserve research evidence and citations
* Studio: harden Deep Research (CI, prompt injection, query PII, config, citations)
- Fix backend CI: add research_runs_router to the synthetic routes stub in
test_desktop_auth so studio.backend.main imports under the health-check test.
- Escape prompt-delimiter tags in the decision and synthesis prompts so gathered
web/document content cannot close an <untrusted_...> wrapper and inject
instructions into the local planner/decision/synthesis model.
- Extend the public-query sanitizer to redact Luhn-valid payment cards, phone
numbers, non-global IPs, and labeled private identifiers before a query can
reach web search.
- Reject nested credential keys in inferenceRequest and ragScope, not just
top-level keys, when persisting a durable run config.
- Treat maxSources as one budget shared across web and document sources
(collection and resume paths) instead of per type, which allowed up to 2x the
configured cap.
- Preserve document citations whose filename contains a closing bracket by
tokenizing valid citations before stripping invalid ones.
- Persist Deep Research off when switching to an external model and when enabling
Web Fetch so a refresh cannot rehydrate a mutually-exclusive state.
- Add regression tests for the query, prompt, citation, and config hardening.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make the research claims table migration atomic
The owner-scoped to global claims migration ran its RENAME, CREATE, INSERT and DROP in autocommit, so an interruption after CREATE left the new table empty, orphaned the rows in the legacy table, and never re-triggered. Wrap the rebuild in an explicit transaction so a crash rolls back cleanly and the migration re-runs on the next boot.
* Studio: block message edits and regeneration during an active research run
After a reload a durable research run is followed by the research store rather than an assistant-ui run, so thread.isRunning is false while research is still active. Message edit, refresh and the edit composer previously gated only on isRunning, which let a normal generation start alongside the running research run. Gate them on the active thread's research state as well.
* Studio: keep the plan review mounted through approval
Keying PlanReview on planRevision remounted it mid-approve when updateResearchPlan bumped the revision, resetting the local pending flag and re-enabling Start research while the approve was still in flight, which allowed a duplicate approve. Key on runId only.
* Studio: drop the redundant deep-research persistence change
setCheckpoint already persists Deep Research off for external models at the top of the function, so the added saveBool was a duplicate, and clearing Deep Research from setWebFetchToolsEnabled guarded a state that is not reachable (Deep Research is local-model only while the Web Fetch pill is external-provider only). Revert both to the pre-hardening version.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden Deep Research citations, query privacy, and message protection
Address review findings in the Deep Research backend:
- Escape an unbalanced ")" in citation destinations so a source URL cannot
close the markdown link early and inject a second link, keeping balanced
parentheses literal.
- Match raw-URL citations on whole tokens so a URL sharing another URL's
prefix is no longer partially rewritten.
- Redact non-global IPv6 addresses in public search queries, matching the
existing IPv4 handling.
- Detect credential key names after normalizing case and separators so nested
openaiApiKey, accessToken, and clientSecret values cannot be persisted.
- Reject client edits to server-managed research prompts and reports at the
storage layer; only the internal writers pass allow_research_update.
- Scope research searches to the first allowed domains instead of dropping
site scoping for large allow lists.
- Persist the same fetch evidence bound used during live synthesis so a
resumed run is not shortened.
- Scope run completion so it only replaces this run's message parts.
Add regression tests for the above.
* Studio: fix Deep Research SSE framing, source counts, and favicon privacy
- Normalize the whole SSE buffer so a CRLF split across transport chunks
still frames events.
- Count web and document sources together in the activity header so a
RAG-only run is not shown as zero sources.
- Cap the plan editor at the run's configured maxSteps instead of a
hard-coded 30.
- Add an allowRemoteIcons opt-out to the sources components and disable
third-party favicon requests for research sources so visited domains are
not leaked.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address final Deep Research review findings
* Studio: fit Deep Research synthesis evidence to loaded context, add opt-in web grounding
Size the synthesis evidence budget to the loaded model context so the prompt is not
silently truncated on small contexts. When the evidence overflowed the window the report
degenerated (it echoed the evidence tail instead of writing); the budget now reserves tokens
for the prompt scaffolding and converts the remainder to chars, keeping the full cap when the
context is unknown.
Add opt-in web grounding for auto-read: read the top search results, ingest them into an
ephemeral RAG scope, hybrid-retrieve the passages most relevant to the question with the
existing knowledge-base retriever, and fold those chunks into the step evidence. The scope is
per call and deleted afterwards, so a user's knowledge base is never touched.
Off by default; enable with UNSLOTH_RESEARCH_AUTO_SCRAPE=1. Gated per run by
budgets["maxAutoScrape"], so runs created without it keep legacy snippet-only behavior, and
grounding is skipped when the loaded context is too small for the prompt.
Add tests for the adaptive evidence budget, scraped-text cleaning, the ephemeral web-RAG
retrieval and scope cleanup, and the auto-read evidence path.
* Studio: read Deep Research synthesis context from the inference orchestrator
Make the adaptive synthesis-evidence budget actually engage in the normal Studio
architecture. _loaded_context_length read core.inference.inference, the low-level backend that
lives in the model subprocess and stays unpopulated in the main web process where the research
supervisor runs, so it returned None and the budget silently fell back to the 32000 character
cap (leaving the report exposed to the truncation this was meant to fix). Read the inference
orchestrator instead, and the llama.cpp backend for GGUF, mirroring
routes.inference._monitor_context_length so the budget sizes to the context the API layer
serves. Verified on a running server: at a 12288 token load the probe now reports 12288 and the
budget adapts to 24576 characters instead of the 32000 fallback.
Also:
- Reserve context for the generated report as well as the prompt scaffolding (raise the reserve
to 4096 tokens) so evidence does not crowd out the output on a small window.
- Honor a numeric UNSLOTH_RESEARCH_AUTO_SCRAPE by passing the per-run maxAutoScrape as the page
cap to the scraper, instead of always reading the maximum.
- Guard the web-RAG connection acquisition so a get_connection failure returns the documented
empty result rather than propagating.
- Add a synthesis-context test that patches the real backend accessor (not the probe itself) so
the production wiring is exercised, plus a scrape page-cap test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden Deep Research query redaction and research autosave
- research_runs: extend the opaque-token allowlist so unlabeled Hugging
Face (hf_) and GitLab (glpat-) tokens are redacted before a query can
reach web search, without over-redacting public model or version ids.
- runtime-provider: for a server-managed research message, echo the
backend-stored metadata verbatim on autosave. Merging the client
metadata re-added client-only fields the server never persisted, so the
server-side guard saw a diff and rejected every streamed or snapshot
update with 409.
* Studio: keep composer tool pills always accessible after merge
The merge left the composer line marked always-expanded (data-expanded
"true") while the inner pill row was still gated behind composerExpanded,
so the Search and Code toggles disappeared once the permission mode was
"off" with no other toggle set. Render the primary tool pills
unconditionally, matching the always-expanded layout, and drop the now
unused composerExpanded and permissionMode locals. Fixes the Chat UI
Playwright check that asserts the Search and Code pills stay visible.
* Studio: update Deep Research composer contract to always-expanded layout
The always-expanded composer no longer routes effectiveDeepResearchEnabled
through a composerExpanded expression, so the frontend contract now checks
that it gates the Deep Research composer button render instead.
* Studio: do not bind a research run to a populated assistant reply
create_run adopted any assistant message under the user turn whose
researchRunId was unset, including a prior answer reused by a retry. On
completion _update_assistant drops the untagged text and source parts, so
that answer was silently overwritten. Only bind to an empty placeholder or
this run's own message, and reject a reply that already carries content.
* Studio: harden Deep Research synthesis budget, prompt shielding, and message protection
- research_runs: split the synthesis evidence budget evenly across notes so a
small context still keeps a slice of every research step instead of dropping
the later steps after the earliest ones fill the budget.
- research_runs: shield the research question and approved plan before placing
them in the decision and synthesis prompts, so a closing delimiter in either
cannot escape its block and inject sibling sections.
- research_runs: redact bearer authorization tokens from public search queries.
- studio_db: include attachments in the research-message change check and guard
direct attachment deletion, so server-managed research prompts and responses
cannot be mutated through the attachment paths.
- chat_history: map the protected-message conflict on attachment deletion to 409.
* Studio: strip invalid document citations that contain brackets
The invalid-citation regex stopped at the first closing bracket, so a
citation whose filename contained brackets left its tail (".pdf, p. 9]") in
the report. Match a balanced bracketed span so the whole invalid citation is
removed; valid citations stay protected by the earlier tokenization pass.
* Studio: free the RAG search slot when a lookup times out or is cancelled
The bounded knowledge-base search held the sole admission slot in a detached
worker until the search returned, so a lookup that outlived its timeout (a
stalled embedding or blocked vector call) kept the slot forever and starved
every later lookup, disabling knowledge-base retrieval globally. Release the
slot from the caller when it stops waiting, exactly once, so a detached worker
finishes without re-holding it.
* Studio: remove Websites label from research composer
* Studio: fix Deep Research review findings (RAG slot bound, orphaned workers, hardening)
- Bound the shared RAG search slot to one running worker. The search that is
doing the embedding/index/GPU work now owns the admission slot until it
finishes, instead of freeing it on caller timeout while the detached worker
keeps running, which let a second search enter and stack concurrent work
behind the capacity-of-one semaphore.
- Cancel active research runs before deleting their thread, project, or all
history. Deleting cascade-drops the run row, but the worker only notices at
its next lease check, so it could keep doing model/web/RAG work for a run
that no longer exists; signalling cancel first shortens that window.
- Shield the planner prompt's conversation and question with _shield_untrusted,
matching the decision and synthesis prompts, so untrusted text cannot forge
planner delimiters.
- Do not let a research key-revocation failure replace a successful
non-streaming completion; log it like the streaming path does.
- Include created_at in the protected research-message guard so a client cannot
reorder server-managed prompt/response messages while leaving the body intact.
- Reject non-scalar ragScope values; a nested container evades the
sensitive-key scan when its inner keys are unlisted and would reach retrieval
code that expects a scalar scope id.
Adds regression tests for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: remove research composer globe icon
* Studio: use Hugeicons telescope in research composer
* Studio: use Telescope02 icon in research composer
* Studio: standardize Deep Research telescope icons
* Studio: move Deep Research below web and code tools
* Studio: merge grounded page excerpts with search snippets instead of replacing
When auto-scrape grounding retrieved page-body chunks, it replaced the raw
search-result text for that step. If the retrieved chunk was a distractor or
dropped the key fact, the answer-bearing search snippet was lost and grounded
runs regressed below snippet-only accuracy on factual questions (e.g. returning
Apache 2.0 instead of the Qwen License, 403 instead of 404, or a single mirror
diameter instead of the sum).
Keep the search snippets and append the grounded excerpts as supplementary
evidence via a small _merge_scraped_evidence helper. Grounding stays opt-in and
off by default, so legacy runs are unchanged. Adds regression tests.
* Studio: improve Deep Research synthesis
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden Deep Research synthesis flow
* Studio: validate Deep Research derived context
* Studio: align Deep Research synthesis evidence
* Studio: restore Deep Research synthesis state
* Improve Deep Research source queries
---------
Co-authored-by: alkinun <alkinunl@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
* Fix Windows no-torch setup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix no-torch env normalization on Windows
* Accept on for Windows no-torch mode
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep no-torch mode across studio update on Windows
Guarding the direct torch/Triton install made `install.ps1 --no-torch`
actually produce a torch-free venv, which then broke the next
`unsloth studio update`. That path exports no UNSLOTH_NO_TORCH, so
$NoTorchMode was false, the stale-venv check read the missing torch as a
broken venv, and setup tried to delete the venv it was running out of:
[ERROR] Could not remove stale venv: Access to the path 'python.exe' is denied.
That teardown can never succeed there, because setup.ps1 runs via
unsloth.exe out of that same venv. The same gap also let the shared
dependency pass reinstall torch from PyPI, unpinned, into a GGUF-only
environment.
install_python_stack.py now records the mode in the install manifest and
setup.ps1 reads it back when no env var is exported, then re-exports a
canonical value for the dependency pass (setup.ps1 drops the manifest
before invoking it, so the child cannot repeat the lookup). The key is
additive and MANIFEST_SCHEMA is unchanged, so existing manifests stay
valid and a missing key keeps today's behaviour.
Also:
- read_manifest() caught only OSError, but UnicodeDecodeError is a
ValueError. That is now on the installer's import path, so a manifest
re-saved as ANSI or truncated mid-write would abort every install.
- The env predicate now trims surrounding whitespace, matching the
Python side.
- The Windows update smoke workflow asserts the update leaves the venv
GGUF-only, which is what would have caught this.
Known follow-up, pre-existing: an install killed between the manifest
drop and the dependency pass leaves no recorded mode, so a later update
still walks the stale-venv path. Closing that needs a marker the
installer never drops.
* Persist no-torch mode in a marker the dependency pass cannot drop
The install manifest alone was not enough. Both setup.ps1 and
install_python_stack.py remove it before every dependency pass, and it is
only rewritten on success, so a no-torch install interrupted in between
left nothing recording the mode. The next update then resolved no-torch
as false, read the expected missing torch as a stale venv, and tried to
delete the environment whose python.exe was running it, which leaves the
install unrepairable from the CLI.
Add .unsloth-no-torch next to the existing .unsloth-studio-owned marker,
written before the pass and cleared when torch is wanted. setup.ps1
writes it as soon as the mode resolves, so the window between the
manifest drop and its own torch install is covered too.
Read order stays manifest key first, then marker, so migrating out of
no-torch is never blocked by a marker an earlier run left behind. Neither
present still reads as "install torch", so nothing changes for installs
made before either existed.
Also adds the AGPL-3.0 header the new test file was missing.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
* Gate the sed commands that run a shell
GNU sed executes a shell through its `e` command, both as a standalone
command (`sed -n '1e CMD' file`) and as an `s///e` flag that runs the
pattern space. It goes through popen(), so it is a literal `sh -c`, but
the terminal scan only ever saw `sed` at command position and treated the
program text as an ordinary argument.
That left `sed -n '1e rm -f victim' /etc/hosts` running with no prompt in
auto mode, and `_find_blocked_commands` returning nothing for it, so the
hard blocklist that applies in every mode missed `rm` as well.
Screens the program the same way the awk arm does. `-e` values are joined
with newlines first, since that is how sed assembles them: `sed -e '1a\'
-e 'e CMD'` appends a literal line and runs nothing, so judging the pieces
separately would prompt on a benign script. The scan then steps over every
region where `e` is data rather than a command: address and substitution
regexes, replacements, `a/i/c` text, `r`/`w` filenames, `b`/`t` labels and
comments. That keeps the common idioms silent, including `:e;N;$!be` loop
labels, `s/e/E/g`, and `s/a/b/we out.txt` where the `e` belongs to the `w`
filename and sed does not execute.
The blocklist scan recurses into a literal `e` payload the same way it
already does for `bash -c`. A bare `e` or an `s///e` can only be prompted,
since what they run is the pattern space, which is input-file text that is
not knowable statically.
Verified against real GNU sed 4.9 rather than the manual: 80 commands run
for real with a marker payload, comparing what sed actually executed
against the classifier, with no mismatches in either direction.
* Close five ways a sed program hid its shell payload
Review found five shapes the first pass missed. All five execute on GNU
sed 4.9, checked by running them rather than reading the manual.
A payload line ending in a backslash continues onto the next line, so the
scan now ends an `e` at an unescaped newline and unescapes the text the way
sed's read_text does. That is what resolves `r''m` back to `rm` for the
blocklist.
A sed comment ends at a real newline, but the terminal scan had already
replaced every newline with `;`, including newlines inside quotes, so
`# comment` swallowed the rest of the program. The sed arm now also sees a
variant where only unquoted newlines become separators, built on a
character-by-character quote scanner rather than a regex: an apostrophe in
a double-quoted word mis-pairs under a regex and inverts the state, which
opened a bypass while this was being written.
Everything attached to `-i` is a backup suffix, so reading `-ifoo` as an
attached `-f` lost the real script. Replaced the shared short-flag helper
with sed's own option grammar, which also fixes `-l 5` and
`--line-length 5` eating the script as their operand.
A sed child of `find -exec` was never recorded, so the blocklist skipped
its payload.
Substituted text splices straight into the program, and an address is as
good a place as any to open `;e CMD`, so a command substitution anywhere
in the program is treated as unresolvable. Scoped to the program: a
substitution in a file operand still runs, a `$(` or backtick inside single
quotes is literal, and parameter and arithmetic expansion are untouched.
The cost is that a substitution used to build a program now asks.
Bounding the -exec walk keeps the blocklist linear; without it a repeated
`-exec sed` line went quadratic.
Verified against real GNU sed across 103 commands run for real, no
mismatch in either direction.
* Fail closed on padded sed lines, and stop gating sed --sandbox
Four more from review, each checked by running it rather than reading the
manual.
The cap that keeps the argument walk linear was itself the bypass: padding
a line with 128 valid options pushes the script past it, and an empty
program read as proof the command only edits text. The budget is now shared
across the sed words on a line, so a lone sed reads its whole argument list
while a line packed with sed words keeps the floor that holds the walk
linear, and overflow fails closed instead of falling through.
The substitution scan counted parentheses without consulting quote state,
so a quoted paren in the substitution body left the span unterminated and
the program never matched. It now balances through the same quote scanner
used elsewhere, since a substitution body reopens quoting.
A wrapper between -exec and its child hid the child from the blocklist.
Following the wrapper also fixes the neighbouring blocked-name check, which
missed find . -exec env rm the same way. The wrapper's own name is still
screened: -exec sudo rm reports both.
sed --sandbox and --posix refuse e outright and exit 1, so gating them was
prompting for something that cannot run. They are now inert, except after
--, where the flag is an input filename and the script still executes.
env -u still hides a child from the blocklist, on this path and at top
level. That is pre-existing and left alone here.
* Resolve the sed program through find, wrappers, globs and variables
Five more from review, each run against real sed rather than read off the
manual.
find's -exec ends at + or ;, but the sed argument walk ran past it into the
next predicate, where a following -exec grep -e safe was read as sed's own
-e and discarded the real script. Stopping at the terminator also removes a
false prompt, since -exec was being parsed as -e xec and inventing a payload.
Hopping a wrapper skipped its name but not an option that takes a separate
operand, so env -u FOO sed returned FOO as the child. The table this file
already keeps for wrapper options covers it, moved up so both layers share
it. That also settles the top level: env -u PATH rm -rf x now reports rm,
as do env --unset, stdbuf -o L and xargs -I {}. Two false positives go with
it, timeout -s KILL 5 rm blaming the signal name and env -u kill blaming a
variable name, while timeout -s KILL 5 kill -9 1 still reports kill.
A program held in a variable was invisible: the assignment regex stops its
value at whitespace, so a program containing a newline never entered the
map in any pass. Resolved at the token level instead, where the value is
already whole. Both the written and the resolved program are screened,
since either can hold the e.
A command-position glob that can resolve to sed is treated as sed. The
auto gate already asks about any unresolved command glob; this is for the
blocklist, which did not know the name.
Inside double quotes a backslash makes the next character literal, so
sed "s/\$(CC)/gcc/" runs no substitution and should never have asked. The
quote scanner now reports an escaped character under its own state.
Left open: on Windows the blocklist lexer keeps quoting in its tokens, so
a multiline program held in a variable resolves there but not to a name
the blocklist reads. The prompt still fires on every platform.
* Ask when the sed program is not a literal we can read
Two from review, and the second one changes the default rather than adding
another case.
sed --sandbox and --posix were being read as disabling e for the whole
invocation. They disable exactly the scripts written after them: sed
compiles each -e as that option is parsed, and the positional script only
after the option list, so sed -e '1e CMD' input --sandbox runs the payload
with no POSIXLY_CORRECT needed. Suppression is now positional. Reading
POSIXLY_CORRECT out of the command text was considered and dropped as
unsound, since export or an outer bash -c puts it somewhere the text does
not show.
A program built by a parameter transformation was invisible: only bare
$NAME and ${NAME} were resolved, so ${p#x } passed through untouched. Rather
than add operators one at a time, a program that still holds a live
expansion after resolution is treated as unreadable and asks. Unhandled
expansion forms are now safe by default instead of silent, which also
closes ${p%Z}, array elements, printf -v, read, and p=$(...) whose binding
shlex had been truncating to a bare $.
Arithmetic is collapsed rather than exempted. It can only ever evaluate to
an integer, so it cannot spell a sed command, but leaving it as written let
"$((c+1))e CMD" read as an append-text command that swallowed the payload.
The cost is that a double-quoted program holding an unassigned variable now
asks: sed "s/$OLD/$NEW/g" f. Measured at 24 of 169 realistic invocations,
all of that one shape. Exempting it would trade enumerating expansion
operators for enumerating assignment forms, and four of the bypasses above
sit outside the assignment pattern, so the blanket rule stays.
Left open: -f prog.sed is still unscreened, since the program is in a file.
* Decide where a sed scan stops by context, not by token text
Four from review, two of them exploiting fixes from earlier rounds.
Stopping the sed walk at a + or ; token read the text after shlex had
already removed its quoting, so a quoted file operand looked exactly like
a find terminator and the scan gave up before the -e that followed. sed
still compiles that -e, because getopt permutes. Termination is now decided
by token index: a separator counts only if it was unquoted, and + or ; only
while a find or fd exec action is open, which is the only place quoting
does not matter. The same shape works with & | ( ) and }, so all of them
are covered.
The assignment map kept the first binding for a name, but the shell uses
the most recent one before the command. Bindings are now ordered and only
those preceding a given sed are folded in, with a later one replacing an
earlier. A value that is not itself literal clears the name rather than
leaving the older literal standing, which would otherwise have dressed an
unread program up as a safe one.
Exhausting the wrapper budget under find -exec returned the same answer as
finding no child at all, so a long enough chain of wrappers hid whatever
followed. It now reports overflow and blocks the chain word. This was
hiding more than sed: the same shape hid a plain rm.
fd spells its exec flags -x, -X, --exec and --exec-batch, none of which
were routed into the nested scan. They are now, but only while a find or
fd word is in scope and no action is already open, so a -x that belongs to
a child command is left alone.
Prompt rate is unchanged at 45 of 169 realistic invocations; this round
adds no new prompts.
* Drop the words the shell removes before a command runs
Two from review, both verified to run for real.
A redirection is performed by the shell and never reaches the command, but
the words stayed in the token list and the first of them was taken for
sed's positional script, so the real one behind it was never read.
`sed </dev/null '1e touch MARKER' input` creates the file, and so do the
`>`, `2>`, `2>&1`, `&>`, `>|` and here-string spellings. Redirections are
now recognised as spans and skipped: the target may be glued on, be the
next word, or sit one further along when a punctuation character splits
the operator. A skip is honoured only where sed would take the word as an
argument, so a pending -e/-f/-l value is still read.
The same words also hid a command outright. `> out.txt rm -rf victim` and
`2>&1 rm -rf victim` both really delete, because the redirection target
was read as the command word and the rm behind it landed in argument
position, where the always-on blocklist does not look.
shlex emits a RUN of punctuation characters as one token, so bash's `|&`
matched no separator and a sed scan ran on into the NEXT command, taking
its `-e safe` for the real script and dropping the payload. Any token
built only from those characters now ends an invocation, and a quoted one
is excluded the same way a quoted `';'` already was.
The third item from that review, `-l N` eating the script as its length
operand, was already closed in 9a5cfddb.
Prompt rate is unchanged at 45 of 169 realistic invocations; this round
adds no new prompts.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Read a sed program from what the shell really hands it
Five from an independent review pass, each verified by executing it.
sed joins its -e and -f sources with newlines, but a source boundary also
closes a line continuation open across it. Reading every -e as one
uninterrupted text let an unreadable -f in the middle hide the piece
behind it: `sed -e '1a\' -f /dev/null -e 'e CMD' input` runs CMD while the
same line without the -f only appends text.
A program flag ahead of the positional script makes that word an input
file. One behind it does so only while getopt permutes, and
POSIXLY_CORRECT turns permutation off from outside the command text, so
the positional is now read as a script as well. The suppression that a
flag written first performs is unchanged.
xargs builds the argv of the command behind it, appending what it reads on
stdin and substituting it into an -I placeholder, so the program need not
be in the text at all. A sed whose program is empty or is only the
placeholder is failed closed. The ordinary idioms are untouched: their
program is present and the placeholder stands where the file goes.
Only a word that really changes shell state rebinds a program held in a
variable. An assignment-shaped argument, one inside a subshell and one
used as a command's environment prefix all leave the variable alone, and
recording them replaced a payload with a value bash never assigned. A
conditional assignment after && or || may or may not run, so it clears the
name rather than being guessed at.
Exec-flag forwarding now starts only at a command word. Any token spelled
fd or find used to turn it on, so a -x or -exec in the text after one was
read as an exec flag and its neighbour hard-blocked; `echo fd -x rm` and
`grep fd -x rm file` were refused outright. A command-position glob bash
resolves to find is still recognised.
Prompt rate is unchanged at 45 of 169 realistic invocations.
* Judge a sed program against what getopt and find really do
Seven from review, each verified by executing it.
A redirection is removed wherever it stands, including where an option
value goes, so `sed -n -e >out '1e CMD' input` takes the word behind it as
the script. The skip is now honoured ahead of a pending value rather than
after it. The target of a detached redirection may itself look like an
option or a quoted operator, and the shell hands it to open() either way,
so `sed > --sandbox '1e CMD' input` and its `> ';'` twin no longer leave
that word standing as a sed flag or script. Only a bare operator is
refused, which is a malformed line.
A program flag written behind the positional script and the positional
itself are ALTERNATIVES, since permutation decides which sed compiles and
nothing in the text settles it. They were joined into one program, where an
unterminated command in the one swallowed the other: `-e safe` is an `s`
with delimiter `a` and no closing one, and it ate the payload behind it.
Each source is now scanned on its own.
find closes its batched form at `{} +` only, so a `+` anywhere else is an
ordinary argument it hands the child. Stopping at one threw away the script
behind it. The `;` spellings need no such test: a quoted `';'` and an
escaped `\;` reach find as the same word and it stops at either, which the
`;` twin of that line confirms by not executing.
An `-f` naming a stream (`-`, /dev/stdin, /dev/fd/N) takes the script off
stdin, which the same command line may well supply through a heredoc. That
is ignorance rather than safety, so the sed fails closed. A named program
file is unreadable in a different way and is unchanged.
bash expands the program word before sed is started, so in a directory
holding a suitably named file `sed *` runs whatever that file contains.
A program word carrying an unexpanded glob now fails closed. Quoted
programs expand nothing and a glob among the file operands is not the
program, so ordinary work is untouched.
Prompt rate is unchanged at 45 of 169 realistic invocations.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep command position and quoting intact through the sed scan
Six from review, two of them regressions the previous commit introduced.
Scoping exec-flag forwarding to a command word lost that position at a
shell keyword and across a wrapper's own operands, so `if true; then find
. -exec rm ...` and the `env -u FOO find ...` and `timeout 5 find ...`
shapes stopped blocking rm entirely. Keywords now keep the position and
wrapper options and their operands are stepped over, the way the command
walk already does.
Reading any operator-shaped token as a separator did the opposite: a
QUOTED one is data the command receives, so `printf '%s' '|&' rm` and
`grep '|&' rm file` were refused although they run nothing. The walk now
applies the same quoted-index exclusion the layout pass does, which also
clears the older `printf '%s' ';' rm` false positive.
ANSI-C decoding flattened the word's whitespace, and a sed program ends
its comment at exactly the newline that flattening destroyed. The decoded
text is re-quoted instead, keeping the spaces and the `#` around it, with
the newline standing as a mark so it stays data for whatever command
receives it rather than a place a new one begins.
An assignment inside a function body has not run and may never run, so it
is no longer recorded as the current value; the name is cleared instead,
which is right whether or not the function is later called.
An `-f` taking a process substitution is a generated /dev/fd/N script, and
the lexer ends the invocation at the `(` before the operand is read at
all. A still-pending program operand now fails the sed closed.
Live expansions were compared against the raw command spelling while the
sed program carried the post-lex one, so an escaped expansion read as
already resolved. Both sides are keyed without their escaping, which can
only make a spelling match and so errs closed.
Prompt rate is unchanged at 45 of 169 realistic invocations.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Read the sed program from the word the shell actually passes
Six from review, four of them bypasses and two false alarms.
find rewrites `{}` with the pathname it found before the child ever starts,
so a sed whose whole program is that placeholder was never read. Nested
under xargs it really runs whatever a suitably named file contains. A `{}`
among the file operands, which is the ordinary idiom, is not the program
and is untouched.
A quoted redirection is a word the command receives rather than something
the shell performs, and it was being removed either way, so a `-f` script
file named `>prog` disappeared and took the `-e` behind it out of view.
Quoting is now read from the operator the token opens with, which leaves
`2>'/dev/null'` a redirection with a quoted target.
An apostrophe in an ANSI-C word sent it down the flattening path, which
destroys the newline a sed comment ends at. The apostrophe is re-quoted
the way a shell does it instead.
fd takes the command attached to its short exec option, and only the exact
`-x` and `-X` spellings opened an action, so `-xrm` reached neither layer.
Conversely nothing behind a bare `--` is an option at all, and reading one
there refused `fd -- -x rm`, which merely lists a file.
The set of live expansions covers the whole command, so matching a sed
program against it by text alone attributed an expansion another command
performs to a program that only spells the same thing. Which occurrence it
was decides it now, and single quoting keeps its meaning while double
quoting does not.
Prompt rate is unchanged at 45 of 169 realistic invocations.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten the comments this PR added
Every comment kept says why a rule exists and, where the reason is a
real tool behaviour, names the one command that proves it. What went is
narration of the code, the history of how each fix evolved, and the same
mechanism re-explained at each site that uses it: it is stated once at
the definition now and referred to from there.
Docstrings on the private helpers give what they return and the one fact
that is not obvious; the worked examples they carried are in the tests,
which already run them. The longest block is 8 lines, from 19.
229 lines off the diff. No code changed.
---------
Co-authored-by: danielhanchen <unslothai@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
findCodeBlockRegions scanned every region found so far for each inline code
match, and accepted inline spans were appended to the same array, making it
quadratic in the number of inline spans. preprocessLaTeX runs on the full
message text every animation frame while streaming and calls it twice.
Fenced and inline matches are both ascending and non-overlapping, so walk the
fenced list with a cursor instead. Only fenced regions can contain an inline
span, so previously accepted inline regions never needed checking.
34,670 chars with 2,100 inline spans: 5.51ms per call to 0.12ms.
Co-authored-by: shimmyshimmer <info@unsloth.ai>
test_shipping_code_names_an_encoding is red on main. #7373 added
sync_marker_llama_backend, whose read_text/write_text pair does not name
an encoding, so both fall back to locale.getencoding():
AssertionError: 2 text read/write call sites in shipping code let the
operator's locale decide the encoding, so they crash or silently
produce mojibake on Windows. Pass encoding = "utf-8":
['studio/install_llama_prebuilt.py:5656: write_text()',
'studio/install_llama_prebuilt.py:5647: read_text()']
Reproduced on a clean checkout of main at 7917c7828: 1 failed, 7 passed.
That guard landed in #7486 a few commits earlier, so the rule predates
these call sites; nothing about the Vulkan work is wrong beyond the
missing kwarg. The create path that writes the same file, 26 lines above
at 5621, already passes encoding = "utf-8", so main is also internally
inconsistent about one file: written as utf-8, read back under the
operator locale.
Scope, stated honestly: json.dumps defaults to ensure_ascii = True, so
the marker this module writes is pure ASCII and round-trips under cp1252
as well as utf-8. The exposure is a marker produced or edited by
something else. A decode failure on the read would not even surface,
because UnicodeDecodeError subclasses ValueError and the surrounding
except (OSError, ValueError) swallows it into the early return, leaving
the backend silently unsynced. So this restores a green suite and makes
the file self-consistent rather than fixing a live crash.
Verified: tests/test_runtime_text_encoding.py 1 failed / 7 passed before,
8 passed after; tests/test_source_read_encoding.py still passes.
* Studio: keep grouped Python scripts visible and save them natively
* Studio: render the executed Python script outside the card collapsible
Ungrouping the aggregate tool group was not enough on its own. Each Python
card still mounts with defaultOpen={isRunning}, so on a reopened turn the
script and its Copy/Download controls stayed hidden behind the card's own
chevron and the reported issue persisted.
Render ToolCodeCell outside ToolFallbackContent for Python, restoring the
behaviour from #7240 that #7455 folded back inside when it unified the code
cell. Status, output and images still collapse. Terminal keeps its command
inside the collapsible: a one-line command is not the artifact a user reopens
a thread to retrieve, a script is.
Verified against a running Studio: reopening a persisted turn with two
adjacent Python calls now shows both scripts and both Download controls with
no clicks, and Download still saves byte-exact script.py.
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
test_tensor_abort_cache_invalidated_on_binary_mtime_change bumped mtime by a
single nanosecond. NTFS stores timestamps as 64-bit FILETIME values in 100ns
ticks, so on Windows that bump rounds away, st_mtime_ns reads back unchanged,
the cache key is identical and the stale abort is inherited, and the assertion
sees True where it wants False.
1ms is still a same-second, sub-second change and is exactly representable, so
the case the test exists to cover actually runs. Skip when the filesystem cannot
record any sub-second change at all rather than asserting product behaviour the
platform cannot exercise.
Not caught before because both jobs in studio-backend-ci.yml are
runs-on: ubuntu-latest, so the studio backend tests only ever run on Linux.
sync_marker_llama_backend read and wrote UNSLOTH_PREBUILT_INFO.json without an
encoding, so the operator locale decided it and the file could crash or turn to
mojibake on Windows. The sibling helper 15 lines above already passes
encoding = "utf-8"; match it.
This is what test_shipping_code_names_an_encoding has been failing on, and since
that test is a repo-wide AST scan it turns Repo tests (CPU) red on every PR that
touches studio/.
* Studio: match llama.cpp SWA cache sizing
* Studio: account for batch-capped SWA ubatch
* Studio: match llama.cpp KV stream padding
* Match llama.cpp batch and FA-off cache sizing
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Skip unusable compact SWA slot saves
* Align KV planning with launched server
* Match cache type casing and narrow the compact SWA slot-save skip
The launcher tested the requested cache type case-sensitively while the budget
lowercases it via _planned_main_cache_types, so a Q8_0 request emitted no
--cache-type flag and llama.cpp ran f16 while the estimate priced q8_0 (1.01 GiB
under-reserved on a 27B SWA model at ctx 32768 with 4 slots).
The compact SWA slot-save skip keyed on the sliding window alone, but the
estimator's SWA path also requires key/value length. phi3 GGUFs report a window
without those dimensions and llama.cpp runs them non-SWA, so their slots restore
fine and were being skipped.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: preserve durations across reasoning blocks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep a reasoning group's timer running when it reopens
A rendered reasoning group can be closed and then reopened: parseAssistantContent
coalesces adjacent reasoning parts, so a provider that emits each block as a
complete <think>...</think> chunk lands several blocks in one group. The tracker
wrote a group's duration once and never revisited it, so such a group froze at
its first close and displayed 0 seconds.
Measure from the first time an index becomes visible rather than from the last
startGroup, and reopen a closed group while its reasoning text is still growing.
Gating on growth is what stops the timer running on into the answer. A duration
supplied by the server is now recorded as authoritative so local timing cannot
overwrite it.
Also fill indices that a single delta skips. startGroup(n) could jump past
earlier indices and leave array holes, which JSON.stringify persists as null; a
skipped group became visible and closed inside the same chunk, so it gets a
measured zero instead.
Test discovery now globs tests/, so a second test file cannot be silently
skipped by CI, and tsconfig.test.json puts tests/ under typecheck for the first
time.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* feat(studio): run chats in parallel in the Chat tab
New Chat used to cancel whatever the current conversation was generating.
It now leaves it running, like switching to the Train or Export tab: the
sidebar shows which chats are still going, and Stop is per conversation.
Plain `unsloth studio` launched llama-server with one decode slot, so the
admission queue serialised every chat regardless of what the UI did. Both
entry points now default to the same slot count as `unsloth studio run`.
A model swap still ends every running chat, since they all decode on one
llama-server. /load and /unload now refuse with 409 and name those chats
unless the caller passes force_cancel_active, and the UI asks first.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): scope the composer tool badge to its own conversation
The green "Running Python: ..." badge above the composer read a single
global store value, so one chat's tool call showed above every other
chat's composer, including a brand-new empty one. Its elapsed counter
also restarted at 0 on every thread switch, and a run ending anywhere
cleared the badge everywhere.
Key the status by thread and store the moment it started, so each
conversation shows only its own tool call and the counter resumes rather
than restarts. Also adds a test that every conversation gets its own
tool sandbox directory, which parallel tool calls depend on.
* Fix stalled tool calls while awaiting approval for PR #7455
Three problems, all from the approval prompt behaving as though only one
chat could ever run.
Arguments were not streamed for a gated call, so the chat stayed blank for
as long as the model took to write the payload, which for a large file is
minutes. Nothing runs before the decision either way, and the code is what
is being approved, so python and terminal now stream their card while
gated. render_html stays suppressed: its card renders the payload.
The status read "Running ..." with a climbing timer while the call had not
started. It now reports that it is waiting for approval, then switches to
running once allowed.
The admission lease was held across the wait, so four unanswered prompts
held all four decode slots and no other chat could start while llama-server
sat idle. A parked run keeps its lease but no longer counts against
capacity.
Measured with four prompts left open: every gated call streamed its code,
none reported running, and a fresh chat answered in 0.4s where it
previously waited 290s and never did.
* Fix duplicated and truncated tool cards for PR #7455
A gated tool call rendered two cards: the provisional one that streams the
arguments, plus a second one keyed by the approval id. Only the second ever
got its tool_end, so the first spun "Running" for the rest of the chat.
Reuse the open part when the approval prompt arrives.
The terminal card also showed nothing but a 60-char trigger label, so a long
heredoc read as no progress at all. It now renders the command the same way
the Python card renders its script, and neither is capped at 10k chars.
Both cells moved inside the collapsible, so one chevron hides the code with
the output and Copy / Download exist only while the card is open. A card
parked on the prompt says so instead of counting up "Running".
* Fix review findings on the parallel-chat gate for PR #7455
Backend:
- /unload rechecks active generations under the lifecycle gate, like /load,
and lets its 409 through the catch-all instead of rewriting it as a 500.
- /load gates only once _load_model_impl has decided this is a real reload,
so an Apply on the already-loaded model no longer refuses, and the retry it
asks for no longer cancels every chat before returning already_loaded.
- The direct /v1/responses stream registers in the cancel registry, so a
non-forced unload can no longer tear llama-server down under it.
- run_server defaults to the same slot count as the CLI. colab.py calls it
without the argument, so Colab was still serialising every chat.
Frontend:
- Cancelling a backgrounded chat aborts its own request rather than only
posting a cancel id, which is the only thing that ends an external-provider
or audio run.
- The model-swap dialog counts local runs only, and falls back to the backend
when this tab's map is empty, so a reload or a second tab still gets asked.
- Context usage and the diffusion canvas are scoped to the chat that produced
them; a compare row reads activity from its member threads.
Tests:
- The extracted-source cancel harnesses supply the active-generations module,
which the tracked-cancel class now depends on.
* Fix the swap confirmation scope and cancel timing for PR #7455
A forced load cancelled every chat before the model identifier, GPU selection,
training coexistence and download checks had run, so a load that then failed
those checks stopped the chats and replaced nothing. The refusal still happens
early, but the destructive cancel now sits immediately before the teardown it
is paying for, and rechecks under the gate like /unload does.
The swap dialog only reconciled with the backend when this tab looked idle, so
one local chat was enough to hide a second tab's runs. Confirming then sent
force_cancel_active, which cancels every backend run, including the ones the
dialog never mentioned. The backend snapshot is now merged in every time, so
the dialog names what will actually stop. External-provider runs are never
registered there, so the union stays local-only.
Also drops the active-generations docstring claim about restoring sidebar
spinners, which nothing consumes.
* Defer destructive cancels and track every local stream for PR #7455
/unload cancelled the running chats before it had resolved that it unloads
anything. A stale model_path, which a second tab produces routinely, killed
every chat and then no-opped, leaving the resident model up. It now refuses
early and cancels only at each teardown, matching /load.
The swap dialog also stopped every chat locally the moment the user confirmed,
which threw away the two-phase backend behaviour: a load that then failed
identifier resolution, GPU validation or the training guard had already
truncated the replies. The backend now owns the cancel.
Three local streams decoded on llama-server without registering, so a
non-forced unload counted zero generations and tore the server down mid
response: /v1/completions streaming, and the plain and server-tool Anthropic
streams, the first of which is the default /v1/messages path. Note this makes
a non-forced load return 409 during those runs rather than draining quietly,
the same trade the /v1/responses fix made.
The safetensors tool loop still announced a gated call as running while it
waited on a human; only the GGUF loop had been fixed. A source-level parity
test now pins both.
Also drops stopAllChatThreads, which has no callers left.
* Studio: close three load/unload gate races found in review
Re-check the in-flight load guard after the stop-running-chats confirm.
The confirm always GETs active-generations before its zero-running
early-out, so the guard no longer sits atomically ahead of the
reservation and two picks in that window both reached performLoad over
the same refs. ejectModel had the same shape and gets the same re-check.
Reject a sidecar swap immediately before the forced cancel in both load
branches. The previous check was back at the top of preflight, so an
install reserving during identifier resolution, the tier probe, the
training guard or the download check made the post-drain recheck 409 a
load whose chats had already been stopped.
Enter the Anthropic passthrough's cancel tracker inside its body
generator. It was entered eagerly and returned through
_sse_streaming_response, which sets no unstarted_cleanup, so a response
whose body never started left the run registered forever and 409'd every
later non-forced load and unload.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments across the files this PR touches
Tightens the comments and doc blocks in the backend, CLI, tests and frontend
files changed by this PR: collapses multi-line explanations to a single line
where they still read clearly, and drops the ones the code already says.
No code changes, verified by an AST comparison against the previous commit.
* Studio: defer the destructive cancel and close two gate gaps
Move the forced cancel behind every check that can still reject a swap.
The drain now runs first with the runs it is about to cancel discounted,
so it waits only for inference the cancel cannot end, then the sidecar
check decides, then the cancel fires, then a second drain lets those runs
unwind before teardown. A sidecar install reserving during the drain no
longer 409s a load whose chats have already been stopped.
Track the non-streaming /v1/completions proxy. It was the last local
decode path missing from active_generations, so an unload, which runs no
drain, tore llama-server down under it and force_cancel_active could not
signal it. It now uses the same tracked cancel event and dedicated client
as the OpenAI pass-through.
Skip the client's preliminary unload while chats are generating and let
/load evict at its own post-preflight point instead. Forwarding
force_cancel_active there truncated replies before identifier
resolution, the GPU and training guards and the download check had run.
Keep per-thread context usage so returning to a chat whose background run
finished restores its bar instead of leaving it blank until the next turn.
Make the running-flag clear run-specific. Every run without a resolved
thread id shares the "__default" key, so concurrent compare panes could
clear each other's flag and strand a live stop handle.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: register the embeddings proxy with the swap gate
/v1/embeddings proxied straight through the pooled client with no tracked
cancel event, so it never appeared in active_generations. /unload runs no
idle drain, so a concurrent non-forced unload counted zero generations and
killed llama-server mid-request, and force_cancel_active had no event to
signal. Mirrors the completions proxy: tracked event, dedicated unpooled
client closed by a cancel/disconnect watcher, unregister in a nested
finally so a close failure cannot leave a phantom generation behind.
* Trim comments on the newest changes in this PR
Comments only, no code changes: shorten the ones added by the load-gate
ordering, embeddings and per-thread usage work down to the same density as
the rest of the diff.
* Studio: register the legacy generate stream with the swap gate
/generate/stream built a cancel event but never entered the tracker, so it
was invisible to active_generations. Being in the keep-warm middleware's
inference suffixes only covers /load, which drains; /unload does not, so a
non-forced unload passed the 409 gate and then blocked on the standard
backend's generation lock, and a forced swap had no event to signal.
Registered inside the body generator under a nested finally so a teardown
failure cannot skip the unregister.
The AST contract test asserted the cleanup finally by overwriting its flag
per Try node, so a nested try made the last one win. Accumulate instead,
which is what the existence claim meant.
* Studio: three more swap-gate gaps found in review
Register /audio/generate with the gate. TTS holds the model for the whole
request and /unload runs no drain, so unregistered a non-forced swap counted
zero generations and tore the model down mid-generation; the orchestrator
path only waits 15s for the generation lock, which real TTS exceeds. No
cancel keys: no backend takes a cancel_event for audio, so the event has no
observer and a forced swap still cannot interrupt audio already in flight.
Thread the tracked cancel event into the /v1/responses admission wait. It
was the only admission caller passing None, so a queued run could not be
reached by cancel_all() and a plain /inference/cancel could not stop it at
all. Same omission fixed at the upstream send there and on /v1/completions.
Let an unforced unload of a stale model path reach the no-op check. Before
this PR that request returned 200 and did nothing; the new gate refused it
with 409 for a request that reaches no teardown branch. Gate both refusal
passes on the disjunction of the route's own teardown conditions, including
not is_loaded, so a mid-load GGUF still refuses.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: register the remaining non-streaming decode paths
stream defaults to false on all three of these, so they are the ordinary
shape of their routes, and each holds a local backend for the whole
request. /unload runs no idle drain, so with no registry entry a non-forced
swap counted zero generations and tore the backend down mid-request instead
of returning 409, and a forced one had no event to signal.
Non-streaming /v1/messages: all three helpers ran with an empty registry,
since only the streaming siblings were tracked. Registered at the call site
because the pass-through takes no cancel_event of its own, and with no
cancel keys, matching those siblings.
Non-streaming standard chat and audio-input chat: the trackers in this route
sit inside their `if payload.stream:` arms, so neither else branch was
covered. The GGUF sibling already registers its own non-streaming branch.
Each exit is in a finally on the branch's existing try, so the except arms
are covered too: a leaked entry 409s every later swap until restart.
* Studio: tighten the swap-gate comments
Comment-only pass over the newest swap-gate registrations: collapse the multi-line rationales in /unload, the legacy generate stream, audio generation and the non-streaming chat branches, and the matching test preambles, to the shortest form that still carries the reason. No code changes.
* Studio: stop the reselect dialog promising a stop that never happens
Picking an external provider leaves the local model resident and stops the
status poll mirroring it, so reselecting that model showed the stop-chats
dialog, and /load then answered already_loaded ahead of its cancel hook.
Confirmed with the live backend: the same pick with force_cancel_active set
still returned already_loaded and the chat kept streaming. Not stopping
those chats is right, since the load never interrupts them, so remove the
prompt rather than honour it. Blanket-skipping is unsafe, because the same
id and variant with one sampling setting changed is a real reload and 409s,
so the branch only fires when a status fetch confirms the resident
checkpoint and variant match, and then adopts it without calling /load.
Redact native model paths from the active-generations response. Registering
/generate/stream recorded backend.active_model_name verbatim, which is an
absolute path for a native local model, and this route is the only place
that serialises it. Redacting at the response covers every tracker rather
than the one that surfaced it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep hydrated context usage in the per-thread map
The history loader restores a saved conversation's usage through
setContextUsage only, and it runs once per mount, so switching away and
back left the bar blank for a hydrated chat even after the per-thread map
landed. setContextUsage now writes the value through to the visible
thread's own entry and clears that entry when passed null, which covers
both hydration call sites and any future writer.
* Studio: unblock load cancellation and share unresolved thread keys
Run the two stop-loading fast paths ahead of the unload route's pre-gate
refusal. _unload_may_evict returns True for exactly the model being
cancelled, so the refusal was blocking the branch that cancels a load which
has replaced nothing and can interrupt no chat. The client made that
unrecoverable: cancelLoading sends the unload without force, drops the
result, and its abort never reaches /load, which takes no signal, so the
load ran on and could later cancel those chats and swap the model. Nothing
else is exempted; an unload that would tear down a serving model matches
neither fast path and still 409s. The comment claiming the client lets that
409 surface is corrected, since it discards it.
Hold every owner behind a shared thread key. Runs with no resolved thread id
share "__default" (concurrent compare panes, since startCompare clears
activeThreadId), so a single owner slot let a second run replace the first's
token and then delete the shared entry while it was still generating, and
the server-cancel map lost the older handle the same way. Both now hold a
list, the running and local flags survive until the last owner clears, and
stopChatThread stops every handle under the key.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: carry a confirmed swap into the sidecar install, key restored usage by thread
Picking a model that needs a newer transformers while chats generate raised the
"stop N chats" prompt, but the answer never reached the install that runs before
the load: /install-latest-transformers refused on those same chats and took no
force flag, so Retry hit the same 409 and nothing in the flow stopped them.
Carry force_cancel_active through the consent dialog into the installer. Only
the pre-gate fast path is skipped: the recheck under the lifecycle gate still
has to pass, so an unconfirmed caller is refused as before. The cancel runs last
inside the gate, after every check that can still reject the install, and the
drain behind it is bounded since it holds the gate and the sidecar reservation.
Also key restored context usage by the thread the loader read. history.load()
captures remoteId before two awaited round trips, so a switch inside that window
filed one thread's usage under another and setActiveThreadId kept re-applying it.
Preserve sibling owners when a run key is cleared without an owner: the image
rejection gate now uses its own token, and the reducer leaves owned runs alone.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: bound the post-cancel drains, and make cancellation reach the paths that ignored it
A forced swap cancels the chats it interrupts, then waits for them to unwind.
That wait had no deadline while holding the lifecycle gate, and TTS on the
subprocess backend observes no cancel event at all, so one audio generation
could pin every load, unload and new request for its whole duration. Bound both
post-cancel drains. Pre-cancel drains stay unbounded: the swap can still be
refused there, so shortening them would weaken what they protect.
/unload had the opposite problem and no drain at all, cancelling and tearing
down on the next line, which turned a clean stream end into a dropped
connection. Give it the same bounded wait, gated on the cancel having cancelled
something so an idle Eject pays nothing.
Make the cancel actually land where it can. GGUF TTS now takes a cancel_event
and a watcher closes its client to break the blocking POST. The Anthropic
non-streaming pass-through did the same thing the completions and embeddings
paths used to: register with the gate, then run both POSTs on the pooled client
that cannot be closed. It now uses a per-request client like they do.
Also: park and unpark the admission queue the reservation actually holds, since
queues are keyed by base_url and a reload mints a new port; key tool output by
remoteId on both sides, so the first turn of a New Chat stops writing under one
key and reading another; and give tool status a run owner, so a finishing run
cannot blank the badge a concurrent one is still showing.
Clamp --parallel to 1 on a llama-server without --kv-unified. The new default of
4 would otherwise split -c four ways on such a build, quartering the context
window for a feature it cannot serve.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: scope a chat's Stop to its own generation, and clear the way before a confirmed install
Safetensors generation is serialized on _gen_lock and the worker has a single
cancel event, so a chat still queued on that lock owns no generation. Its Stop
handler called reset_generation_state() anyway, which set the shared event and
ended whichever conversation was actually running. Parallel chats is what makes
that reachable.
_generate_inner now records its cancel_event as the current holder once it takes
the lock, and reset_generation_state drops a reset from anyone else. Every route
call site passes its own request event. A reset with no event stays global, so
unload and model switch cannot leave a generation alive, and a reset while
nothing runs still resets, so an error path before generation is not a no-op.
The other two backends take the argument too, or the standard one raises
TypeError on every cancel.
The sidecar install had the mirror of the /load ordering problem: it cancelled
the chats first and drained second, so an unrelated counted request the cancel
cannot reach (a count_tokens, say) was still there for the recheck, which then
refused an install that had already stopped every chat for nothing. Drain the
unreachable remainder first, discounting the registered chats, then cancel.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: close the windows the previous round's fixes left open
Three follow-ups, two of them holes in the fixes just before them.
The worker claim went in after _send_cmd, so the command was already running
unclaimed and a queued chat's Stop in that window still reset it. Claim first,
with the send inside the same try, so a failed send releases it too.
Tool status kept one entry per key with an owner. That stops a foreign clear but
not an overwrite: under the shared unresolved-thread key the second run replaced
the first's entry, and its own clear then removed the only one while the first
tool was still running. Keep per-run entries and render the newest.
/unload gated its drain on having cancelled something, so a request that passed
the keep-warm middleware but had not reached its tracker yet was invisible to it
and the teardown landed on an already-admitted request. Drain on the middleware
count instead, which covers that window as well as the cancelled runs, then
re-cancel whatever registered while waiting. Bounded, not a refusal: an unload is
deliberate, and on expiry it proceeds exactly as before.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim the parallel-chats comments to their reasons
Compress the multi-line rationales added by this branch into shorter forms and drop
restatements of the code below them. The reasons behind the drain bounds, the deferred
cancel, the per-request generation ownership and the thread-scoped tool and usage keys
are kept, just said in fewer lines.
* Studio: own the worker per generation, and make a resumed chat requeue for its slot
Ownership was a single lock holder, so dispatched runs (compare mode bypasses
_gen_lock by design) never claimed it and the guard fell straight through to the
global reset: a Stop on one of them ended its siblings. Track the generations
actually running instead, claimed before the send and released in the same
finally on both paths. A reset still proceeds when nothing is running, so an
error path ahead of generation is not swallowed.
park() hands the freed slot to a waiter, so a chat resuming from a tool approval
could take it back while that waiter was still decoding, putting two holders on
a one-slot server and sending the resumed tool loop past the admission limit.
unpark_async waits for room; the plain unpark stays for a holder tearing down,
which will not decode again.
Audio only observed its cancel event on a forced swap. An explicit Stop just
aborts the fetch, and this route has no cancel id, so llama-server ran on to the
request timeout after the chat reported it stopped. Watch the disconnect.
Also read tool status by remoteId, matching the key the adapter writes and the
fix already made for tool output, and stop an unresolved run from writing its
usage into whichever conversation the user moved to.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: let only the generation the worker is running speak for it, and hold a slot for a resuming chat
The ownership list recorded admission, but the subprocess runs generations one
at a time, so a dispatched request queued behind another counted as an owner and
its Stop signalled the shared cancel event, ending the request that was actually
running. Keep admission for release bookkeeping and gate ownership on execution
instead, promoted when the worker first answers that request. Nothing executing
still permits a reset, so an error path ahead of generation is not swallowed.
The worker has one cancel event and no per-request cancellation, so this decides
who may pull the lever rather than making the lever per-request.
A resuming chat also polled for a slot it could never see: release() grants to
the next waiter under the same lock, so later arrivals overtook an approved chat
indefinitely. A pending unpark now reserves the next slot and they queue behind
it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: cover the prefill window, and keep a first turn's tool output readable
Gating worker ownership on execution left the interval between the send and the
first response uncovered: nothing is executing then, and the empty case admitted
anyone, so a queued chat's Stop still ended the one in prefill. Split the empty
case. Nothing claimed at all still permits a reset, so an error path ahead of
generation is not swallowed; claimed but unanswered resolves to the oldest
claim, which is what a FIFO command queue is working on.
Putting both sides of the tool-output scope on remoteId left the first turn of a
New Chat writing under the unresolved scope for its whole life while the readers
recomputed the moment the autosave assigned an id, so the card blanked mid-run.
The readers now fall back to the unresolved scope, which only an unpersisted
first turn can occupy.
* Studio: order the parked approvals, and tie a worker claim to its enqueue
The reservation added for admission fairness was a bare count, so every approved
holder counted against every other: park two chats, approve both, and once the
last decoder released, nothing could ever satisfy the check again. That is a
deadlock where the problem it fixed was only unfairness. Make it a FIFO ticket
so a pending unpark blocks the ones behind it and no others.
_owns_worker reads claim order to decide which request the worker is prefilling,
which only holds if claiming and enqueuing cannot interleave. Hold one lock
across both on the dispatched and the locked path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: follow a first turn's run when its thread gets an id, and key the denoising canvas per chat
A run started before its thread existed filed every handle under "__default". Nothing
moved them once autosave assigned the real id, so the sidebar row showed no spinner and
Stop could not reach the generation, which kept holding a slot.
adoptDefaultThreadRun re-keys the run maps onto the real id from the thread adapter's
initialize(), where the id first exists; anything already filed under that id wins, since
that is a later run. The adapter captures its key once at run start, so it now resolves
the live key per use through runKeyForOwner, looking its own serverCancel up in the owner
map. Without that the migrated entries are stranded and the spinner never clears.
The denoising canvas was one global slot, so two diffusion chats overwrote each other and
the ownership tag then hid the visible preview until that thread emitted again. It is now
activeDiffusionCanvasByThreadId, written and cleared per thread, and the frame no longer
carries a threadId of its own. The bubble reads threadListItem.remoteId, dropping the dead
threadListItem.id arm: the writer tags unstable_threadId, which is exactly remoteId.
Two existing backend tests needed the same treatment. _bare_orchestrator skips __init__, so
it now sets the claim bookkeeping the worker ownership check reads. The Anthropic
passthrough gate test anchored on comment prose that a rewrap had broken; it anchors on the
code instead.
* Studio: hand the worker over cleanly between generations, and stop unresolved runs sharing each other's state
Worker ownership moved off the consumer and onto the dispatcher. Consumers read their
mailbox whenever they get around to it, so a request whose gen_done had been routed still
owned the worker while the next one ran, and a late Stop for it cancelled that one. The
dispatcher is the only place responses arrive in the order the worker produced them: it
now retires a request at its terminal response and promotes the next one, and answering a
request makes it the sole executor, since the subprocess runs one generation at a time.
reserve()'s immediate path ignored the unpark tickets that _grant_waiters_locked already
honours, so a request arriving between a slot freeing and an approved chat's next poll
took it, repeatedly. It applies the same reservation now.
Three places let concurrent first turns share state through the "__default" key. Nothing
links a run filed there to the id its thread later receives, so rather than guess, each
now declines when the key is ambiguous: adoption only re-keys a lone run, the composer
badge only claims a lone status, and the tool-output fallback only applies to a thread
that is still running. That leaves two concurrent first turns where they were before
adoption existed instead of handing one thread the other's handles.
A first turn's usage was never filed, because its key stayed null for the whole run while
autosave moved activeThreadId to the real id, so the context bar went blank after the
first reply. It resolves the adopted key like the cleanup handles do.
Cancelling a forced load left the UI with no model: the previous one stays resident until
/load's teardown, and the cancel path cleared the checkpoint without rolling back. It now
resyncs from the backend, which is right whether or not the load got that far.
The sidecar install drain is weighted 1:4 rather than halved, total unchanged. Only the
second half benefits from patience, and cutting it short refused installs whose chats had
already been stopped for nothing.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: give a first turn its real thread id before the run starts
A first turn filed every run handle under a shared unresolved key because
assistant-ui binds unstable_threadId before the thread is persisted. Two of them
overlapping there is unresolvable afterwards, and the last round's migration could
only decline rather than guess, which left neither sidebar row showing its run.
The id is available earlier than I claimed. append() already tracks
threadListItem.initialize() by the user message id, and createPersistedRunAdapter
already awaits that promise before invoking the adapter, so the thread is persisted
by the time the run begins. It was only being discarded: the tracked promise resolved
to void. It now resolves to the assigned id, and the wrapper hands it to the adapter
when assistant-ui had none. An id that is already set is never replaced, since that
would move a running chat's handles out from under the row watching them. The
existing unresolved-key guards stay as a safety net but should no longer carry weight.
The sidebar counted running thread ids rather than rows, so one compare conversation
read as two chats. It folds ids into rows through the same threadIds the row spinner
uses, and still counts a running id that matches no row.
_TrackedCancel always registered kind="chat", so an embeddings or raw completions
request appeared in the model-swap prompt as an unnamed conversation and confirming
cancelled it while calling it a chat. The non-conversation routes now pass their own
kind, and the prompt says "requests" whenever the snapshot is not all chats.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: withhold the shared worker cancel from a request the worker has left
Moving ownership to the dispatcher fixed reset_generation_state, but the token loop
signals the shared worker event directly and did not carry the same rule. A dispatched
consumer runs with mark_started off and can still be draining tokens buffered before
its gen_done was routed, so stopping it there ended whichever request the worker had
started next.
It now signals only when _owns_worker agrees, the same predicate reset_generation_state
uses. The local drain and return are unconditional, since those touch nothing but this
stream. The remaining _cancel_generation callers are deliberately global: subprocess
shutdown, the pre-load kill and unload_model.
* Studio: add the AGPL-3.0 header to the first-turn identity test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stop the dispatcher and a _gen_lock stream fighting over the response queue
Nothing stopped the dispatcher starting under a _gen_lock generation, so once compare
was opened while an ordinary chat was still streaming, both consumed _resp_queue and
whichever response the dispatcher took without a mailbox was dropped, gen_done included.
That chat truncated or hung. This PR is what makes it reachable, since navigating into
compare no longer ends the chat behind it.
Delaying the dispatcher would serialise compare behind whatever chat happens to be
streaming, so the direct readers get a mailbox instead. _direct_reader returns a reader,
a cancel drain and a release, and files the mailbox under _direct_mailboxes rather than
_mailboxes, which means "compare requests are in flight" to the unload and distributed
paths and must not count an ordinary chat.
Both directions close. The dispatcher finds the direct reader's mailbox instead of
dropping. And this reader can already be blocked on the queue when a compare request's
dispatcher starts, so a response that is not ours goes to its own mailbox rather than
being consumed, which would have corrupted the chat and hung the pane. All three
_gen_lock readers use it, and the cancel drain goes through it too.
The sidebar's return target still picked a raw pane id while the count grouped by row,
and /chat addresses compare with `compare`, not `thread`. It resolves through the same
items now, so a running compare row returns to its pair.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep worker ownership honest across audio, API traffic and a replaced worker
The audio-input send got a mailbox last round but stayed unclaimed, so a compare request
queued behind it looked like the oldest owner and stopping that queued request signalled
the shared event into the audio chat. It claims under the send lock and releases in the
finally, like _generate_inner.
Ownership is keyed on cancel-event identity with nothing tying it to a worker generation,
so a consumer still blocked on its mailbox when the process was replaced stayed recorded
as the executor, and a generation on the fresh worker could not be stopped.
_shutdown_subprocess clears that state once the process is confirmed dead, mailboxes
included: nothing routes to them again, and a stale one reads as compare activity to the
unload path. Not on the survived-SIGKILL path, which keeps its handle on purpose.
The four public /v1/messages trackers were registering as chats. The distinction is a
Studio thread, not the protocol, and those branches already say "No thread_id: public API
surface" while the Studio path passes payload.thread_id separately. They carry their own
kind now, so the swap prompt stops calling an external request a chat.
The swap confirmation still counted raw pane ids, so a compare conversation asked to stop
two chats and listed its title twice. It folds panes onto pairId and lowers the count by
what it collapsed, leaving a first turn the backend can count but not name.
Deep Research set runningByThreadId but registered no server-cancel handle, and that map
is how Stop, archive and delete reach a thread that is no longer active. Leaving the
outgoing thread running is this PR's doing, so the run was left unreachable while its
supervisor kept working against a conversation the user could delete.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten the parallel-chats comments
* Studio: replay a Deep Research stop that arrived before the run existed
The handle is registered before createResearchRun resolves because the thread can be
stopped while that request is in flight, but it had no id to act on and dropped the stop.
The supervisor then followed a run the user had already stopped, archived or deleted.
It latches instead: a stop with no id yet sets a flag, and the adapter replays it against
the id the moment creation returns rather than starting to follow.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix worker ownership on a raced reroute, and the stop-chats prompt
Four review findings on the parallel-chats work, all reproduced first.
- _direct_reader hands a foreign response to its own mailbox, but skipped the
ownership move the dispatcher makes. A _gen_lock reader already blocked on
resp_queue can beat the compare dispatcher to that request's first response,
and the compare consumer opts out of marking, so nothing promoted it: the
direct request stayed the recorded executor, its late reset cancelled the
compare generation, and the compare chat's own Stop was ignored.
- A chat stopped while queued on _gen_lock was still claimed and sent once the
lock freed. Cancellation is only checked on a token, so a long prefill, or a
generation reaching gen_done without one, occupied the worker after Stop.
Same hole in the audio-input path, which shares the lock.
- The stop-chats prompt counted generation handles, not conversations. One chat
holds several while a tool continuation registers its next leg before the
previous unwinds, so it offered to stop two chats and listed one title.
- Ejecting a model confirms through that dialog, which told the user
"Unloading the model reloads the model" and offered "Stop and reload".
Confirming calls /unload and leaves nothing loaded.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: name the TTS run's thread so the stop prompt counts it once
The audio branch registers its run locally under the thread key but sent no
thread_id, so the backend tracker filed the same generation under no thread.
The stop-chats prompt then had a named local run and an unnamed backend one and,
since e8e7594 started adding unnamed entries to the named ones, counted a single
TTS chat as two requests. The backend already reads payload.thread_id, so
sending it lines both registries up on the same run.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Remove Studio desktop auth handoff flash
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix Windows Codex temporary home path
* Fix Codex ephemeral session cleanup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden Codex temp home reclamation
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Handle llama.cpp tool schema limits
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix Claude client tools under server tool policy
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Preserve Anthropic client tool routing
* Match text editor schemas by version
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: detect an interrupted dependency install instead of launching a backend that cannot import
An installer killed part-way leaves a venv with a working CLI but without
studio.txt's dependencies. Nothing recorded that, so three separate places all
reported it healthy:
- the desktop preflight probed only `unsloth -h` (typer + rich) and a hardcoded
desktop-capabilities dict, neither of which touches studio.backend, so it
returned ManagedReady and spawned a backend that died on `import structlog`;
- setup.sh's fast path compared the installed unsloth version against PyPI,
which matches on a half-built venv because unsloth is installed early, so
`unsloth studio update` printed "up to date" and repaired nothing;
- start_managed_repair calls that update and then re-checks with the same blind
probes, so Repair reported success without fixing anything.
install_python_stack.py now clears a completion manifest before the dependency
pass and writes it only after the final step. `unsloth studio verify-install`
and desktop-capabilities' new studio_install_ok field read it, the preflight
turns a false answer into ManagedStale so auto-repair runs, and setup.sh /
setup.ps1 gain an escape hatch next to the existing anyio one.
Separately, the wheel ships studio/ and studio.backend* but declared none of
their dependencies, so `unsloth train`, `export`, `chat`, `inference` and
`studio` all ended in a rich traceback after a plain pip install. structlog is
the only hard module-level import that chain reaches once starlette's
annotation-only import moves under TYPE_CHECKING, so it becomes a core
dependency and the rest of the server stack becomes a [studio] extra mirroring
studio.txt. The CLI import sites now report missing dependencies as a sentence
with two remedies.
Fixes#4701, #5260, #7147
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Match the trimmed comments merged on the pip branch
* Put the install manifest in the preflight fingerprint for PR #7492
The capability cache keyed the venv on pyvenv.cfg, uv.lock, requirements.txt,
the interpreter and site-packages/unsloth_cli/commands/studio.py, none of which
a repair touches when it only reinstalls studio.txt. So an entry cached while
the install was healthy stayed valid after the manifest was dropped, and the
probe returned Ready on exactly the half-built venv this is meant to catch.
* Address the review findings on PR #7492
Fail the install when the completion manifest cannot be written, instead of
exiting 0 without the record every later check requires, which is a repair
loop by construction.
Compare the version of the package the manifest names, so `studio update
--package X` does not read as a permanent version change.
Read the manifest from the venv that owns it when the CLI runs outside the
managed venv, and drop the dependency verdict in that case: the walk ran
against the wrong interpreter and says nothing about that venv.
Name the import that actually failed. `unsloth train` reaches torch through
the same guard, and the studio extra does not carry it, so recommending that
extra alone left the command failing in the same place.
* Declare click, which typer stopped providing, for PR #7492
unsloth_cli/commands/start.py imports click at module scope and
unsloth_cli/__init__.py imports that module, so every unsloth command needs
it. typer carried click through 0.19 and dropped it in 0.27, and the declared
floor is typer>=0.12.0, so a fresh resolve gets no click. On the published
wheel it still arrives because huggingface_hub requires click<9,>=8.4.2, which
is luck rather than a declaration. A wheel built from this branch's
dependency list has neither, and every command dies at import.
Verified: before, `unsloth --help` on a fresh venv raised ModuleNotFoundError
for click; after, it exits 0. The drift test now covers it.
* Keep a running backend from the previous app version manageable
The manageability bump gated two unrelated things through one constant. For
the managed CLI probe 2 is right: a CLI reporting 1 cannot answer
studio_install_ok. For a RUNNING backend it is wrong, because a process
already started cannot change what it reports, so bumping studio/backend/main.py
in lockstep does not help one the previous app version spawned.
That backend is proven ours by root id and ownership token, but
lifecycle_control_block_reason returned Unmanageable, and that branch never
calls adopt_verified_backend. has_owned_backend() stays false, so Repair falls
into block_external_conflict, which finds the same process and refuses: the app
could no longer stop a backend it owns the token for. The same regression in
backend.rs turned a terminal-launched same-root server from AttachedReady into
ExternalConflict.
Split the constant: DESKTOP_BACKEND_MANAGEABILITY_VERSION = 1 for the two
live-backend probes, DESKTOP_MANAGEABILITY_VERSION = 2 for the CLI probe. Every
real gate (protocol, auth, ownership, desktop-login, MIN_DESKTOP_BACKEND_VERSION)
is untouched, so an old backend still reaches OwnedStale, adopt, stop, repair.
Also stop the installer when the stale manifest cannot be removed. Windows
raises on a read-only or locked file, and the pass would then run behind a
marker that still names this version and these digests, so a run killed
part-way would verify as complete.
* Answer for the managed venv, not the one the CLI happens to run in
The guard matched ModuleNotFoundError.name, an import name, against
missing_requirements(), which returns distribution names. So a missing PyJWT
printed 'pip install jwt', and jwt, docx and fitz are each a real but unrelated
PyPI project (fitz is a neuroimaging workflow tool), so following the advice
installed the wrong package and left the backend just as broken. Map the import
to its distribution before deciding, and never offer the import itself.
install_state() verified the caller's own prefix. The wheel ships studio/, so a
CLI installed outside the managed venv always finds its own copy of the helper
first, and a healthy managed install reported studio_install_incomplete with a
missing list copied from the wrong venv. Selecting the root is not enough:
_installed_version() reads the running interpreter and req_root defaults to the
caller's studio.txt, so both checks still answered for the wrong venv. Hand
verify_install() that venv's own metadata, enumerated through
Distribution.discover(context = ...path), which does not fall back to sys.path.
The candidate order is untouched, so shadowed-tree detection is unchanged.
setup.ps1 replaces pip, torch and triton before install_python_stack.py runs,
so the manifest it drops is not dropped before the first mutation. A run killed
in between kept a marker that still verifies while torch was half-replaced;
drop it at the top of the dependency pass instead. setup.sh is unaffected, the
stack is the first thing its pass runs, and a test now pins both.
pip uninstall rewrites nothing that was fingerprinted, and cache_matches
re-reads the cached studio_install_ok rather than re-checking, so a venv that
lost a studio.txt package kept being served the healthy verdict. Fold a sorted
hash of the installed dist-info names into the marker hash.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* A missing manifest helper is a torn install, not an old one
studio/install_manifest.py ships in the same wheel as _studio_deps.py, so
nothing legitimately has one without the other: a CLI predating both never
reaches this code, and the desktop already calls such a CLI stale on
desktop_manageability_version.
Returning ok=true there reported a healthy install for a tree the package
update had half replaced, and the preflight then launched a backend whose
own run.py could be just as absent. Report it incomplete so repair runs.
* Tighten comments across the install-detection changes
* Validate Studio dependency readiness
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* split parallel tool calls for single-call-only chat templates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
The STT model list used text-[9px] and text-[10px], which ignore the UI
font-size preference and fail the test_no_raw_pixel_text_utilities contract.
Swap them for the scale-aware text-ui-9 / text-ui-10 tokens.
test_inline_font_size_styles_reference_the_scale compares source-relative paths
against FONTSIZE_PROP_ALLOWED_DIRS and FONTSIZE_STYLE_ALLOWLIST, both written
with forward slashes. It built those paths with str(path.relative_to(SRC)),
which is backslash-separated on Windows, so startswith() never matched and the
allowlists silently did nothing.
The suite is green on Linux CI and fails locally on Windows with 22 phantom
offenders, all of them the chart cards the allowlist already covers.
Route the paths through a _rel() helper that returns .as_posix(), and use it for
the other two offender messages too so failures read the same on every OS.
The ROCR-vs-HIP masking comments cite "a gfx1103 iGPU under a gfx110X
prebuilt" as an example of a GPU the build has no kernels for, but the
shipping gfx110X prebuilt does build gfx1103: unsloth-prebuilt-rocm.yml
passes -DGPU_TARGETS=gfx1100;gfx1101;gfx1102;gfx1103 on both Linux and
Windows, and the b10079 manifest maps all four. install.sh also routes
gfx1103 to gfx110X-all and is_rdna() includes it.
Swap in gfx1036 under gfx103X, which is genuinely unbuilt: that bundle
maps only gfx1030/1031/1032/1034.
Comment-only, no behavior change.
* feat(install): opt-in Vulkan llama.cpp backend and HIP gfx fallback (#7357)
Add UNSLOTH_LLAMA_BACKEND=vulkan and --llama-backend vulkan to force the
upstream Vulkan prebuilt on any host, persist llama_backend in the install
marker, and re-assert it during Studio updates.
On Windows AMD, auto-fallback to Vulkan when no detected gfx arch is in the
upstream win-hip-radeon GPU_TARGETS set (e.g. gfx803 / RX 480). Mixed setups
where at least one card is HIP-supported still default to HIP unless opted in.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(install): address Codex P2s on Vulkan gfx routing (#7357)
Honor ROCm family tokens (gfx110X), include fork-supported gfx1103, require
a known active gfx before auto-Vulkan, and base the HIP floor check on the
visible-device target instead of every physical GPU in hipinfo.
* Address Codex review: env namespace, physical-NVIDIA guard, test kwarg
- llama_backend_from_env: stop reading UNSLOTH_LLAMA_CPP_BACKEND. That is a
separate pre-existing setup variable meaning auto/cpu; setup.sh/setup.ps1
warn and ignore other values, so reading it here forced Vulkan behind that
warning. Vulkan opt-in stays on UNSLOTH_LLAMA_BACKEND / UNSLOTH_FORCE_VULKAN.
- _should_auto_vulkan_for_amd_windows: gate on not has_physical_nvidia (not
merely has_usable_nvidia). A CUDA-masked NVIDIA card keeps has_physical_nvidia
while has_usable_nvidia goes False; Vulkan ignores CUDA_VISIBLE_DEVICES and
could enumerate the reserved card. Mirrors the Intel auto path. Explicit
opt-in still overrides.
- test fakes: validate_prebuilt_attempts/validate_prebuilt_choice gained a
llama_backend kwarg; the four fake signatures in the fallback tests now
accept it, clearing the TypeError that reddened Backend CI / Repo tests (CPU).
Tests: UNSLOTH_LLAMA_CPP_BACKEND=vulkan no longer triggers Vulkan; hidden
physical NVIDIA suppresses AMD auto-Vulkan while explicit opt-in overrides.
* Keep gfx1034 on the ROCm path (fork gfx103X bundle covers it)
The WINDOWS_HIP_PREBUILT_GFX_TARGETS allow-list omitted gfx1034, so
_route_to_vulkan_prebuilt downgraded RX 6500/6400-class hosts to the upstream
Vulkan prebuilt before published_rocm_choice_for_host could match the fork
windows-rocm gfx103X bundle (whose members include gfx1034). Add gfx1034 to the
allow-list and a regression test asserting it stays on the fork ROCm asset.
* Fix auto-Vulkan stealing fork windows-rocm gfx908/gfx90a hosts for PR #7373
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix Vulkan marker claiming a backend that was never installed for PR #7373
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten the Vulkan backend routing comments for PR #7373
* Keep the visible-device-aware gfx when setup forwards --rocm-gfx
setup.ps1 resolves the gfx arch from its own probe, and that pick is not
fully visible-device aware: neither the hipinfo nor the amd-smi branch
reads CUDA_VISIBLE_DEVICES, and the amd-smi branch matches a bare integer
only, so a comma-separated HIP/ROCR mask such as 1,0 also falls back to
GPU 0. The resulting arch was then forwarded through --rocm-gfx and
replaced the arch detect_host() had already resolved for the
runtime-visible GPU.
On a mixed-AMD Windows host that flipped the auto-Vulkan decision: with
GPU 0 gfx1100 and a masked-in gfx1010, the forward reinstated gfx1100,
_should_auto_vulkan_for_amd_windows() saw a HIP-supported arch and the
HIP bundle was installed for a GPU that cannot run it.
Fold the forward in as a fill rather than a replacement: it still supplies
the arch on amd-smi-only, driver-only and name-inferred hosts where the
probe reports none, which is what --rocm-gfx exists for, but no longer
overwrites a successfully detected active arch. An explicit
UNSLOTH_ROCM_GFX_ARCH stays authoritative, since it is the documented
manual override for hosts whose arch the probes get wrong.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope the Windows AMD Vulkan fallback per device and per repo
Three follow-ups on the auto-Vulkan routing for #7357.
Keep an explicit --rocm-gfx authoritative. The previous round stopped a
forwarded gfx from replacing an arch detect_host() had already resolved,
but --rocm-gfx is also the documented operator override for hosts whose
probe is wrong or stale, and both arrive as the same argv. Narrow the
advisory case to the two shapes setup can actually be describing: an arch
the probe saw on this host (setup picked a different physical GPU of the
same box), or a family label such as gfx110X, which is a bundle name the
update path derives from the marker asset rather than a real GPU arch.
Any other value is an override for an arch no probe reported and stays
authoritative. Keeping family labels advisory also preserves the rule that
an in-generation-but-unbuilt arch (gfx1033) is never upgraded into the
gfx103X bundle.
Do not auto-route to Vulkan from a HIP-only device mask. HIP_VISIBLE_DEVICES,
ROCR_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES select the active arch, but the
Vulkan runtime honours none of them: it enumerates through
GGML_VK_VISIBLE_DEVICES and Vulkan ordinals in
LlamaCppBackend._get_gpu_free_memory_vulkan. Masking down to a below-floor
card therefore used to install a backend that could still enumerate the
HIP-capable card the user deliberately hid, possibly one reserved for another
workload. Require every physical AMD gfx to be below the floor, matching the
has_physical_nvidia gate right above it. So the per-GPU list survives to that
check, a forward that agrees with the probe no longer collapses
rocm_gfx_targets to a single entry.
Make the HIP support predicate repository-specific. The floor constant is a
union of ggml-org's windows-hip gpu_targets and the fork's windows-rocm
bundles, so it only answers "is this arch served" for the fork. With
--published-repo ggml-org/llama.cpp, direct_upstream_release_plan() offers
win-hip-radeon then CPU and never Vulkan, so the four fork-only archs
(gfx908, gfx90a, gfx1034, gfx1103) were declared supported and fell through
to CPU instead of the Vulkan bundle that would actually run. Add
UPSTREAM_WINDOWS_HIP_GFX_TARGETS and select the set from the planned repo.
* Keep probe-confirmed AMD GPUs in the physical list when a gfx is forwarded
rocm_gfx_targets is the physical inventory _should_auto_vulkan_for_amd_windows()
reads, so a forwarded --rocm-gfx that the probe never reported was deleting cards
the probe had confirmed. On a mixed Windows AMD box whose active device is masked
down to a below-floor card, a stale UNSLOTH_ROCM_GFX_ARCH or a name-inferred arch
for the other GPU collapsed the list to that one arch, the floor check concluded no
AMD GPU on the host reaches the Windows HIP prebuilt, and the install auto-fell back
to Vulkan, which honours no HIP mask and would enumerate the reserved HIP-capable
card. Add the forwarded arch to the list instead of replacing it: it selects the HIP
target, it does not redefine what hardware is present.
An empty probe still yields a single-entry list, so the driver-only Windows AMD host
the forward exists for keeps its automatic Vulkan fallback, and an explicit
--llama-backend vulkan is unaffected.
* Do not auto-fall back to Vulkan when a HIP device mask filtered the probe
hipinfo is itself a HIP application, and AMD documents HIP_VISIBLE_DEVICES as
"only devices whose index is present in the sequence are visible to HIP", with
that spelling recommended on Windows. Under a mask the Windows probe therefore
enumerates the visible devices, so rocm_gfx_targets is what survived the mask
rather than the physical inventory the auto-Vulkan floor check assumes. A
masked-out gfx1100 next to a visible gfx803 made the check conclude that no AMD
GPU on the box reaches the Windows HIP prebuilt and route the install to Vulkan,
which honours none of these masks and would enumerate the reserved card.
Decline to guess when a mask is set: the physical inventory is unknowable from a
masked probe, so keep the HIP / fork / source path. This only ever turns the
automatic fallback off, never on. The driver-only single-GPU host the fallback
exists for sets no mask, an all-hiding "" / -1 mask is still handled as no active
target rather than a partial view, and an explicit --llama-backend vulkan or
UNSLOTH_LLAMA_BACKEND=vulkan is unaffected.
Reading the physical inventory through an unmasked re-probe would also correct
_pick_rocm_gfx_target, which indexes the token list by the mask value and so
already assumes an unmasked probe. That is pre-existing behaviour on main and is
left alone here.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Treat an all-hiding HIP device mask as suppressing the Vulkan fallback too
The mask guard exempted an empty or -1 value on the grounds that the probe reports
no active target under it, but that only holds for the probe: a forwarded
--rocm-gfx still reconstructs an active arch, and setup infers that arch from the
display-adapter name, which no HIP mask touches. A user who hid every AMD GPU from
HIP could therefore still be auto-routed to Vulkan, which honours none of these
masks and would then use all of them. That is the strongest form of the hazard the
guard exists for, not an exemption from it.
Presence of any of the three variables is now the whole test, which also removes
the value parsing. An explicit --llama-backend vulkan or UNSLOTH_LLAMA_BACKEND is
still unaffected.
* Grant the fork-only Windows HIP coverage to the fork, not to every mirror
The floor set is a union of the fork's windows-rocm bundles and only the fork is
planned from its manifest: resolve_simple_install_release_plans() compares
== DEFAULT_PUBLISHED_REPO and sends every other --published-repo through
direct_upstream_release_plan(), whose AMD branch offers win-hip-radeon then CPU
and never Vulkan. Exempting only the exact ggml-org spelling therefore told a
mirror carrying upstream-standard assets that fork-only archs such as gfx1034,
gfx1103 and gfx908 were HIP-served, landing them on HIP or CPU instead of the
Vulkan bundle that would actually run. Gate on the fork instead.
Matching the dispatch exactly, spelling included, also fixes a differently cased
repo: that really does take the upstream path, so it must be answered with
upstream coverage rather than the fork superset. An empty repo still defaults to
the fork, as the resolver does.
* Derive the Windows HIP gfx floor guard from the published manifest
The guard compared WINDOWS_HIP_PREBUILT_GFX_TARGETS against a second hardcoded
tuple in the same test file, so a windows-rocm arch newly published by the fork
passed both. Affected hosts would then be routed off the hash-approved fork ROCm
bundle onto an unhashed upstream Vulkan build with nothing failing.
Read the fork's llama-prebuilt-manifest.json through the installer's own
resolver instead, and assert the floor, the family labels, and the routing
tuple all still cover what it publishes. The manifest ships only as a release
asset, so an unreachable release skips with an explicit reason rather than
flaking. Both literals match the manifest as published today.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Compress the Vulkan backend routing comments and docstrings for PR #7373
* Correct the family-label rationale in the Windows HIP coverage check
The comment justified serving gfx103X / gfx110X against any repository by
claiming upstream's windows-hip targets build every member of those families.
The fork manifest maps gfx103X to gfx1030..1032 plus gfx1034 and gfx110X to
gfx1100..1102 plus gfx1103, and UPSTREAM_WINDOWS_HIP_GFX_TARGETS carries
neither gfx1034 nor gfx1103, so the stated reason is wrong even though the
answer is right.
State the real reason instead. A family label is a bundle name, not an arch,
so the concrete GPU is unknown at this point; answering unsupported to cover
the two uncovered members would move gfx1030..1032 and gfx1100..1102 off a
working HIP build onto Vulkan for a card the label cannot identify. Those two
archs still reach Vulkan through the concrete-arch branch below, which does
answer per repository.
Comment only. No behaviour change: the 5850-combination override sweep still
reports 0 rocm_gfx_target changes, 0 auto_vulkan False to True flips and 680
True to False flips all backed by a probe-confirmed HIP GPU, and both the
feature and override profile matrices are byte-identical.
* Pin that a deliberate CPU install outranks Vulkan for PR #7373
UNSLOTH_LLAMA_CPP_BACKEND (setup.sh / setup.ps1, "auto" or "cpu") and
UNSLOTH_LLAMA_BACKEND (this module, a backend name) are separate variables at
separate layers, and both accept "cpu". setup translates its own =cpu into
--force-cpu, which is what pins the CPU-only bundle on a GPU host and keeps
Intel iGPU Vulkan crashes away (#7213), so no trigger this PR adds may
outrank it.
_route_to_vulkan_prebuilt already gets this right, since force_cpu
short-circuits ahead of the forced, auto-Intel and auto-no-HIP triggers.
Cover it so it stays that way: the matrix runs [Linux, Windows, macOS] x
[NVIDIA, AMD, Intel, CPU only] x [unset, vulkan, hip, rocm, cpu] with the
legacy UNSLOTH_FORCE_VULKAN set as well, and asserts the published bundle
survives every one. WSL presents as Linux to this resolver, so it rides the
Linux row.
Also assert the guard is not vacuous: the same host still takes Vulkan once
the CPU pin is gone, so the matrix cannot pass on a resolver that had simply
stopped routing to Vulkan.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: LeoBorcherding <borchborchmail@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
test_torch_constraint.sh asserted how many times each pin appears in install.sh.
Every hardware branch assigns its own torch/torchvision/torchaudio triple, so
#7354 adding gfx906 pushed three of those counts up by one and Backend CI has
been red on main since:
FAIL: default TORCH_CONSTRAINT assignment exists (expected '1', got '2')
FAIL: hardcoded torch>=2.4 appears exactly once (expected '1', got '2')
FAIL: torchvision bounded (<0.26) at default + custom-leaf (expected '2', got '3')
FAIL: torchaudio bounded (<2.11) at default + custom-leaf (expected '2', got '3')
install.sh is correct; the numbers were the stale part. Assert the invariants
instead, so the next hardware branch is not a test edit:
- the default assignment is the top-level one, so anchor the grep at column 0
rather than counting every occurrence. An indented branch pin no longer
satisfies it, which the old count did not distinguish either.
- what "appears exactly once" really guarded is that no pip install line spells
a pin out instead of using "$TORCH_CONSTRAINT", so check that directly.
- companions must be bounded everywhere, so compare bounded assignments against
total assignments rather than pinning a count of 2. That is strictly stronger:
it now covers all 7, not the 2 the old numbers happened to name.
45 pass, 0 fail. Each new assertion fails when its property is broken: a bare or
unbounded companion, a hardcoded pin on an install line, or a missing top-level
default.
The `/~/` rule was root-anchored, so it only caught a stray "~" directory
created at the repo root. Tools run from a subdirectory create it there
instead, and studio/frontend/~/ once reached a PR as 708 tracked files (a
Node compile cache plus cwd markers). Unanchor it.
Also ignore /temp/, the repo-root scratch dir
tests/studio/test_chat_preset_builtin_invariants.py writes into. Nothing
under either path has ever been tracked on main.
* fix: keep offline GGUF export off the Hub for VLM tokenizers (#7481)
Resolve cached snapshot directories before loading PreTrainedTokenizerFast
during VLM processor fallback so transformers does not call is_base_mistral()
-> model_info() when HF_HUB_OFFLINE is set. Also probe the local cache in
_has_tokenizer_model instead of model_info when offline.
Fixesunslothai/unsloth#7481
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test: add real-cache offline GGUF integration checks for #7481
Download unsloth/gemma-3-270m-it-bnb-4bit (~430MB) and verify offline
snapshot resolution and tokenizer load with network blocked. Full unsloth
import tests remain GPU-gated.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: address Codex review on offline GGUF tokenizer paths (#7481)
- Only rewrite Hub repo ids to cached snapshot dirs when offline
- Copy tokenizer.model from cache offline in preserve_sentencepiece
- Do not cache negative offline tokenizer.model probe results
- Add regression tests for all three review items
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: probe HF cache before model_info for local-only GGUF saves (#7481)
Always resolve tokenizer.model from the local Hub cache before calling
model_info, and skip Hub metadata when the tokenizer was loaded with
local_files_only or offline env vars. Fixes Codex review on PR #7482.
* Fix lint blocker, false-green tests and offline defaults for PR #7482
Drop the two unused _env_says_offline imports that fail the Source lint
import-hoist check.
test_has_tokenizer_model_offline_skips_model_info and its local_files_only
twin set model_info.side_effect = AssertionError, but _has_tokenizer_model
wraps that call in "except Exception: return False", so the AssertionError
was swallowed and both passed on the merge base with the fix absent. Assert
model_info.call_count == 0 instead; both now fail on the base with
assert 1 == 0.
The real-cache integration tests called hf_hub_download and
PreTrainedTokenizerFast directly, so they exercised plain huggingface_hub and
passed identically on both trees. Route them through the resolver this PR
adds, and gate the file at module level since importing unsloth needs a GPU
host either way.
_resolve_hub_repo_local_dir and _resolve_hub_repo_cached_file defaulted to
local_files_only = False, so a helper named "resolve local dir" would
download with backoff retries when called without the flag. Every caller
already passes it explicitly, so default it closed.
Use tempfile.gettempdir() rather than a hardcoded /tmp, which silently
skipped both files on Windows, the platform in the bug report. Patch
socket.socket connect rather than replacing the class, which broke
isinstance checks.
Wire the unit tests into the Bucket-A CI list; Repo tests (CPU) ignores
tests/saving, so none of these ran anywhere.
* docs: note transformers 4.57.2-5.5.4 window for local tokenizer resolve
Name the version range where from_pretrained still probes model_info under
local_files_only, and point at the 5.6.0 upstream fix so the helper can be
removed once the supported floor moves past it.
* fix: enable real-cache suite in offline GGUF integration runner
Pass UNSLOTH_INTEGRATION_IMPORT=1 into the pytest subprocess so the
documented runner actually executes the real-cache tests instead of
reporting success after only the fake-cache unit file runs.
* docs: note integration runner enables UNSLOTH_INTEGRATION_IMPORT
Document that the runner sets the gate itself and still needs a host
that can import unsloth.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep an explicit local_files_only load local-only at save time
transformers takes local_files_only as an explicit from_pretrained parameter,
so it never lands in tokenizer.init_kwargs, and _offline_aware_load restores
HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE as soon as the load window closes. A VLM
loaded with local_files_only = True but no offline env var therefore came back
with the Hub repo id in name_or_path and nothing recording the request, so
_tokenizer_wants_local_only returned False on the later save and
_has_tokenizer_model fell through to HfApi.model_info - and then
_preserve_sentencepiece_tokenizer_assets fetched tokenizer.model from the Hub
with local_files_only = False. On a disconnected host that is a network wait
before the export gives up.
Stamp the load's local-only mode onto the returned processor and its tokenizer
inside the forced-offline window, and honour that stamp in
_tokenizer_wants_local_only, so the save path inherits the load's contract.
Verified against a real hub-cache layout whose snapshot has tokenizer metadata
but no tokenizer.model: before, one model_info call plus an hf_hub_download with
local_files_only = False; after, zero model_info calls and cache probes only.
Two tests added to tests/saving/test_offline_gguf_vlm_tokenizer_7481.py; both
fail with the loader_utils hunk reverted and pass with it in place.
* Carry the load's cache_dir through to saving for PR #7482
The local-only stamp added in e7b7400de preserved only the boolean. Saving
still derived its cache from HF_HUB_CACHE or HF_HOME, which does not see a
caller-supplied cache_dir, and FastBaseModel.from_pretrained threads one all
the way down. So a local_files_only load against a custom cache missed on the
probe, and the stamp then stopped the Hub fallback that used to cover it, and
tokenizer.model was silently left out of the GGUF staging directory.
Stamp the cache_dir alongside the local-only marker and prefer it at both
sites in save.py that derive one from the environment. Reverting save.py
alone, with the helper still present, fails the new test on behaviour.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Merge main and drop an unused import for PR #7482
Brings the branch up to date with main, which clears the stale Source lint
blocker inherited from #7476 by taking studio/backend/utils/hardware/__init__.py
out of this PR's changed-file set.
pytest was imported in the new test file and never used, which the
import-hoist check flags in its own right.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Comment-only follow-up to #7454. That change carried 523 comment lines, many of
them three and four line preambles where one line says the same thing. This
collapses them and drops the ones restating what the code already says, for a
net 77 lines.
Scope is limited to comments #7454 itself introduced. The files it touched hold
about 3,761 comments in total; the rest predate it and are untouched, verified
by checking that every removed line is one that commit added.
Nothing that records why a non-obvious decision was made was dropped, only
compressed. Still stated: the normcase-before-versus-after Windows separator
trap, the innermost-indexed-model rule for nested directories, an HTTPException
being a decision rather than a failure to decide, that only an explicit False is
anonymous to huggingface_hub while None borrows the server owner's login, the
fail-closed tri-state custom-code gate, and the regressions each test was
written for.
Code is provably unchanged: comment_tools.py check reports 17/17 files
comments-only. Backend CI command 10337 passed, 0 failed. tsc -b clean.
* Add CLI GPU memory mode selection
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Preserve manual GPU layer overrides
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Add community-maintained legacy support path for gfx906 (MI50 / Radeon VII)
rocm6.4+/7.x torch wheels bundle ROCm libraries whose Tensile kernels
dropped gfx906 (rocBLAS 'TensileLibrary.dat ... not read for gfx906',
ROCm/TheRock#1844), so on MI50/Vega 20 hosts with newer ROCm the
installer picked wheels that fail at the first BLAS call. The rocm6.3
index is the last one whose wheels run on gfx906 (torch 2.7.0 verified
on MI50 32GB, up to 2.9 in community use). Dynamo/Inductor codegen is
also broken on this arch, crashing compiled graphs that train fine in
eager mode.
- install.sh: when the runtime GPU is gfx906 and the picked index is
newer than rocm6.3, reroute torch to the rocm6.3 index and reset the
constraint trio to the default <2.11 window (a rocm7.2 pick raises
the floor to 2.11, which rocm6.3 cannot satisfy), with a legacy-path
warning.
- install_python_stack.py: mirror the reroute in _ensure_rocm_torch
using the _default pkg specs, including repairing an existing
+rocm7.x torch and leaving a working rocm6.3 install alone.
- device_type.py: default TORCHDYNAMO_DISABLE / TORCH_COMPILE_DISABLE /
UNSLOTH_COMPILE_DISABLE on gfx906 (setdefault, user override wins).
Windows allowlists are untouched: repo.amd.com publishes no gfx906
wheel family (verified in the RDNA2 enablement PR). 16-bit LoRA and
full finetuning work out of the box; 4-bit QLoRA needs a source-built
bitsandbytes for gfx906. Based on the verified MI50 32GB setup in
namnguyen0503/mi50-gfx906-unsloth-bnb4bit-lab.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gfx906: second Codex pass (bnb skip under pin, override beats Strix)
- Compute the gfx906 runtime-target flag independently of any torch-index
pin or Strix override, so the bitsandbytes skip still applies when a user
pins the ROCm index and sets UNSLOTH_ROCM_GFX_ARCH=gfx906 (the pin
suppresses the torch reroute, not the bnb skip). Probe only when no pin
is set (an explicit pin means don't second-guess it, matching the Strix
path's asserted no-probe invariant); an explicit gfx906 override needs
no probe.
- Let UNSLOTH_ROCM_GFX_ARCH=gfx906 suppress the Strix reroute (both
install.sh and install_python_stack.py) so a mixed Strix + MI50 host
routes to rocm6.3 instead of the gfx1151 wheels probe order would pick.
- Fix test_hardcoded_torch_constraint: the default <2.11 window literal now
legitimately appears on two TORCH_CONSTRAINT= assignments (default + the
gfx906 reroute reset after the rocm7.2 floor bump); assert it only ever
appears on assignment lines, never on a pip install line (its real intent).
New tests: bnb skipped under an explicit pin, gfx906 override wins over
Strix, install.sh suppresses Strix on the override. rocm_support +
selection + cross-platform parity: 667 passed; structural constraint 9/9.
* gfx906: collapse single-line asserts to match pre-commit formatting
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gfx906: keep bnb skip + rocm6.3 routing correct under pins and suffixed overrides
Address the four Codex P2 findings on #7354:
- bnb skip under a pinned index (install.sh + install_python_stack.py):
a real gfx906 host that pins UNSLOTH_TORCH_INDEX_URL to rocm6.3 without also
setting UNSLOTH_ROCM_GFX_ARCH no longer reinstalls the generic bitsandbytes
wheel over a source-built gfx906 bnb. A pin now suppresses only the torch
reroute, not the gfx906 detection used for the bnb skip (Python drops the pin
gate on _runtime_is_gfx906; bash _is_gfx906_bnb_skip probes via
_probe_amd_gfx_arch when the index is pinned).
- clear the Radeon marketing-name flag for every gfx906 target, not only when
the >=6.4 reroute fires, so a Radeon VII already on rocm6.3 does not divert to
the repo.radeon.com branch (whose wheels lack gfx906 kernels).
- normalize a copied HIP gcnArchName (gfx906:sramecc-:xnack- -> gfx906) before
the exact comparisons in install.sh and install_python_stack.py, mirroring
device_type.py.
Tests: relax the three Strix-pin tests (the gfx probe may now run for the bnb
flag but must not reroute the pinned index) and add coverage for the pinned
bnb skip, the suffixed override, and the bash Radeon-clear / pinned-probe paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gfx906: log skipped vLLM aimv2 fix + robust source-scan test bounds
Follow-up review polish:
- import_fixes: log at info level when the vLLM aimv2 fix is skipped because
the dist metadata is unreadable, so the skip is diagnosable instead of silent.
- test_rocm_support: bound the gfx906 install.sh source-scan on the ';;' that
closes its case arm via a shared _gfx906_reroute_block helper, replacing the
brittle fixed-length (3200/3800) slices that shift when the block grows.
* gfx906: trim whitespace on UNSLOTH_ROCM_GFX_ARCH in install.sh (py parity)
The bash gfx906 comparisons lowercased and stripped the gfx906:… feature
suffix but not surrounding whitespace, while the Python paths do .strip().
A stray newline (e.g. export UNSLOTH_ROCM_GFX_ARCH=$(cmd)) would make bash
miss gfx906 while Python catches it. Trim with `tr -d '[:space:]'` at both
comparison sites so the reroute target and bnb-skip agree across bash/Python.
* gfx906: remove generic bitsandbytes pulled in transitively after the skip
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Vulkan GPUs: real device names and selectable ordinals
Rebases the durable half of #7356 onto the inference_gpu transport #7476
landed on main. Those two PRs solve an overlapping problem and disagree on
the data model, so merging #7356 as-is would ship two parallel Vulkan
device concepts with different index semantics. This keeps main's transport
and adds what #7356 had that #7476 does not.
- _vulkan_probe.py emits a 5th column, ggml's device description, sanitized
for the tab protocol and UTF-8 safe. Reader tolerates 4- or 5-column
output so an older probe still parses.
- llama_cpp gains _run_vulkan_probe (shared parse) and
vulkan_device_inventory (names + is_igpu + real totals).
- get_vulkan_inference_gpu_info reports the real name and an explicit
is_igpu instead of "Vulkan<i>" and a total == 0 guess.
- index_kind becomes "vulkan", not "relative", and gpu_ids picks are
supported on Vulkan builds once the probe enumerated ordinals. The XPU ban
no longer applies to them: a Vulkan pick is a ggml ordinal, not a torch-xpu
index, so it works on an Intel host too.
- Frontend picker reads the Vulkan inventory as the pickable set.
Memory deliberately still comes from _get_gpu_memory, not the inventory.
That path applies _apply_igpu_host_reserve_mib and zeroes a shared total;
budgeting an APU off its raw shared total would hand out the whole machine's
RAM with no OS headroom. Identity is joined onto it by ordinal, so a probe
failure degrades to Vulkan<i> names with the memory readings intact.
Dropped from #7356 as superseded: validate_vulkan_gpu_ids (main's
resolve_requested_gpu_ids already rejects duplicates and
_resolve_gguf_gpu_ids_for_request already probes for existence), the
gguf_devices transport, and the iGPU budget fallback in 71619891e, which
main's aggregateGpuMemoryTotalGb handles better by counting a shared pool
once.
Also keeps #7356's removal of the late diffusion raise, so the graceful
gpu_ids drop stays reachable for a GGUF only classified as diffusion after
download. #7415's real guard, _reject_vulkan_diffusion_gpu_ids_before_
teardown, is untouched.
Verified on Windows + Strix Halo: backend Vulkan/GPU-selection suites at the
same 4 pre-existing failures as main, tests/studio 1671 passed with no new
failures, frontend typecheck clean. Hardware confirmation of the underlying
behavior is on #7356 from @Bebiv24 (RX 9070 XT + RX 480).
Co-authored-by: LeoBorcherding <borchborchmail@gmail.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: LeoBorcherding <borchborchmail@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: say which model is missing instead of "No model loaded"
A /v1 request naming a model that is not downloaded returned the generic
"No model loaded. Call POST /inference/load first.", which cannot fix it.
Return 404 model_not_found naming the model and listing what can serve,
and make the API usage examples name a model the server actually has.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: page the API monitor, show model load/unload, pin the example quant
The monitor rendered all 50 retained entries in one scroller: page it 5 at a
time, freezing history while paged back so live traffic cannot reorder it.
Add model load/unload rows so the feed shows what the server is doing, and
stop the header rendering the loaded model as a raw host path. Advertise each
model's GGUF quant on /v1/models so the example pins repo:QUANT, and move the
auto-switch section above the monitor with shorter copy.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: optionally download a model named in an OpenAI API request
Auto-switch only ever loaded models already on disk, so naming one this
server does not have either 404s or, when something else is loaded, gets
quietly answered by the resident model.
Add openai_api_auto_download_model (off by default, gated on auto-switch).
When on, a /v1 request naming a GGUF repo that is not downloaded starts a
background fetch and returns 503 with Retry-After and a typed
model_downloading code. The resident model keeps serving in the meantime,
and the retry after the download completes is served by the new model
through the existing auto-switch path.
The download reuses the Hub manager's service layer, which already does
repo-id validation, casing, claim bookkeeping, disk preflight, resume and
cancel. The in-loader download is deliberately not used: it silently falls
back to a smaller quant under low disk, which is wrong when the caller
named an exact one.
Admission is narrow, since a request only needs an API key:
- namespace/name only, so gpt-4 and other foreign ids fall through to the
resident model exactly as before
- GGUF only, decided from the remote file list rather than the repo name
- anything declaring auto_map is refused, so trust_remote_code stays a
deliberate opt-in in the UI and can never be granted over the API
- a single download at a time, plus a free-disk reserve
- one model_info call answers existence, gating and the quant list, so a
missing repo, a gated repo and a wrong quant each get their own error
With the setting off every one of these paths is byte-identical to before.
Also:
- monitor rows for downloads, with a live percentage
- public_model_id resolves an HF cache snapshot to its repo id, so a
cache-loaded model is no longer labelled with a commit sha; this drops
the duplicate helper added for the monitor and fixes the same leak in
the inference status response
- the unedited sk-unsloth-YOUR_KEY from the copyable examples now says so
instead of "Invalid or expired API key"; every other bad key keeps the
generic message
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add an Unload button to the API monitor
The monitor names the loaded model but offered no way to free it. Idle
auto-unload is the only existing release path, and it needs a TTL and a
wait.
The button sits next to Refresh, appears only while a model is loaded and
is disabled mid-unload. /unload matches on the internal identifier, which
this response deliberately omits because it would be a host path, so the
click reads it from /api/inference/status the same way the chat runtime
does rather than widening the monitor payload.
Also stamp the manual unload row with the quant, read before the teardown
clears it, so it reads repo:QUANT like the load row it pairs with.
* Studio: keep the API monitor Unload button visible when idle
It only rendered while a model was loaded, which hid the one manual
release path at exactly the moment someone goes looking for it. Render it
always, disabled with a "No model is loaded" tooltip when there is nothing
to free.
* Studio: never answer a named model with a different one
Asking for a model this server is not serving returned 200 from whatever
was resident. Requesting gemma-4-E2B-it-GGUF:UD-Q6_K_XL while UD-Q4_K_XL
was loaded got a confident answer from the wrong quant, with nothing in
the response saying so.
A name carrying a namespace (org/model, optionally :QUANT) is a concrete
reference, so 404 instead, with the reason:
- wrong quant -> names the quants that are actually downloaded
- not on disk -> lists what is available
- on disk but auto-switch off -> says to turn it on
Ids without a namespace (gpt-4, claude-3, default) are foreign labels
rather than references, so they still fall through to the resident model
and drop-in clients are unaffected. A bare org/model is still satisfied by
any loaded quant of that repo; only an explicit :QUANT must match.
The check runs whatever the auto-switch and auto-download toggles are,
since serving the wrong weights is wrong in every configuration. It is
skipped when nothing is loaded, where the existing no-model-loaded error
already says the right thing, and when the model is on disk with
auto-switch on, where a failed swap should still fall back.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: use a simpler prompt in the API usage examples
"What is Unsloth Studio?" rather than "Can Unsloth Studio do API calling?".
One constant feeds all nine snippet tabs.
* Studio: only refuse a model reference meant for this server
A namespace alone was treated as a concrete model reference, so a /v1
request naming anthropic/claude-3.5-sonnet, openai/gpt-4o or any other
LiteLLM or OpenRouter style vendor/model id started returning 404 instead
of being answered by the resident model. Refuse only on evidence the
caller meant this server: an explicit GGUF quant label, or a repo that is
actually on disk here. gpt-4 and vendor/model alike fall through again,
while the wrong-quant and wrong-repo cases this PR exists for still
refuse.
Also from review:
- Release the single download slot by object identity, not repo id. A
stale watcher could clear a newer download of the same repo and let a
second multi-GB fetch start alongside it.
- Catch BaseException around admission: CancelledError is not an
Exception, so a cancelled request stranded the slot for the process
lifetime.
- Honour the download service's accepted=False, which it returns without
raising for a cross-variant conflict, instead of promising a download
that was never dispatched.
- Treat a failed status probe as unknown rather than idle, so a transient
read cannot fail the monitor row and free the slot under a live worker.
- Check gated repos with auth_check. The Hub serves metadata for a gated
repo without granting its files, so the licence gate was being reported
as the unrelated custom-code refusal.
- Size the disk reserve from the download plan, which includes the mmproj
and MTP companions the worker fetches with every quant.
- Never fetch under the server's own HF token. The repo is named by
whoever holds an API key, so the ambient token let that key pull the
owner's private repos.
- Refuse an explicit quant on a backend with no quant identity, gated on
the suffix really being a quant so Ollama style :latest tags still match.
- Raise instead of falling through when the diagnosis fails: the mismatch
is already established by then, only the wording is uncertain.
- Report a failed switch as 503 model_switch_failed rather than answering
as the resident model.
- Fail an open monitor row under the same lock as the check, so a finish
landing in between cannot stamp an error onto a completed row.
- Usage examples never emit a hardcoded model id: the catalog is tri-state
and the panel asks for a model to be loaded instead of printing one the
server cannot serve. It also refreshes when the loaded model changes.
- Keep the monitor pager reachable while frozen entries expire.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: scope the auto-download 404 cache to the caller's credentials
The Hub answers 404 for a private repo the caller cannot see, so caching
that verdict per repo alone let one anonymous request mark a private repo
unservable for everyone for the whole TTL. A later caller sending a valid
X-Unsloth-HF-Token skipped the probe and fell through to the resident
model instead of downloading what it asked for. Keyed on the repo id plus
a digest of the token now, so the token itself is never held.
Two more from the same review:
- Clear the chat runtime checkpoint after unloading from the API monitor,
as the chat eject flow already does. The store went on treating the
freed checkpoint as loaded and the usage examples kept naming it.
- Point gated and not-found callers at the X-Unsloth-HF-Token header.
Automatic download deliberately ignores the server's own Hugging Face
identity, so telling the user to add a token in Studio sent them round
the same 403 forever.
* Studio: tighten the comments added by this branch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep API auto-download off the server's Hugging Face identity
Passing None for the caller's token was not anonymous. spawn_worker
substitutes the backend's HF_TOKEN for a falsy one, and HfApi(token=None)
falls back to a cached login, so a repo named by an API-key holder could
still be fetched under the owner's Hub identity and land in the shared
catalog. The metadata probe and auth_check now pass an explicit False,
and dispatch threads allow_ambient_token=False so the worker stays
anonymous too. The flag defaults to True, so the UI download path keeps
the ambient fallback that private repos rely on.
Three more from the same review:
- Require an exact hf_variant match only when the suffix is really a
quant. The llama.cpp branch still compared Ollama style :latest and :8b
against the loaded quant and refused the resident model, which is the
opposite of what looks_like_quant classifies them as.
- Decode an HF cache repo id only when the models-- component is followed
by snapshots. An ordinary directory whose name merely starts with
models-- was being read as an encoded repo id.
- Return the probing response before consulting the job registry when an
adopted claim has no variant yet. A stale error on the whole-repo key
could otherwise release the slot the first request's probe still holds,
letting a second large download start beside it.
* Studio: stop treating a namespace as what decides model intent
The rule refused a reference only when it carried a namespace, which was
wrong in both directions. vendor/model is how LiteLLM and OpenRouter name
every provider, and a standalone or custom-folder GGUF is advertised
without one, so asking for a path-free local id such as model-Q4_K_M was
answered by whatever else happened to be resident. The slashless early
return is gone and the same evidence test now applies to every id: an
explicit quant, or a model that actually resolves here. gpt-4 and default
still fall through because they are not local, not because of their shape.
Also:
- Recognise bits-per-weight quant labels. _extract_quant_label emits
IQ4_XS-3.53bpw and the resolver and downloader both accept it, but
_GGUF_KNOWN_QUANT_RE has no bpw group, so looks_like_quant rejected a
reference the rest of the machinery understands.
- Upper-case the synthetic names handed to _pick_best_gguf. Its preference
tokens are upper case and matched case-sensitively, so a repo with
lower-case filenames skipped the preference and took the first entry,
which can be F16.
- Only offer a downloaded but unloaded model as a runnable example when
auto-switch is on. It is off by default, so the copied snippet hit the
no-model-loaded error, which is the failure this branch exists to fix.
The tool-passthrough cancel test stubbed asyncio.to_thread module-wide, so
it cancelled at the first thread hop rather than the generation hop it
means to test. Model resolution runs off the loop before the monitor row
opens, so that stub now passes the resolver through.
* Studio: tighten the comments added since the last pass
* Studio: match a resident model through its resolver alias
A manual load stores the model by its on-disk path while the resolver and
/v1/models advertise it as publisher/model, so _loaded_satisfies could not
recognise the alias. Reducing the resolution to a boolean then threw away
the load path that would have proved the match, and the request was
refused with 404 for a model the server was serving at that moment. Common
for LM Studio models and custom-folder aliases. The resolved path is
compared against the resident backend before anything is refused.
Also:
- Size disk admission on what is left to fetch. expected_bytes is the whole
plan, so a resumed quant or a companion already pulled in by another
quant was charged for twice and could 507 a download that fits. Cached
blobs are subtracted through existing_blob_bytes, the same accounting the
worker's own preflight does, and it falls open to the full size when no
blob hashes are available.
- Report a cancelled download as cancelled. The catch-all sent every state
other than complete or idle through fail_open, so a deliberate cancel
rendered as a download failure rather than the monitor's cancelled state.
- Keep polling the servable ids while nothing is loaded. The poll settled
as soon as auto-switch was on, so turning it back off left the examples
naming an unloaded model until something else remounted the panel.
* Studio: shorten the comments added in the last pass
* Studio: keep the FLA fast-path tests hermetic across transformers versions
_discover_fla_model_types scans the *installed* transformers for modeling
files importing `from fla.`, so `models/qwen3_5/` only exists from
transformers 5.x. The backend supports transformers>=4.51, and on a 4.x
install the Qwen3.5 gate returns False, so 14 tests in
test_training_worker_flash_attn.py silently exercised a no-op instead of the
install path and failed their call-count assertions.
Pin the discovered model_type set in those 14 tests, the same way
test_hook_does_not_install_tilelang_for_model_outside_allowlist already pins
it against newly added FLA model_types. Test-only change: the production
gate and the _discover_fla_model_types unit tests are untouched.
* Studio: keep the /v1 admission check off the model-scanning path
The admission check added here runs on every /v1 request, including with
auto-switch off, where the route used to return straight away. It called
resolve_local_gguf, whose index is cached for 5s and otherwise rebuilt by
walking ./models and every HF cache root, under a lock the next caller waits
on. On an install with a large cache that scan measured 6.1s, longer than the
TTL that is meant to amortise it, so steady traffic would keep rebuilding it.
Answer from the last built index instead and never rebuild from the request
path: a stale answer is fine here, since what is on disk barely moves and a
finished download already invalidates the index. The first request, before any
scan has completed, warms the index on a background thread and skips the check
rather than blocking on it. That also makes the lookup a dict read, so it no
longer needs handing to a thread.
Cold resolve on this box goes from 6152495us to 0.4us, and the whole hook now
costs the same for a foreign label as for the resident model.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix the admission hook's cold, stale and contended index paths
Five review items, four of them on the admission hook added here.
Skipping the check until the first scan lands also skipped explicit quant
mismatches, so the first request after startup could ask for :Q8_0 while
Q4_K_M was resident and be answered by it. The early return was redundant as
well: with an empty index resolved is None and here is False, so the gate below
already lets a bare name through and refuses an explicit quant, which is what
the except branch has always concluded. Dropped it and index_is_built with it.
index_is_built took _lock, which _index holds for the whole scan, so once a
warm was running every later request blocked on the event loop for exactly as
long as the scan it was there to avoid. The warm now has its own lock and reads
the timestamp unlocked, which is safe because _scan is only ever rebound.
Warming only when the index had never been built left a model fetched in the
Hub UI, or dropped into a scan folder, invisible for the life of the process,
since only the auto-download watcher calls invalidate_index. Warm on staleness
too, and unconditionally, so it refreshes within a TTL without a scan on the
request path. Rescanning is capped at a tenth of the scan's own duration: a big
install takes longer to scan than the TTL, and warming on the TTL alone would
keep a thread scanning continuously.
An Ollama-style tag names no quant, so the resolver misses it and auto-download
saw a model the resident one already answers to, then 404'd it for having no
such quant. Return early when the loaded model satisfies the reference.
Frontend: a cancelled download said "Model download failed", because the label
collapsed everything non-completed into failure.
The backend tests get an autouse fixture that stops the warm from walking the
developer's real HF caches; that scan starved the loop under the timing
sensitive streaming tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make /v1/models and the admission hook agree on what is local
Three review items, all on the seam between the catalog scan and the resolver
index, which run on separate schedules.
/v1/models can advertise a local GGUF the resolver has not indexed yet. A bare
id carries no quant to refuse on, so a client asking for one it had just been
handed was answered by the resident model instead. The hook now reads the
catalog cache as evidence too, never scanning it. It takes the path rather than
a yes/no because the converse also happens: the catalog can list the resident
weights under an alias the loaded entry does not answer to, and those must stay
served.
That alias was also emitted twice by /v1/models, once as the loaded basename a
manual load records and once as publisher/model marked unloaded, because the
dedup only compared ids. Compare the path as well.
A directly loaded standalone .gguf takes its quant from the filename, but the
resolver stores such files with no quants, so the advertised <stem>:<quant>
stopped resolving as soon as anything else loaded. Advertise a quant only when
that reference resolves, and downgrade only on a definite answer so a cold
index leaves the metadata alone.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten the comments this branch adds
Collapse the multi-line notes in the auto-download path, the /v1 admission
hook and their tests to one line each, keeping the reason and dropping the
restatement. No behaviour change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: four admission and catalog fixes from review
Lowercasing paths in _resolves_to_resident made /srv/models/Foo and
/srv/models/foo the same weights on any case-sensitive filesystem, so a request
for one could be answered by the other and /v1/models could mark the wrong
entry loaded. That helper now backs residency as well as admission, so use
os.path.normcase, which folds case only where the filesystem does.
Advertising a quant whenever the resolver could not disprove it kept the bug it
was meant to fix: a standalone .gguf loaded before the first scan still got
<stem>:<quant> published, and the usage examples persist that. No proof is not
proof, so omit it and warm the index instead.
A 401 from an expired or invalid X-Unsloth-HF-Token skipped the 403 and 404
branches and surfaced as "could not reach Hugging Face, retry shortly". It now
says to replace the token, kept apart from the gated refusal since a rejected
credential is not an unaccepted licence.
An image request naming an undownloaded text-only GGUF started the whole
download and only then hit the capability guard, which never sees a remote
target, so every retry 400d and the bytes were wasted. Thread require_vision
into admission and check it against the mmproj companions the disk preflight
already asks build_gguf_variant_plans for.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make the Hub error fixture carry a status on both hub majors
The 401 test built HfHubHTTPError directly, which works on 0.x and fails on 1.x
where response is required and keyword-only, so all four Python jobs failed
while the same test passed locally.
_hub_error already handled both constructors, but the 0.x branch left the
exception with no response at all, and hf_error_status reads the status off it
for the types that do not encode it in their name. So it could only produce a
usable error on 1.x, which is why the test bypassed it. Attach the status when
the constructed exception lacks it, and use the helper.
Cover the helper itself against stand-ins for both constructor shapes, since
whichever hub is installed only ever exercises one of them.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: invalidate on every download, resolve bare tags, keep polling
Three review items.
Only the API auto-download watcher dropped the resolver cache, so a GGUF
fetched in the Hub UI stayed absent to the cache-only request path and the
request was answered by whatever was resident. finalize_worker_exit is the one
point every download worker exits through, so invalidate there. That closes the
window without leaning on the TTL, which the scan-duration throttle can stretch
past 5s on an install where the scan itself takes longer than that.
A downloaded but unloaded GGUF asked for as org/model:latest missed the
resolver, since the suffix was always treated as an exact quant. With
auto-download on that probed the Hub and returned a 404 for a quant that was
never a quant; with it off it refused without switching. Fall back to the base
entry when the suffix is not quant-shaped, and keep exact matching for real
quants so a swap can never serve the wrong weights under the right name.
The usage examples stopped polling once a model was resident, but idle unload
frees one without touching the store, so nothing re-ran the effect and the
examples kept naming a model that could no longer be reloaded. Slow the poll to
60s instead of stopping it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: hold the download slot while it is in use, and keep quants to llama.cpp
_loaded_satisfies refuses a quant reference against the Transformers backend by
name, but the path match did not carry that rule. A Transformers model active
from a directory that also holds GGUF exports therefore matched a request for
one of those quants and answered it with the safetensors weights. Only
llama.cpp has a quant identity, so admission now passes llama_only whenever the
reference is quant-qualified. A bare name still matches either backend, and
/v1/models residency keeps the default so a loaded Transformers model is still
reported loaded.
The 24 hour watch window was bounding ownership of the single-flight slot when
it should only have been bounding progress reporting, so a legitimately slow
download had its slot handed back while the worker was still writing, admitting
a second multi-gigabyte download beside it. Resolve the row on the clock, but
keep the slot on a slower poll until the job is actually terminal. Past the
deadline an unknown state does release it, since it means the worker cannot be
probed and holding it on that forever would wedge auto-download.
* Studio: keep what the resolver already knew when a download lands
Invalidating cleared the index to empty. The request path reads that cache
without scanning, so from a completed download until the rebuild landed it had
no evidence about any local model, not just the new one, and a bare request for
any of them was answered by whatever was resident. Wiring the hook into the
shared completion path in the last commit widened that from auto-download to
every download.
Mark the scan stale and keep the entries instead. Both _index and
warm_index_soon rebuild on a zero stamp, while the request path still sees
everything it knew a moment ago. Only a completed download invalidates, and
that only ever adds models, so nothing retained goes false.
Warm from the completion hook too, so the rebuild starts when the download
lands rather than when the next request happens to need it.
* Studio: match the quant, not just the directory, and default-select bare tags
Two quants of one repo share a directory, so the path match could not tell them
apart and an explicit :Q8_0 was answered by a resident Q4_K_M that
_loaded_satisfies had already refused by name. The llama_only fix in the last
commit only ruled out the wrong backend, not the wrong quant on the right one.
Both path matches now require the resident hf_variant to equal the requested
quant whenever the reference is quantified; a bare name still matches on the
path alone, since it claims nothing about the weights.
The local resolver already treated a tag that names no quant as meaning the
repo, but remote admission still looked for a quant literally called "latest",
so the same reference resolved locally and 404d remotely. Branch on
looks_like_quant there too. A real quant the repo does not have is still a 404
and never a substitution, which is what separates this from the loader's
low-disk fallback.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: one quant preference, and stop trusting a stale checkpoint
list_local_gguf_variants sorts by descending size, so the head of variants was
the biggest quant, often F16, while remote admission and a plain load both rank
through _pick_best_gguf. A bare id therefore meant a different quant depending
on which side answered it, and the local answer was the one that could evict a
working model and then fail or OOM starting an F16 next to a usable Q4.
/v1/models advertised that same head for pinning. Pull the ranking into one
preferred_quant helper and have both sides use it.
The usage examples returned a stored checkpoint without ever consulting
/v1/models, and the polling added last round was gated on not having one, so
for a stored checkpoint it never ran. An idle unload then left the panel
showing a snippet that could not run. Poll whenever mounted, and prefer the
checkpoint only while the catalog still backs it or switching can reload it. A
catalog that has not answered yet is not evidence against it.
The static contract pinned the old dependency array, so it now asserts the
intent it documents: a finished load re-runs the fetch, and the effect is not
gated on having no checkpoint.
* Studio: fix the Windows path compare, and advertise a label the worker knows
The case fix normalized the separator to "/" and then called os.path.normcase,
which on Windows folds case and rewrites the separator back to a backslash, so
the descendant checks compared against a "/" the path no longer had. A manually
loaded GGUF reached through an alias then read as a different model, giving a
false 404 and an alias marked unloaded. Run normcase first and normalize the
separator after it.
There are two quant-label extractors and they only agree while a recognized
quant token is present. With none, _extract_quant_label takes the last
hyphenated segment, "7b" of llama-7b, while build_gguf_variant_plans and the
worker key the whole stem: the plan lookup missed and the job exited on a
variant it had no shards for. Use the canonical extractor for the unrecognized
case only. Checked across real filenames first, the two match on every
recognized quant and part on bpw-qualified labels, which _extract_quant_label
keeps apart on purpose so byteshape's IQ4_XS at 3.53, 3.97 and 4.19 bpw stay
separate variants.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: a stored checkpoint needs catalog evidence, not just the switch setting
Preferring it whenever switching was on short-circuited the catalog check, so a
checkpoint the store still held after the model was deleted or moved kept being
named even though /v1/models had already proved it absent, and the snippets 404d
instead of falling back to a model that is actually there.
A lookup rather than a disjunction, which settles the whole matrix in one place:
no answer yet keeps the checkpoint, since that is not evidence against it; listed
and resident keeps it; listed but unloaded keeps it only when switching can
reload it; absent falls back whatever the setting says.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: normalize the quote style pre-commit would have rewritten
* Studio: cover the model that just landed, and pin the quant the catalog has
Retaining the index on invalidation protects what was already scanned and by
construction cannot contain the model that just finished downloading, so a bare
request for it in the window before the rebuild was still answered by the
resident model. Record the repo at the completion hook and treat that as
admission evidence alongside the resolver and the catalog; the next completed
scan clears the notes, since the index then covers them. Publishing a rebuilt
index before completion becomes observable would have closed it too, but that
blocks the download worker for the length of the scan.
Catalog membership proves the repo, not the saved quant, and the examples then
pinned the stored one. A quant deleted while another quant of the same repo
remained produced repo:deleted-quant, a missing-quant 404 with a runnable
alternative listed right beside it. Pin what the catalog advertises: for a
resident entry that is the resident quant, for an unloaded one it is a quant
actually on disk. The store is only consulted before /v1/models has answered.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: apply three rules everywhere they belong, not only where reported
The trust probe was the last credential handoff still passing a raw token.
huggingface_hub reads None as "use the cached login", so a caller-named repo
was read with this server's Hugging Face identity whenever the caller sent
none, which is exactly the isolation the metadata probe and the worker already
keep. It takes _hub_token now. Enumerated the rest of that path while there:
auth_check, model_info and spawn_worker were already correct.
finalize_worker_exit is shared with dataset downloads, so the resolver hook
fired for every completed dataset, scanning the model directories for nothing
and recording the dataset id as local-model evidence, which turns a bare /v1
request naming that id into a refusal instead of a foreign-id fallthrough.
Gated on repo_type.
_already_serving decided "bare" on the presence of a colon while
_loaded_satisfies and the resolver decide it on whether the suffix names a
quant, so org/model:latest against a serving Q8_0 read as a mismatch and
swapped in the preferred Q4_K_M for a request either one answers. That rule now
lives in four places, each fixed in its own round, so this time I looked for
the rest and found a fifth: describe_local_miss splits on the bare colon and
its docstring claims it splits like the resolver. It no longer did, and would
report a missing quant named "latest". Fixed here too, unreported.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: probe before refusing busy, and scan once when the index is cold
The busy refusal fired before anything established the requested label was a
model at all, so any namespaced id a drop-in client sends was told to wait out
an unrelated download for as long as it ran. Probe first and refuse only a
label the Hub actually serves as GGUF; anything else falls through to the
resident model as before. A probe failure answers "not downloadable", since
stranding ordinary traffic costs more than missing a busy refusal.
Treating an unbuilt index as "nothing here" let the first request after startup
be answered by the resident model under another model's name. That was a
deliberate trade to keep the scan off the request path, and it was the wrong
one. Cold, the scan now runs once on a thread, bounded so a pathological
install falls through rather than hanging the request. Built, the request path
still never scans, so the latency fix stands.
The watcher freed the slot the moment it saw an error, while Retry-After is
thirty times the poll interval, so the client came back to an empty slot and
restarted the same failing download instead of being told. Hold the failure on
the slot until a retry surfaces it, and let another repo take it after three
retry intervals so a client that never returns cannot keep it.
The watcher also invalidated on completion, which now lands after
finalize_worker_exit's warm and marks that fresh scan stale, pushing a
synchronous rescan onto the retry. Removed.
_loaded_satisfies lowercased paths as well as aliases, so it returned satisfied
before the case-preserving compare below could run. Both now go through one
helper: paths compare with normcase, aliases stay case-insensitive.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: an unfinished scan is not absence, and a decided refusal is not a failure
Bounding the cold scan then reading the bound as "not here" left the same hole
one branch over. A timeout now answers 503 model_indexing with a Retry-After
and leaves the warm running. A foreign label sent inside that window is asked
to retry rather than falling through, which is a real cost, but the window is
one request on an install whose scan exceeds ten seconds and it clears itself,
where answering with the wrong weights does not.
That uncovered a worse one. Every check here runs inside a broad except whose
job is "could not verify, so fall through", so an HTTPException raised in the
block was logged as a verification failure and the request was answered by the
resident model. Any refusal decided in there was being swallowed. Re-raise it
ahead of that handler.
Canonicalizing generic labels made them real variant keys, but the matcher
still decided on shape, so repo:llama-13b fell past an exact match and fetched
llama-7b. Match exactly first, whatever the shape; a quant-shaped suffix that
matches nothing is still a miss and never a swap.
Marking a catalog alias loaded while publishing the preferred on-disk quant
claimed alias:Q4 was loaded while Q8 was serving, and requiring the resident
quant to match then made pinning it a 404. Advertise the resident variant when
the entry resolves to the resident model.
* Studio: keep the asyncio.timeout fallback tests runnable on Python 3.10
Both tests deleted asyncio.timeout to force _wall_clock_timeout down its
pre-3.11 branch, but monkeypatch.delattr raises when the attribute is already
absent. On Python 3.10, the one version the fallback exists for, there is
nothing to delete, so the two tests errored with AttributeError before reaching
the code they cover. Passing raising=False makes the deletion a no-op there and
leaves the assertions running against the same branch on every version.
Every other delattr in the repo already passes raising=False for exactly this
reason. Verified with asyncio.timeout removed from the interpreter: the two
tests fail with the CI AttributeError before this change and pass after, and
the file still runs 89 passed on 3.13 where the deletion is real.
* Studio: decide GGUF residency, servability and variant keys by one rule each
Four admission and catalog fixes, each closing a gap between two places that
were answering the same question differently.
The /v1/models catalog asked _resolves_to_resident without llama_only, so a
Transformers model live from a directory that also holds GGUF exports marked a
GGUF alias loaded and gave it a GGUF quant. The usage examples then pinned
alias:quant that nothing could serve with switching off. Every entry in that
loop is advertised as GGUF, so residency there is llama.cpp residency.
The busy probe accepted any .gguf sibling while admission excludes mmproj, MTP
drafters and big-endian builds. A repo holding only companions is not
downloadable, so it was held at model_download_busy for the length of an
unrelated download instead of falling through to the resident model as it does
when no download is running. It now reuses _gguf_variants, the same filter.
split_model_ref refused any slash-bearing suffix, but an unrecognized GGUF below
a subdirectory keys on its path (build/llama-13b), which is_valid_gguf_variant
allows and the catalog advertises. Pinning such a variant could not parse, so
only the default-ranked one was reachable. A slash-bearing suffix is now a
variant exactly when a real Hub repo precedes it, which still leaves
C:/models/x.gguf a path rather than a quant.
The usage examples treated a downloaded-but-unloaded model as runnable only
under auto-switch, but a standalone UNSLOTH_MODEL_IDLE_TTL reloads exactly what
it freed on the next request. The panel hid runnable examples after an idle
unload. Tracked apart from auto-switch, because the stash restores the stored
checkpoint only and never an arbitrary catalog entry.
Also stub the index walk in the three cold-index tests that missed it: a real
multi-root scan inside the cold-wait budget made them time out into a 503 under
load rather than assert what they are there for. One of them flaked locally.
Verified each fix is load-bearing by reverting it and watching its test fail.
Backend CI command: 10195 passed, 0 failed. tsc -b and the frontend build clean.
* Studio: bound the Hub admission probes and stop guessing at nested model paths
Three review fixes plus a test-isolation one.
_resolves_to_resident matched on a path prefix, so two separately indexed models
that nest (/models/A alongside /models/A/sub/B) both satisfied it: loading B
made a request for A resident and answered it with B's weights, and the catalog
marked A loaded. A prefix match now counts only when no catalog entry sits
deeper, which is the innermost indexed model that actually owns the file. With
nothing indexed there is no nesting to tell apart, so the directory-to-weights
match this exists for is unchanged.
auth_check and hf_hub_download take no timeout of their own, and both ran while
the provisional single-flight slot was held, so an unresponsive Hub stalled the
request far past the metadata budget and reported every other model busy for the
duration. Both are bounded now. Each default errs the safe way: an unchecked
repo is not a cleared one, so the custom-code probe refuses on timeout, while a
slow gated-repo check stays inconclusive because the download's own auth is the
real gate.
The usage examples caught a failed refresh into an empty catalog and a disabled
auto-switch, which made a transient error authoritative and blanked every
example while the model was still servable. The catalog is deliberately
tri-state; a failure now keeps the last answer and retries.
Also start the backend tests from a built, empty model index. Stubbing only the
background warm still left the cold path walking real caches synchronously
inside the admission wait, so on a large install a test asserted against a 503
"still indexing" instead of its subject. _build_index is untouched, so the tests
that call it directly still exercise the real walk.
Verified each fix is load-bearing by reverting it and watching its test fail.
tsc -b clean. Backend CI command green apart from two failures reproduced only
on this box (a real model-dir scan and an orphan-process cleanup), neither
touched by this PR; staging CI is the gate for those.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Keep the newer-mapper probe alive when the fetched mapper has no fp8 tables
_get_new_mapper reads the two fp8 tables out of the fetched mapper.py under
that file's own names, unlike the three NEW_ names it renames itself. A
mapper.py that does not define them raises KeyError, the bare except swallows
it, and the function returns five empty dicts, so the 4bit and 16bit upgrade
check stops firing as well. That check is the reason the probe exists.
Every mapper.py older than the fp8 tables is such a file: fetching the
2025-11-07 one leaves the probe with [0, 0, 0, 0, 0] instead of
[400, 997, 591]. Reading the two names with .get keeps the 4bit half working
and empties only the fp8 half, which costs nothing, since the probe runs only
after the installed tables have already missed.
Add a regression test that also pins the fetched-only fp8 upgrade error, which
the existing test cannot catch: it serves the repo's own mapper.py as both the
installed and the fetched source, so any fresh dict satisfies its identity
assertions.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: admission control on /v1/messages, slot pool that tracks --parallel
/v1/chat/completions was gated by the llama admission queue but /v1/messages
was not, so an Anthropic client could oversubscribe llama-server's slots and
stall the backend. Wire the same queue into all six /v1/messages dispatch
sites, and rework the queue itself into an explicit slot pool.
- Queue keyed by base_url, so both API surfaces share one pool of slots.
- Waiting is unbounded by default instead of timing out; the wait line is
sized at 16 x the serving slots so it follows --parallel.
- Neutral UNSLOTH_LLAMA_ADMISSION_* env names, legacy UNSLOTH_OPENAI_COMPAT_*
spellings still honored.
- Passthrough retries once against a respawned llama-server, which comes back
on a new ephemeral port.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix over-admission on capacity shrink and restore the stream cancel contract
Review of the previous commit turned up two real regressions plus smaller gaps.
- Pool sizing looked only at free slot ids, so when capacity shrank while slots
were held (an unload resets effective_parallel_slots to 1) a freed low id was
handed out even though the holdovers already met the new ceiling. Count every
held slot against capacity instead. A 1-slot backend could run 4 generations.
- The streaming wrapper closed the monitored body with aclose(), delivering
GeneratorExit where _SameTaskStreamingResponse deliberately throws
CancelledError. The monitor entry was never finalized, so it leaked as
"running" for the process lifetime and cancel_event was never set. Close
through the shared helper so cancellation reaches the handler.
- Finalize the monitor when a stream is abandoned before its body starts, and
when a queued non-streaming request is cancelled (which also leaked the
un-awaited generation coroutine).
- Floor the scaled wait line at 64, so a 1-slot backend keeps the depth it had
before scaling existed instead of dropping from 64 to 16.
- Use the canonical Anthropic type map: a full queue is 429 rate_limit_error,
which SDKs back off on; overloaded_error is 529.
- Treat non-positive max_queue/queue_per_slot as unbounded rather than "reject
everything", and reclaim the slot if a waiter's event loop is gone.
Tests: regression tests for both defects, verified to fail without the fix.
Adds env coverage for QUEUE_PER_SLOT and the legacy fallbacks, a structural
check that all six dispatch sites stay admission-wrapped, and clears the new
env var in the isolation fixtures.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Run the response pre-start cleanup when a queued stream is abandoned
From automated review of the earlier commits.
_anthropic_passthrough_stream enters its _TrackedCancel eagerly and relies on
the stream's finally to exit it, but aclose() on an async generator that never
started is a no-op, so that finally never runs. Admission made this reachable:
a client that disconnects while queued leaves the cancel id registered in
_CANCEL_REGISTRY forever.
- Give the passthrough response an unstarted_cleanup hook that exits the
tracker, via a new optional arg on _sse_streaming_response.
- Chain to that hook from the admission wrapper rather than replacing it, and
run it when the wrapper gives up before the body started.
- Defer to an in-progress MTP fallback instead of respawning underneath it;
only the first caller gets True from _maybe_recover_from_mtp_crash.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Restore Python 3.9 support, and stop the floor overriding an explicit setting
Second review round. The first item is a real break shipped by the earlier
commits, the rest are correctness and contract fixes.
- dataclass(slots = True) and int.bit_count() are both 3.10+, but the package
declares requires-python >=3.9 and CI only runs 3.12, so nothing caught it.
Importing the module raised TypeError on 3.9, taking down the whole backend,
not just admission. Drop the dataclass slots and track the popcount in a
counter. A test now asserts neither API comes back.
- The queue-depth floor applied even when an operator set QUEUE_PER_SLOT
explicitly, so asking for a shallow line silently got 64 and, with no queue
timeout, callers blocked instead of failing fast. The floor now only backs
the default multiplier.
- Never let a failing close strand a slot: closing runs in its own try so the
release always happens. A lost slot shrinks the pool permanently.
- Close the generation coroutine when reserving fails for any reason, not only
on a full queue.
- Exit the passthrough cancel tracker if the client drops while the opening SSE
lines are still being sent; those yields sit outside the teardown try.
- snapshot.free now reports what a caller could actually take, so the admission
log cannot show free slots next to queued requests after a shrink.
- Correct the class docstring: the wait line is bounded by default, not
unlimited. Document that abandoning wait() requires cancel(), and pin the
thread assumption in _deliver_lease.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make the leak guards real, and cover the untested admission branches
Third review round, which attacked the previous round's tests by reverting each
fix. Two guards turned out to be hollow.
- The pre-start cleanup chain could be severed with the suite still green: the
existing test drove the generator finally, never the response hook. A real
pre-start disconnect leaked the passthrough cancel tracker permanently.
Replaced with a test that runs the response hook and asserts _CANCEL_REGISTRY
is empty; verified against both ways of reintroducing the leak.
- The structural check only asserted the unstarted_cleanup keyword was present,
so passing a literal None passed it while leaking. It now asserts the hook is
actually built.
- test_shares_queue_with_openai_by_base_url never touched the OpenAI helper; it
was a duplicate under a misleading name. It now reserves through the same
helper /v1/chat/completions uses, so it fails if either surface ever derives a
different key. That is the PR's central shared-queue claim.
- Cover the passthrough dispatch site, 499 on disconnect-while-queued, and the
streaming admission timeout. Four of six sites previously had only an AST
node count behind them.
- Clear admission env in the autouse fixture rather than per test: an ambient
canonical name silently beat the legacy name a test was exercising.
- Loosen the wall-clock assertion, which guarded against serialising on the
uncontended path, not against a slow runner.
- The class docstring claimed a global concurrency cap; Studio's own chat
endpoint does not reserve, so it is not one.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep dataclass slots on 3.10+ via a version gate
Dropping slots = True for 3.9 gave it up everywhere, including the 3.12 CI runs
and every supported interpreter but one. Gate it instead: _SLOTS is
{"slots": True} on 3.10+ and empty below, unpacked into each dataclass.
The AST scan now requires the unpack rather than merely forbidding a literal
slots keyword, so a dataclass added later cannot quietly lose slots. Added a
test that the gate matches the running interpreter, since a gate that never
applies is worse than no gate. Verified the 3.9 branch by forcing _SLOTS empty
and reloading: the full admission suite passes either way.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix two slot leaks, and cover the guards that had no test
Third review round, three reviewers working independently on the admission core,
the route wiring, and whether the PR regresses anything it is not about.
Leaks:
- cancel() made the same call_soon_threadsafe as _grant_waiters_locked but
without its RuntimeError guard. Routes cancel from finally blocks, so a closed
loop masked their exception and skipped the release, stranding the slot and
pinning is_idle() false so the queue was never evicted either.
- The pre-start cleanup released the slot after an await that can raise
BaseException, which is swallowed upstream. Nested it in a finally, as the
streaming and OpenAI paths already do.
An unparseable QUEUE_PER_SLOT dropped the 64 floor while falling back to the
default multiplier, quietly giving a 1-slot backend a 16-deep line. Explicit now
means it parsed.
Guards that were correct but had no test. Each was reverted, confirmed the suite
stayed green, then covered and confirmed red:
- the slot released when stream setup raises, which is the reachable one:
count_chat_tokens is a blocking call to llama-server, so a dead backend raises
after the slot is taken and before a body exists to release it
- coro.close() on a cancelled queued request, the api_monitor.fail that
distinguishes an admission timeout from a client hang-up, the MTP fallback
short-circuit, and the BaseException guard around the opening stream lines
- the queue-full test asserted a type string OpenAI's 429 also uses, so it
passed against an OpenAI envelope. It now pins the Anthropic shape.
Anthropic requests were invisible in the admission log while sharing the pool
with chat completions, so the same events are logged there with a mode. Renamed
the helper to match, since it is no longer OpenAI-only.
Corrected two comments that described behaviour the code does not have: the
slot is taken when the streaming response is built, not when the body starts
iterating, and the pool is not a cap on every generation, since /v1/completions,
Studio's chat endpoint and RAG captioning all reach llama-server directly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Cover the admission telemetry, and drop a dead helper
Fourth review round. No bugs found in the code this time; the finding was that
most of the previous commit's telemetry had no test. Only queue-full was
asserted, so removing any of the other four log calls left the suite green.
All five are covered now, each verified by removing its call and confirming only
its own test reds. Fixing the first attempt turned up a test bug of my own: the
log line carries a queued=N field, so asserting "queued" in the message matched
every admission log ever emitted. It asserts the event name now.
Also covered two guards that were correct but unguarded: waiters whose futures
die out of band stop counting against the queue depth, and a newcomer cannot
barge past a parked waiter. The second is pinned as behaviour rather than as the
`if not self._waiters` check, because that check cannot actually change the
outcome: _take_slot_locked consults _can_admit_locked anyway, so either alone
refuses the newcomer. The test fails only if both go.
_optional_positive_int_env lost its last caller when the env parsing was
rewritten last round. Removed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Keep the newer-mapper probe from replacing the installed FP8 mappers
get_model_name calls _get_new_mapper() whenever a name misses the local
tables, only to answer whether a newer Unsloth would support it. That
helper fetches mapper.py from main, prefixes INT_TO_FLOAT_MAPPER,
FLOAT_TO_INT_MAPPER and MAP_TO_UNSLOTH_16bit with NEW_, and execs the
result into globals().
The slice starts at __INT_TO_FLOAT_MAPPER, so it also carries
FLOAT_TO_FP8_BLOCK_MAPPER, FLOAT_TO_FP8_ROW_MAPPER, the _add_* helpers
and the builder's loop variables, and none of those are renamed.
Exec'ing into globals() therefore rebinds the two FP8 tables that
loader_utils imported from the installed mapper, so every later
get_model_name(..., load_in_fp8 = ...) in the process resolves through
main's table instead of the installed one. The probe deliberately does
not adopt the new 4bit mappers (it raises NotImplementedError asking the
user to upgrade), so silently adopting the new FP8 ones is inconsistent,
and it also leaves loader_utils and mapper disagreeing about the same
tables. Reaching it needs nothing unusual: any org/model name absent
from the tables triggers the fetch.
Exec into a throwaway namespace and read the three mappers out of it, so
the probe stays a read and the installed mappings are left alone.
Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com>
* Hand the fetched FP8 tables back from the probe instead of dropping them
Isolating the exec stopped the probe corrupting the installed FP8 tables, but
it also removed the only reason the probe ever saw the fetched ones: the
_resolve_with_mappers call still read FLOAT_TO_FP8_BLOCK_MAPPER and
FLOAT_TO_FP8_ROW_MAPPER off the module globals. A newly added FP8 repo would
then miss both the installed tables and the probe, so an older install would
stop raising the upgrade NotImplementedError for it.
Return the two fetched tables and let _resolve_with_mappers take them as
optional arguments, defaulting to the installed ones. The probe now answers
for new FP8 repos without writing over what the installed version resolves.
_get_new_mapper returns five tables now, so the two existing stubs in
test_get_model_name.py and test_bad_mappings_redirect.py are updated to match.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
JoshuaL3000