Adds a configurable Hugging Face model download cache location to Unsloth Studio, selectable from Settings, with per-cache download manifests, scoped deletion, and read-only inventory of previously selected caches.
* Studio: reject whitespace-only passwords
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: reject any whitespace in passwords
* Studio: surface whitespace error in setup form, isolate auth test import
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* studio: classify embedding models from the HF cache and honor offline mode
is_embedding_model() went straight to huggingface_hub.model_info() for any repo
id, so in offline mode (no DNS, or HF_HUB_OFFLINE set) selecting an
already-downloaded model hung on network retries that could never succeed and
training/export never started (#6817).
Check the local HF cache first: a sentence-transformers repo carries
modules.json in its snapshot (the same marker used for local paths), so a cached
model is classified with no network call. When HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE
is set, anything not positively an embedding model returns False without a
network call instead of retrying a doomed request. Online, uncached lookups still
fall through to model_info(), so tag-only embedding models (feature-extraction)
are unaffected.
Adds _embedding_marker_in_hf_cache() over the existing _iter_hf_cache_snapshots.
* studio: judge the active cached revision, harden the cache probe, stop stub leaks
Three review fixes on the cache-first embedding detection:
1. Prefer the revision refs/main resolves to. The HF cache keeps snapshots of
older revisions, so an any-snapshot scan could classify a repo by a stale
revision -- e.g. a repo that used to be a sentence-transformers model would
short-circuit even the online lookup. When refs/main is recorded, only its
snapshot is consulted; the newest-first scan remains the fallback for caches
with no ref.
2. Keep the cache probe inside the detection error boundary. The snapshot
iterator stat()s entries and could raise if a cached model is deleted
concurrently, propagating a 500 out of the config/check-embedding routes.
_embedding_marker_in_hf_cache now catches everything and reads as
not-cached, so callers keep their normal Hub/offline fallback.
3. Stub loggers/structlog in the test only when the real modules are absent
(try-import, mirroring test_windows_gpu_detection_mock), so collecting this
file first can no longer shadow the real packages for later tests in the
same pytest process.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: treat a missing active-ref snapshot as a cache miss, don't cache offline misses
Two review fixes on the cache-first embedding detection:
1. When refs/main is recorded but points at a commit whose snapshot dir is
absent (partial download / cache pruning), the recorded ref is still
authoritative: return None (cache miss) instead of falling through to scan
older snapshots, which could report a stale historical revision's
modules.json as the active one -- the same stale-cache class this helper
avoids.
2. Do not cache the offline negative. When HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE
is set and the repo is not positively an ST model from modules.json,
is_embedding_model stored False under the (model_name, hf_token) key shared
with online lookups; after the env var cleared in the same process, a
tag-only (feature-extraction) embedder returned the cached False and never
reached model_info(). The offline negative is now returned without caching.
* studio: defer online embedding detection to the Hub, re-probe offline
The local modules.json marker short-circuited is_embedding_model() even
online, so a repo that dropped (or added) the marker since it was cached
was judged by its stale local revision instead of the current remote one.
Online now treats model_info() as authoritative and uses the cache marker
only as an uncached fallback when the Hub is unreachable, so a transient
failure never poisons the memo. Offline re-probes the marker on every call
without consulting or populating the memo, so a model downloaded later in
the session (or a cached online negative that predates the download) is
detected. _embedding_marker_in_hf_cache() now treats an unreadable refs/main
(a non-FileNotFoundError OSError) as a cache miss rather than scanning stale
history -- only a genuinely missing ref enables the fallback scan.
* studio: harden offline embedding detection against empty refs, offline flips, and cache casing
- _embedding_marker_in_hf_cache: an existing-but-empty/whitespace refs/main
(a partial write or in-progress truncate-and-rewrite) now reads as a cache
miss (None) instead of falling through to scan stale snapshots; only a
genuinely missing ref enables the historical scan.
- is_embedding_model: while offline, retain a positive already confirmed online
this session (model_info only ever memoizes Hub-derived results), so
_hf_offline_if_dns_dead() flipping the process to offline mid-load can't
downgrade a verified tag-only embedder to False. Cached negatives are still
bypassed and re-probed.
- resolve_cached_repo_casing + settings route: persist the embedding model in
the casing its local HF cache dir uses. Validation accepts a case-insensitive
cache hit, but an offline SentenceTransformer load resolves the cache by exact
case, so storing the requested spelling (baai/bge-m3 vs models--BAAI--bge-m3)
made the model fail to load on a case-sensitive filesystem.
* studio: reuse the exact-match-first case resolver and preserve the default
Replace the ad-hoc resolve_cached_repo_casing with the existing
resolve_cached_repo_id_case, which already prefers the exact-case cache dir
before any case variant and tie-breaks variants deterministically -- so an
exact requested id is never rewritten to a differently cased directory just
because iterdir() happened to yield it first.
Skip the normalization entirely when the submitted model equals the default:
rewriting its casing would make set_rag_embedding_model()'s exact-string
default comparison treat it as a custom override, pinning it so later changes
to the configured default stop taking effect.
* studio: don't let a stale cache marker mask a permanent Hub error
is_embedding_model's Hub-failure fallback consulted the local modules.json
marker for ANY model_info() exception, so a permanent error -- a deleted repo,
a gated repo without credentials, or a typo that matches stale cache casing --
could pass online validation on a stale marker instead of returning the
documented 409, and the persisted model could then fail when the loader
refreshes from the Hub. Classify permanent Hub errors (RepositoryNotFound,
GatedRepo, RevisionNotFound, EntryNotFound) as False, matching the nearby
GGUF/vision detectors, and reserve the cache fallback for transient/5xx failures.
* studio: honor TRANSFORMERS_OFFLINE in the embedding preflight, skip casing for local paths
- The embedding-model save reached the offline-aware is_embedding_model() only
after two preflight helpers made direct huggingface_hub calls that honor just
HF_HUB_OFFLINE: _st_module_subdirs() downloads modules.json and the security
scan fetches Hub metadata twice. In a TRANSFORMERS_OFFLINE-only session those
blocked on network timeouts before the offline return, so saving an already
cached model stalled. Both now consult a canonical hf_env_offline() helper --
the download passes local_files_only, and the metadata-only security scan
short-circuits to its documented fail-open instead of burning both timeouts.
- Skip cache-casing normalization for local paths: a relative directory such as
"org/model" is loaded from disk, so rewriting it to a case-insensitive HF
cache collision ("Org/model") would stop resolving to that directory and be
read as a Hub repo id instead.
* studio: never skip the security scan on TRANSFORMERS_OFFLINE alone
The previous commit skipped the Hub security scan whenever either offline flag
was set, but huggingface_hub honors only HF_HUB_OFFLINE: under a
TRANSFORMERS_OFFLINE-only session the later SentenceTransformer load still
reaches the network, so the scan was being skipped while the repo's pickle could
still be downloaded and deserialized -- waving through exactly what
_guard_model_security exists to block.
Split the flags: hf_hub_offline() (HF_HUB_OFFLINE, the only one that actually
prevents a fetch) gates the security short-circuit, while hf_env_offline()
(either flag, the user's intent) is used only where local-only behavior is
forced explicitly. The SentenceTransformer load now passes local_files_only from
that intent, so TRANSFORMERS_OFFLINE genuinely stops the loader fetching instead
of merely being assumed to.
* studio: short-circuit the security preflight under either offline flag
With the loader now pinned to the local cache by local_files_only =
hf_env_offline(), a TRANSFORMERS_OFFLINE-only session can no longer fetch
anything -- yet the preflight still fell through to two model_info() attempts on
10s and 20s timeouts, stalling every save and load of an already-cached embedder
for half a minute before failing open anyway.
Skip the metadata-only scan whenever either flag is set. The scan's job is to
stop a poisoned pickle being downloaded and deserialized, and nothing can be
downloaded under that predicate; the residual case -- a model cached BEFORE it
was flagged -- is the same fail-open this function has always documented for an
unavailable scan, and is exactly what HF_HUB_OFFLINE already did.
That safety argument depends on every loader behind the gate honoring the same
predicate, so it is pinned as a test invariant instead of a comment: removing
local_files_only from the SentenceTransformer construction now fails the suite.
Drops the short-lived hf_hub_offline() helper, which no longer has a caller.
* studio: scope the offline scan bypass to callers that load local-only
The previous commit put the offline short-circuit inside _fetch_security_status,
which is the malware gate shared by every loader -- so TRANSFORMERS_OFFLINE=1
disabled it for all of them, while only the RAG embedder had been changed to
pass local_files_only. MLX inference (core/inference/worker.py -> FastMLXModel
.from_pretrained), training and export call from_pretrained with no local-only
argument, and huggingface_hub ignores that flag, so those paths could still
fetch and deserialize an unscanned model with the gate switched off.
The bypass is now an explicit local_only_load argument, defaulting to False, and
only the two RAG embedding callers -- whose loader is pinned to the local cache
by the same predicate -- opt in. Tests pin both halves: the shared gate must
still scan under either offline flag by default, and no other caller may pass
local_only_load without constraining its loader.
* studio: capture offline state once, and probe the ST cache root
Two holes in the offline embedding path:
- _get() read hf_env_offline() twice: once inside _guard_model_security and
again for local_files_only. _hf_offline_if_dns_dead() mutates the process-wide
offline vars and restores them on exit, so a concurrent load could see True in
the guard -- skipping the Hub malware scan -- and False by the time the
constructor ran, fetching and deserializing the unscanned repo and breaking
the very invariant that licenses the bypass. The value is now read once in
_get() and passed to both; _guard_model_security takes it as an argument
instead of re-deriving it.
- The cache probe searched only HF_HUB_CACHE. SentenceTransformer downloads into
SENTENCE_TRANSFORMERS_HOME when that is set, using the same
models--org--name/snapshots layout under a different root, so a model fully
present there looked uncached and was rejected with a 409 offline even though
the local-only loader could load it. Snapshot lookup now covers both roots.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: probe the cache the ST loader actually uses, and require it be loadable
Adding SENTENCE_TRANSFORMERS_HOME to the shared snapshot iterator was too broad
in one direction and too narrow in another:
- _get() builds SentenceTransformer with no cache_folder, so with ST_HOME set it
searches THAT root only, never the Hub cache. Probing the union let offline
validation pass on a repo cached only in the Hub cache, after which the loader
looked in ST_HOME and failed. The Sentence-Transformers probe now resolves to
exactly one root: ST_HOME when set, the Hub cache otherwise.
- The shared iterator is also used by the GGUF detectors, whose downloads go
through hf_hub_download with no cache_dir and therefore really do use the Hub
cache. It is back to Hub-cache-only so detection cannot pick a snapshot the
GGUF load will not find.
- Casing normalization ran through resolve_cached_repo_id_case, which scans the
Hub cache, so with ST_HOME set the requested spelling was persisted unchanged
and the exact-case offline load missed the differently cased directory that
detection had just accepted. It now resolves against the same roots detection
uses, exact match first.
- A snapshot carrying only modules.json no longer counts as cached: the online
security preflight downloads that single file itself, and a partial download
leaves it behind, so validation passed for a snapshot with no weights and the
first RAG load then failed. A hit now requires the marker plus a config and at
least one weight file.
* studio: thread the captured offline state into the module probe, fix the gate shard
- _st_module_subdirs() re-read the process env for its local_files_only. With
_hf_offline_if_dns_dead() flipping those vars from another thread, a load that
captured local_only=False could still force this probe local-only, get () back
because modules.json is not cached, and leave the scan with NO module load
roots -- a Hub-flagged pickle under 0_Transformer/ would then pass as an
unreferenced nested artifact while the loader fetched and deserialized it. It
now takes the captured predicate as an argument, and the settings route reads
the state once and uses that single value for both the probe and the scan.
- Skip ST-cache casing on the llama-server backend. Nothing there loads through
SentenceTransformer: the embedder derives a GGUF companion from the saved
spelling and fetches it from the HUB cache, so normalizing to an ST_HOME
spelling would point it at a repo _hf_gguf_backend_error() never validated
(BAAI/bge-m3-GGUF instead of the checked baai/bge-m3-GGUF).
- Fix the security-gate shard, which the signature change had broken: the direct
_guard_model_security / _st_module_subdirs callers now pass the new argument
(they were raising TypeError before reaching any assertion), and the casing
tests patch utils.models.resolve_st_cached_repo_id_case, which the route
actually calls, instead of the Hub-only resolver it no longer uses -- those
patches were being silently ignored.
* studio: accept only torch-loadable weights in the offline ST probe; fix re-export lint
_snapshot_is_loadable_st_model accepted a cached snapshot whose only weights
were .onnx (or .pt), but the RAG loader builds SentenceTransformer with the
default torch backend, so such a snapshot passed offline validation and then
failed on the first load, the exact validate-then-fail this helper exists to
prevent. Restrict _ST_WEIGHT_SUFFIXES to .safetensors and .bin and add a
regression test for an ONNX-only snapshot.
Also teach scripts/verify_import_hoist.py that names listed in a module-level
__all__ are uses, so the legitimately added resolve_st_cached_repo_id_case
re-export in utils/models/__init__.py no longer trips HOISTED-IMPORT-UNUSED.
Covered by two new self-test cases.
* studio: probe the exact repo dir and revision an offline load resolves
The cache probe modelled the cache loosely rather than modelling what
SentenceTransformer actually does with local_files_only=True:
- It merged snapshots across every case-variant repo dir and then read refs/main
from whichever held the newest one. With both models--baai--bge-m3 and
models--BAAI--bge-m3 present, a complete embedding snapshot in the directory
the loader opens could be judged by a newer partial snapshot in the other,
failing validation for a usable model. It now selects the ONE directory the
loader opens, by the same exact-case-first rule resolve_st_cached_repo_id_case
uses to choose the spelling that gets persisted.
- It fell back to scanning historical snapshots when refs/main was absent. With
local_files_only the default revision is resolved THROUGH that ref, so a
snapshot directory alone is not discoverable: the settings request succeeded
and the loader then failed at first indexing. A missing, empty or unreadable
ref is now a cache miss, and the historical scan is gone.
The tests exercise the real lookup against a built cache tree instead of
patching the snapshot iterator, so they now cover the directory selection and
ref resolution the loader depends on.
* studio: record refs/main in the ONNX-only probe test
The ONNX-only regression test predates the refs/main requirement, so after that
change it returned None (a cache miss for want of a ref) before ever reaching
the weight-format check it exists to make. Recording the ref restores its
intent: the snapshot resolves, and the answer is False because an ONNX export is
not loadable by the RAG loader's default Torch backend.
* studio: recognize base-model weight files and gate the offline positive on a materialized snapshot
_snapshot_is_loadable_st_model matched any .safetensors/.bin by suffix, so a
partial cache carrying only a commonly published non-weight bin such as
training_args.bin (or an adapter-only artifact) passed offline validation and
then failed the local_files_only load at first indexing. Match recognized Torch
base-model weight filenames (model / pytorch_model, including sharded) by name.
is_embedding_model retained an online-confirmed positive offline even when no
files were cached, so a metadata-only /check-embedding result let an uncached
repo be saved and then fail at first indexing. Retain the positive only when the
active revision is materialized locally, which still covers a downloaded tag-only
embedder whose snapshot carries no modules.json.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: require a complete weight set offline and persist embedder verdicts across restarts
Two follow-ups to the offline embedding-model classifier:
- _snapshot_is_loadable_st_model now requires a COMPLETE Torch base-model
weight set in one snapshot directory, not just any single recognized weight
file. A partially downloaded sharded model (model-00001-of-00002 without its
sibling) no longer passes offline validation and then fails at first indexing
under local_files_only. Weight files are grouped by directory and a directory
counts only when it holds a single model.safetensors / pytorch_model.bin or a
full shard set whose indices cover 1..total.
- Online-confirmed embedder verdicts are now recorded under the resolved Studio
home (embedding_verdicts.json). The session memo is lost on exit, so a
downloaded tag-only feature-extraction embedder (snapshot present but no
modules.json) was misclassified as non-embedding the first offline call after
a restart. The offline branch consults this durable allowlist in addition to
the memo, still gated on the active revision being materialized on disk, so an
uncached repo is never trusted. Writes are best-effort and only positive
verdicts are stored.
* studio: require complete weights (with shard index) and resolve default casing offline
Follow-ups to the offline embedding-model classifier from the latest review:
- Trust a recorded embedder verdict (session memo or persisted allowlist) offline
only when the active snapshot carries a COMPLETE, loadable weight set, not merely
that it is materialized. A partial download (config present, weights missing or an
incomplete shard set) makes _embedding_marker_in_hf_cache read False rather than
None, so the previous marker-is-not-None gate wrongly returned True and the
local_files_only load then failed. Split out _snapshot_has_complete_weights (config
plus complete weights, modules.json aside) and _active_snapshot_dir, and gate the
known-embedder positive on the weight set.
- Require a sharded checkpoint's index map (model.safetensors.index.json /
pytorch_model.bin.index.json) in addition to every shard before accepting it:
transformers discovers and wires shards through that index, so a complete shard set
without it fails the local-only load.
- Resolve the embedding model name to its exact cache casing in the RAG loader before
constructing SentenceTransformer. The settings route persists that spelling for a
custom override but deliberately leaves the configured default verbatim, so a
default whose casing differs from the cache dir would miss it and fail offline.
Resolving at load time covers the default too; a no-op for a local path or when
nothing case-matching is cached, and idempotent for an already-normalized override.
Adds regression tests for the partial-snapshot verdict, the missing shard index, and
the loader casing resolution; updates the offline-invariant source assertion to the
resolved-name variable.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: require a tokenizer, case-fold verdict ids, and serialize verdict writes
Three follow-ups to the offline embedding-model classifier from the latest review:
- _snapshot_has_complete_weights now also requires a tokenizer asset. A
SentenceTransformer Transformer module builds an AutoTokenizer, so a snapshot with
a complete weight set but no tokenizer.json / tokenizer_config.json / vocab still
fails the local_files_only load. The check is a permissive union over the common
fast-tokenizer, config, and WordPiece/BPE/SentencePiece assets, so an unusual but
valid layout is not rejected -- only a genuinely tokenizer-less partial download.
- The persisted embedder allowlist is now keyed case-insensitively. model_info() is
queried under the requested casing while the settings route saves the cache-resolved
casing, so an exact-string lookup missed the persisted positive after a restart
(baai/model recorded, BAAI/model looked up) and a loadable tag-only embedder was
rejected. Both persist and lookup case-fold the id.
- _persist_embedder serializes its read-modify-write under a lock and writes through a
per-thread temp file, so concurrent confirmations of different embedders no longer
drop each other's entry or collide on the temp path. Cross-process writers stay
best-effort (os.replace is atomic; a dropped verdict is only an optimization miss a
later online re-confirmation heals).
Adds regression tests for the missing-tokenizer reject, alternate tokenizer assets,
cross-casing verdict match, and concurrent verdict writes; updates the snapshot test
helpers to materialize a tokenizer alongside config and weights.
* studio: tighten comments in the offline embedding-model classifier
Comment-only pass over the PR's changed files. Collapse the long block
comments and docstrings around is_embedding_model, the cache-snapshot and
weight-completeness helpers, the embedder-verdict persistence, the offline
security gate, and the offline/casing tests to short one- or two-line forms.
Preserve the rationale (issue #6817, the local_files_only invariant, the
casing and weight-gate reasons) in far fewer words. No code changes.
* studio: drop redundant comments in the offline embedding-model classifier
Second comment-reduction pass over the offline embedding-model cache work:
delete comments and trailing notes that restate the adjacent code or an
assertion, and trim the remaining docstrings and rationale comments to their
load-bearing invariants. Comments and docstrings only; no code changes.
* studio: pin embedder verdicts to a revision, canonicalize default aliases
- A persisted verdict recorded that the Hub tagged ONE revision an embedder, but
was stored per repo. Once refs/main advanced to a complete but non-embedding
Transformer snapshot, the offline path still returned True: the settings route
accepted the updated model without force and RAG could silently load it as an
embedder. Verdicts now carry the commit they were confirmed at and are trusted
only while the active revision matches. One confirmed before the repo was
cached has no revision to compare, so the first revision observed afterwards is
pinned then -- which is what lets a later advance be caught. The persisted file
gains a {id: commit} form and still reads the previous list format.
- tokenizer_config.json no longer counts as a tokenizer asset. It only DESCRIBES
a tokenizer, so a snapshot with config, weights and just that file passed
validation and then failed AutoTokenizer.from_pretrained(local_files_only=True)
at first indexing for common BERT/GPT-style models.
- A casing-only alias of the default is canonicalized to the default up front.
Repo ids are case-insensitive but every gate here compares exact strings, so
saving "Unsloth/bge-m3" against a default of "unsloth/bge-m3" ran the
verification and scan for a custom model and then persisted an override --
after which later changes to the configured default stopped applying.
- verify_import_hoist.py replays __all__ assignments in order instead of unioning
them. Only the final value exports anything, so a later plain "=" that drops a
name must leave its import counted as unused; "+=" still extends, and an
unreadable rebind keeps the earlier names rather than flagging real re-exports.
* studio: validate the real ST load root, and pin verdicts to the Hub revision
Four ways the offline probe still disagreed with what the loader does:
- Verdicts were pinned to the LOCAL refs/main, but model_info() describes the
current HUB revision. With a stale cache the two differ, so an older snapshot
nobody verified was allowlisted. The pin is now info.sha, taken from the
ModelInfo that produced the positive. A verdict carrying no revision (a legacy
entry) is no longer trusted at all -- trusting it meant pinning whatever
happened to be cached, which is the same bug; the next online check re-records
it properly.
- config, tokenizer and weights had to exist somewhere in the snapshot, not
together. modules.json can send SentenceTransformer at 0_Transformer/, which is
loaded FROM that directory, so a cache with the config at the root and only
0_Transformer/model.safetensors passed and then failed the local-only load.
Each directory is now checked as a complete load root, which covers both the
plain HF layout and the ST module layout.
- vocab.json and merges.txt counted independently, but BPE needs the pair unless
a serialized tokenizer.json is present, so half a pair validated and then
failed AutoTokenizer.from_pretrained(local_files_only=True).
- A slashless short name like all-MiniLM-L6-v2 is a supported ST alias that the
loader resolves through the sentence-transformers/ organization, so its
snapshot is cached under that full id. Probing only the bare name reported a
miss and 409'd a model that was cached and loadable; the bare id is still tried
first, matching the loader's own order.
* studio: fail closed for an offline security scan instead of failing open
A local_only (offline) load cannot fetch Hugging Face's malware scan, and the previous
behaviour skipped the scan and failed OPEN, so a cached repo with a poisoned pickle weight
could deserialize under SentenceTransformer(local_files_only=True). Evaluate it fail-CLOSED
against the cached files instead: block a base-model pickle weight the load would deserialize
(pytorch_model.bin and its shards, in a directory with no safetensors alternative) and allow a
pickle-free (safetensors / gguf are inert) cache. A cached pickle model must be reloaded online
once to be scanned, or shipped as safetensors. Nothing cached is not a security event.
_fetch_security_status no longer needs the local_only_load skip (the offline branch is handled
in evaluate_file_security). Adds a regression test covering the safetensors-allow and
pickle-block paths with no Hub call.
* studio: only suppress an offline pickle when a loadable safetensors weight exists
The offline security gate treated any .safetensors in a directory as covering a
pickle weight, so a cache with pytorch_model.bin beside a bare adapter_model.safetensors
(or an orphan shard with no index) passed the fail-closed check even though
from_pretrained still selects and deserializes the pickle. Require a genuinely loadable
safetensors weight -- an unsharded base file or a complete indexed shard set -- before
treating the pickle as covered.
Also make the import-hoist analyzer preserve uncertainty when __all__ is extended by a
value it cannot read statically (__all__ += dynamic()), matching how it already handles
an unreadable rebind, so a dynamically-supplied re-export is not flagged HOISTED-IMPORT-UNUSED.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: scope the offline pickle scan to load paths; reset __all__ opacity on rebind
Address three review follow-ups on the offline security gate and the import-hoist analyzer:
- The offline pickle scan walked the whole snapshot, so a stray pickle in a non-load
subdirectory (archive/, nemo/) that SentenceTransformer never deserializes was blocked.
Scope it to real from_pretrained load roots -- the snapshot root, or a subdir that holds
its own config.json -- matching the online scan's load-path scoping.
- _collect_dunder_all kept a sticky opaque flag: a readable replacing assignment after an
unreadable extend (__all__ += dynamic(); __all__ = []) still credited every import, so a
genuinely unused hoist went unreported. A replacing assignment now resets opacity.
- A bare __all__: list[str] annotation has no runtime value; it was treated as an unreadable
assignment and marked the export set opaque. Skip annotation-only declarations.
* studio: recase slashless ST aliases and accept a pinned embedder after a transient failure
Two offline-detection gaps on well-formed input:
- resolve_st_cached_repo_id_case bailed on every slashless name, so a differently-cased
short alias (all-minilm-l6-v2) validated case-insensitively but was loaded verbatim; the
SentenceTransformer loader rewrites it to sentence-transformers/all-minilm-l6-v2 and looks
it up case-sensitively, missing the canonical sentence-transformers/all-MiniLM-L6-v2 cache
dir. Resolve through _st_cache_repo_dir, which follows the same org alias, and hand back the
on-disk casing.
- On a transient (non-permanent) Hub failure, is_embedding_model only accepted a cached
modules.json marker, so a downloaded tag-only embedder (no modules.json) with a verdict
pinned to the active revision was rejected even though the offline branch accepts the
identical cache. Mirror the offline branch's pinned-verdict acceptance.
* studio: scan modules.json-declared module roots in the offline pickle gate
The offline pickle scan treated only the snapshot root and config.json-bearing subdirs as
load roots, so a pickle in a non-Transformer SentenceTransformer module directory that has no
config.json (e.g. a 0_WordEmbeddings/ module: wordembedding_config.json + pytorch_model.bin)
was skipped even though the loader deserializes it. Parse modules.json (and thread through
load_subdirs) to treat every declared module directory as a load root, so such a pickle is
scanned and fail-closed offline.
* studio: classify cached non-Transformer SentenceTransformer models offline
_snapshot_has_complete_weights recognized only a Transformer-shaped load root (config +
tokenizer + weights co-located), so a fully-cached model built from a non-Transformer module
(0_WordEmbeddings uses wordembedding_config.json + embedding weights and its own tokenizer, no
HF config.json; BoW keeps its vocab in config.json) was classified non-embedding offline and
the settings endpoint returned 409.
Add _snapshot_modules_all_loadable, which parses modules.json and accepts a snapshot when every
declared module's path directory carries the files that module class's own load() reads (a
Transformer/root module still needs the full HF load root; a WordEmbeddings module needs its
config plus a complete weight set; other modules need their *_config.json), and at least one
embedding-producing module is present. It is OR-ed after the Transformer check, so it only ever
accepts more and cannot regress the existing path or reject a pruned cache.
* studio: scan PEFT adapter pickle weights in the offline security gate
from_pretrained auto-detects an adapter_config.json in the load root and deserializes the
adapter weights on top of the base model, so adapter_model.bin is a separate pickle RCE vector
that a safetensors base weight does not cover. The offline scan matched only base-model pickle
names, so an offline local-only load with safetensors base weights plus a cached
adapter_model.bin was allowed despite the live adapter pickle. Scan adapter pickles too, scoped
to a load root where adapter_config.json is present and no adapter_model.safetensors exists.
* studio: require weights for Dense/CNN/LSTM SentenceTransformer modules offline
_module_dir_is_loadable accepted a Dense, CNN, or LSTM module dir with only its config, but
those modules' load() hard-load model.safetensors else pytorch_model.bin (verified against
sentence-transformers source: no fallback, raises if neither exists) -- exactly like
WordEmbeddings. A cache with such a module's config but no weights would validate and then
fail the local_files_only load. Require a complete weight set for every weighted module, not
just WordEmbeddings.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: scan root-index subdir pickle shards offline; handle __all__.append/.extend
- The offline pickle scan followed only load-root directories, so a shard mapped by a root
pytorch_model.bin.index.json into a non-root subdirectory was skipped even though
from_pretrained follows the index weight_map and deserializes it (a layout an attacker can
craft to evade the scanner). Read the local index and scan its referenced pickle shards,
covered by a loadable base safetensors at the index root -- mirroring the online scan.
- The import-hoist analyzer ignored __all__.append("X") / __all__.extend([...]) runtime
re-export mutators, so an import added solely for one tripped HOISTED-IMPORT-UNUSED. Read
their string args like +=, and treat any other __all__ method call as opaque.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: classify StaticEmbedding offline, require WordEmbeddings tokenizer, bound model_info
- A StaticEmbedding module (e.g. sentence-transformers/static-retrieval-mrl-en-v1's
0_StaticEmbedding/) holds tokenizer.json + weights and NO config, so the config-gated
non-Transformer path 409'd it offline. Recognize it by what StaticEmbedding.load() reads: a
tokenizer.json plus a complete Torch weight set.
- WordEmbeddings.load() rebuilds its tokenizer via the configured tokenizer_class.load() from the
module dir, so a WordEmbeddings module now also requires a tokenizer artifact
(whitespacetokenizer_config.json / phrasetokenizer_config.json, or a shared HF tokenizer asset),
not just its config + weights.
- With neither offline env var set, an unbounded model_info() could hang on connect/DNS retries
for networkless users (the #6817 symptom). Bound it with a 15s timeout so a dead network fails
fast and the existing transient-failure cache fallback resolves a cached model, while a
reachable Hub still wins. (Documented caveat: a stalled DNS getaddrinfo may exceed this.)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Resolve indexed safetensors shards relative to their index
_safetensors_index_complete compared shard basenames against the flat
set of files in the index directory, so an index whose weight_map names
shards in a subdirectory was treated as incomplete whenever a legacy
pytorch_model.bin sat beside it. That falsely blocked a snapshot whose
pickle weights are fully covered by a complete, loadable safetensors
shard set. Resolve each shard path relative to the index directory
instead, and add a regression test for the subdir-mapped shard case.
* Restrict offline weight-completeness check to declared load roots
_snapshot_has_complete_weights scanned every directory in a snapshot and
accepted it when ANY directory was a complete Transformer load root. When
modules.json is present a SentenceTransformer load only opens the declared
module paths, so a snapshot whose declared modules are incomplete but which
happens to contain an unrelated complete directory was accepted offline and
then failed at the first local_files_only load. Restrict the candidate
directories to the roots a load actually opens: the snapshot root plus each
modules.json module path. For a well-formed snapshot the verdict is
unchanged; only a complete directory at an undeclared path no longer vouches
for an otherwise-incomplete snapshot.
* Scan SentenceTransformer Router child module weights offline
A Router (legacy Asym) snapshot declares its child sub-modules only in
router_config.json, not the top-level modules.json, and Router.load()
deserializes each child's weights from its own subdir. A config.json-less
child such as query_0_WordEmbeddings (wordembedding_config.json plus a
pickle pytorch_model.bin loaded via torch.load) was therefore neither a
modules.json-declared load root nor a config.json-bearing dir, so the
offline gate skipped its pickle even though the loader deserializes it.
Parse router_config.json at each load root and treat every declared child
subdir as a load root (bounded BFS, so nested routers are covered), so
those child pickles are scanned. Add Router regression tests: a pickle
child blocks, a safetensors child is allowed, and a Router in a declared
subfolder is followed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Do not treat an unreferenced config subdir as an offline load root
The offline pickle gate skipped a directory only when it was neither a
declared load root nor held a config.json. Because _st_load_roots already
resolves every real load root (snapshot root, modules.json / load_subdirs
dirs, Router children), the config.json fallback only ever promoted an
UNREFERENCED subdir -- a nested checkpoint-500/ or archive/ that ships its
own config.json + pytorch_model.bin -- to a load root. from_pretrained
never descends into such a subdir and the online scan ignores the same
unindexed pickle, so offline mode wrongly blocked a model the loader reads
from a clean safetensors root. Scope the pickle to directory in roots
only, and add a regression test (a stray checkpoint-500/ no longer blocks;
a modules.json-declared module dir still does).
* Classify a root Router (Asym) model as loadable offline
_module_dir_is_loadable applied Transformer root requirements (config +
tokenizer + weights) to every root module, so a Router saved at the
snapshot root -- which carries only modules.json + router_config.json and
loads its weights from child subdirs -- was classified not loadable
offline, and is_embedding_model missed a cached Router embedder. Dispatch
on the module class before the root Transformer fallback: a Router/Asym
dir is loadable when router_config.json parses and every declared child
subdir is loadable (validated recursively through _module_dir_is_loadable,
so nested routers and every child type are covered) with at least one
embedding-producing child. This also tightens a non-root Router, which
previously validated on the mere presence of router_config.json without
checking its children. Add Router regression tests (root and declared
subfolder, complete and incomplete-child).
* Require every declared module before accepting an offline cache
_snapshot_is_loadable_st_model returned has_complete_weights OR
modules_all_loadable, so a complete 0_Transformer short-circuited the or
and vouched for the whole snapshot even when a declared sibling module was
missing its serialized weights; SentenceTransformer builds every module in
modules.json, so that snapshot passed offline validation and then failed
the local-only load. When modules.json declares a non-empty list it is now
authoritative (modules_all_loadable validates every declared module);
has_complete_weights stays the fallback only for an empty/non-list
modules.json (the plain from_pretrained root). Also add the weight-bearing
modules whose load() hard-loads via load_torch_weights and previously fell
to the config-only path -- LayerNorm, WeightedLayerPooling, SparseAutoEncoder
-- to _ST_WEIGHTED_MODULE_NAMES, with source citations and the deliberate
exclusions (Pooling/Normalize/BoW/WordWeights read no weights on load).
Add parametrized regression tests over LayerNorm/WeightedLayerPooling/Dense
(a weightless sibling rejects, a complete sibling accepts).
* Reject self-referential Router children instead of recursing forever
_router_dir_is_loadable validates each router_config.json child through
_module_dir_is_loadable, which re-enters _router_dir_is_loadable for a
Router child. A malformed types entry naming the router's own directory
(a key of ".", which normalizes to the same dir) made that recursion
never descend, so it looped until RecursionError -- breaking the
documented never-raises contract and turning a crafted/corrupted cached
model into a 500 from is_embedding_model instead of a graceful
unverifiable result. A real child reference is a subdir and always
resolves deeper, so reject any child whose resolved path is the router
dir itself. Add a regression test (a router_config naming "." as a
Router child returns False without raising).
* Treat a destructuring __all__ assignment as opaque
_collect_dunder_all detected __all__ only as a direct ast.Name assignment
target, so a binding through a destructuring target (__all__, meta = [...],
v -> an ast.Tuple) was skipped entirely, leaving an empty, non-opaque
export set. A newly hoisted import re-exported only through that assignment
was then falsely flagged HOISTED-IMPORT-UNUSED. Its value cannot be mapped
statically, so mark the export set opaque when __all__ is reached only
through a destructuring / item / attr target, matching how the collector
already handles other unreadable __all__ forms. Add a self-test case.
* Canonicalize declared module paths before scoping the offline pickle gate
A repo could declare a traversing module path such as 0/../evil in
modules.json (or a router_config child), which SentenceTransformer resolves
to evil/ and deserializes evil/pytorch_model.bin. _st_load_roots recorded
the raw snap/"0/../evil", which never equals the snap/evil that rglob
yields, so the offline pickle gate skipped that directory and a malicious
repo slipped a pickle past the newly added gate. Add _canonical_load_dir
to collapse ./ and ../ components lexically and reject an upward escape,
and route the modules.json paths, load_subdirs and router children through
it so the gate scopes the same normalized directory the loader opens. Add
regression tests for a traversing modules.json path and router child.
* Close offline embedding-classification completeness gaps
Five real offline misclassifications, each a false negative (the #6817 hang
recurs) or false positive (accepted then 409s at the local_files_only load).
Dispatch _module_dir_is_loadable on the module class before the root
Transformer fallback. A module with save_in_root=True (every InputModule:
WordEmbeddings, StaticEmbedding, SparseStaticEmbedding, Transformer, Router)
is saved at the snapshot root, so a root WordEmbeddings was wrongly held to
Transformer requirements (an HF tokenizer it never writes) and classified not
loadable.
CLIPModel is Transformer-shaped: CLIPModel.load() reads AutoModel weights plus
AutoProcessor, so a config-only CLIP dir must not validate.
SparseStaticEmbedding needs a tokenizer plus either idf.json or a complete
torch weight set (conditionally weight-bearing); a config alone is not enough.
A present but empty or malformed modules.json is not loadable and does not fall
back to a root Transformer: with modules.json present the loader never takes
the plain-Transformer path (base/model.py _load_config_modules). The tag-only
no-modules.json embedder is classified separately via
_snapshot_has_complete_weights.
Validate a sharded weight index against its weight_map (every mapped shard
present, resolved relative to the index dir) instead of trusting the index
file's mere existence, mirroring the security-side check.
Add regression tests for all five.
* Close case-folding and online-traversal holes in the offline pickle gate
Two gate bypasses where the security scan credited or scoped a path
differently from what the loader actually resolves:
The safetensors credit was case-folded. _cached_pickle_weight_files lowercases
every filename, and the loadable-safetensors and adapter checks tested those
folded keys against the exact-lowercase names. On a case-sensitive filesystem
(Linux, the Studio default) a crafted repo shipping Model.SafeTensors plus a
malicious pytorch_model.bin makes transformers and sentence-transformers miss
the exact-name model.safetensors and deserialize the pickle, while the gate
credited an inert safetensors and did not block. Credit safetensors
case-sensitively against real filenames, and drop pytorch_model.safetensors
from the credit set (transformers loads only model.safetensors, never that
name). Pickle matching stays case-insensitive (over-blocking a mis-cased
pickle the loader would not load is the safe direction).
The online scan did not canonicalize traversing paths while the offline gate
did. A repo-controlled modules.json path (threaded into the online scan via
the RAG guard) or a weight_map shard entry like 0/../evil / ../evil was
compared verbatim, so a flagged evil/pytorch_model.bin never matched and
evaded the online scan though the loader resolves and deserializes it.
Canonicalize the repo-controlled load-subdir prefixes and weight_map shards
the same way the offline gate does, so offline and online agree.
Add regression tests for both bypasses.
* Treat a conditional __all__ mutation as opaque in the import-hoist linter
_collect_dunder_all replayed only top-level module statements, so an __all__
assignment or mutation inside a module-level if / try / for / while / with /
match (or a deeper scope) was ignored, leaving the export set understated. A
newly hoisted import re-exported only through such a conditional __all__ was
then falsely flagged HOISTED-IMPORT-UNUSED, blocking a valid change. A
conditional value cannot be replayed statically, so mark the export set opaque
when __all__ is bound or mutated anywhere other than a top-level statement.
Add a self-test case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope Router child sub-modules as load roots in the online embedding scan
The RAG embedding security guard unions the SentenceTransformer module dirs
from modules.json into the load roots it scopes for the Hub scan, so a flagged
pickle directly under a Transformer module blocks. A Router (legacy Asym)
module declares its child sub-modules only in router_config.json, not in
modules.json, and Router.load() deserializes each child from its own subdir.
The online scan therefore dropped a flagged child pickle (for example
query_0_WordEmbeddings/pytorch_model.bin) as an unreferenced nested shard while
the loader still deserialized it, the counterpart to the offline gate which
already expands router children via _router_child_dirs.
_st_module_subdirs now reads router_config.json for any Router-typed module and
adds each declared child (joined onto the module path, canonicalized so a
traversing entry is dropped) to the load roots. The config is read only for a
Router-typed module, so a plain embedder pays no extra fetch, and every failure
path still returns () so the guard never bricks the embedder.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Allow a recorded-clean pickle embedder to load offline
The offline embedding security gate is fail-closed: with no network to reach
Hugging Face's scan, a cached pickle weight cannot be verified, so it is blocked
and a model the user already downloaded and used online will not load offline.
This adds a persistent cache of clean Hub verdicts so that exact content can load
offline, without weakening the gate for an unknown or never-scanned pickle.
When an embedding repo is loaded online and HF's scan returns a completed clean
verdict, the load roots are hashed and recorded under the scanned commit as an
exact map of snapshot-relative pickle name to sha256, in a per-user JSON store at
studio_root()/security/embedding_scan_verdicts.json (atomic write, 0600, thread
and cross-process locked, 30-day TTL). Offline, a cached pickle model loads only
when the active cached commit and every load-root pickle's sha256 match the
recorded verdict; a missing record, moved commit, changed or added pickle,
expired record, or any error keeps blocking. Online loads always re-query the
Hub and an authoritative unsafe verdict deletes any stale record, so a
now-flagged commit cannot keep loading on an old clean record.
The store binds repo id, full commit, and a per-file sha256 map so a locally
swapped pickle at the same commit, a branch advance, or an added load-relevant
pickle is detected. A same-user attacker who can rewrite the model cache or the
store is outside the enforceable boundary and this is documented; the sha256 is
computed just before load, so a narrow verify-to-load window remains, and a Hub
scanner false negative is recorded faithfully (safetensors stays the stronger
defense).
Recording is triggered post-load in the RAG embedder because the settings route
only validates and the pre-load guard runs before the constructor downloads;
recording is skipped when the loaded commit differs from the scanned commit. The
blocked-pickle enumerator now returns snapshot-relative Paths so two module dirs
that ship the same pickle basename are hashed and reported distinctly.
* Harden the embedding verdict cache against review findings
Tighten the offline verdict cache and its enumeration so every uncertain or
malformed input fails closed and the recorded hashes always match the files the
loader reads:
- Hash every case-colliding pickle in a load root, not one representative. On a
case-sensitive filesystem pytorch_model.bin and PYTORCH_MODEL.BIN are distinct
files; keying by lowered name dropped one and could hash a decoy instead of the
loader's target. The enumerator now returns every variant Path.
- Only persist a clean verdict for a COMPLETED, entirely-benign scan. Require
scansDone to be the boolean True (not a truthy string), filesWithIssues to be a
well-formed list, and every flagged file to be a definitively-safe level; a
pending, error, unknown, or malformed entry no longer records as clean. The
online block decision is unchanged.
- Fail closed when the offline cache cannot be inspected: an rglob error now
propagates and blocks instead of reading as pickle-free, and a snapshot that
errors on resolution (vs a clean not-cached) blocks. The offline guard also
raises instead of returning when its own inspection throws, so the constructor
never deserializes an unverified cached pickle.
- Expand online Router children recursively (bounded BFS with a seen set),
mirroring the offline load-root expansion, so a flagged grandchild pickle is
scoped online and cannot be recorded clean.
- Reject absolute and drive/UNC declared paths in the load-root canonicalizers;
the loader would resolve them outside the snapshot, so collapsing them to an
in-snapshot relative dir scoped the wrong place.
- Pin verdict recording to the scanned commit's snapshot and take the offline
verify commit from the snapshot directory name, removing a second refs/main read
and the skew it allowed.
- Drop the now-unused pickle-name wrapper.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten offline embedding classification and the pickle gate
Close a set of offline edge cases where validation accepted a cache the
local_files_only load then rejects, and one gate bypass:
- Credit a sharded model.safetensors.index.json for a pickle sibling only at a
from_pretrained root. A non-Transformer SentenceTransformer module (Dense,
WordEmbeddings, StaticEmbedding) loads via Module.load_torch_weights, which
reads model.safetensors then pytorch_model.bin and never the index, so a sharded
safetensors index in such a module dir must not vouch for its pytorch_model.bin.
- Stop counting pytorch_model.safetensors as loadable in the offline classifier:
the loader probes model.safetensors (then its index) or pytorch_model.bin, never
pytorch_model.safetensors, matching the gate that already treats it as a decoy.
- Treat a present but unreadable weight index as incomplete: transformers opens
and parses any present index, so a malformed one or one without a weight_map
fails the load rather than falling back to filename-numbered shards.
- Require the CLIP image-processor config (preprocessor_config.json) for a CLIP
module: CLIPModel.load builds a CLIPProcessor that needs it, so a tokenizer
alone is not enough.
- Require a SparseStaticEmbedding config to actually select idf.json (a path
ending .json) or ship loadable weights; a bare idf.json the config does not name
falls through to load_torch_weights and raises.
- Do not use the tag-only recorded-verdict fallback when modules.json is present:
with the file present the loader takes the modules.json path, so a present but
empty or malformed manifest must not be validated as a plain root Transformer.
- Import-hoist linter: only a module-level conditional mutation or a function that
declares global __all__ makes the export set opaque; a __all__ bound as a local
in a nested function or class no longer masks a genuinely unused hoisted import.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope Router-child pickles to their deepest load root and gate the ST offline kwarg
The online scan stripped the first matching load-subdir prefix from a flagged file, so a
nested Router child pickle (0_Router/query_0_WordEmbeddings/pytorch_model.bin) matched the
parent 0_Router root, looked like an unreferenced nested shard, and slipped the gate even
though Router.load() deserializes that child directly. Match the deepest (longest) load
subdir instead, so the child becomes root-level under its own load root and blocks.
pyproject sets no lower bound on sentence-transformers and the local_files_only constructor
arg is absent on older releases, so always forwarding it broke every embedder warm on those
installs. Pass it only for an offline load; an online warm never forwards it and works as
before, while the offline capability still requires a version that supports it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reject snapshot-escaping shard paths and credit Transformer submodule safetensors
The offline pickle enumerator joined a weight-index weight_map value straight to the load
root and followed it, so a repo-controlled index mapping "../.." into a sibling snapshot
made an offline from_pretrained deserialize an out-of-snapshot pickle, and an online load
would then hash and record that external file as the scanned commit's clean content. Reject
any shard path that escapes the snapshot root and fail closed, mirroring the canonical-root
check the online shard scan already applies.
A complete model.safetensors.index.json was credited over a sibling pickle only at the
snapshot root, but a Transformer module subdirectory (0_Transformer/) is loaded via
AutoModel.from_pretrained, which honors that shard set and never reads the pickle. Credit the
sharded index for Transformer-typed modules declared in modules.json so a cached model that
ships both a sharded safetensors checkpoint and an unused PyTorch checkpoint is no longer
falsely blocked offline. Non-Transformer modules (Dense, WordEmbeddings, StaticEmbedding) read
a flat weight with no index and keep their pickle blocked.
Limit the import-hoist verifier's global __all__ scan to the declaring function's own scope so
a nested inner-scope local __all__ no longer marks the module export set opaque and mask an
unused hoisted import.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope Router children against the snapshot and mirror the ST alias rewrite
Router.load resolves each child at Path(subfolder, model_id) relative to the Router dir, so a
nested 1_Router with a "../evil" child points at evil/ inside the snapshot and the loader
deserializes evil/pytorch_model.bin. The offline enumerator canonicalized the child against
the Router dir alone and dropped anything with "..", so that pickle was never scanned and the
gate reported the cache pickle-free. Canonicalize router children against the snapshot,
retaining in-snapshot siblings as load roots and failing closed on a child that escapes the
snapshot itself, matching the online scan which already joins the prefix before normalizing.
The security gate resolved a slashless model id by probing the bare cache dir first, but the
SentenceTransformer constructor rewrites a non-basic slashless name to sentence-transformers/
<name> and loads THAT snapshot (only the basic ORIGINAL_TRANSFORMER_MODELS load bare). With
both models--<name> and models--sentence-transformers--<name> cached, the gate inspected the
bare dir while the loader read the namespaced one, so a pickle there bypassed the local-only
gate. Mirror the constructor: try the namespaced candidate first for non-basic slashless names.
Add the same not (snapshot / modules.json).is_file() guard to the transient-Hub-failure
tag-only fallback that the offline branch already carries, so a cache whose present manifest is
empty or malformed is no longer reported as a loadable embedder.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten root shard credit, module-path escapes, and weight-set probe order
Credit the sharded safetensors index at the snapshot ROOT only when the root is actually loaded
through an AutoModel/from_pretrained path. A modules.json root module of a non-Transformer type
(StaticEmbedding / WordEmbeddings / Dense) loads via load_torch_weights, which reads
pytorch_model.bin and ignores the index, so crediting a root shard index there suppressed a live
root pickle and let the offline gate report the cache pickle-free.
Recognize the Transformer subclasses CLIPModel and MLMTransformer as index-honoring load roots
(they load via from_pretrained), so a sharded-safetensors CLIP/MLM submodule with a legacy
pytorch_model.bin sibling is no longer falsely blocked offline. Mirrors the classifier dispatch.
Fail closed on an absolute or snapshot-escaping modules.json module path (or load_subdirs entry)
instead of silently dropping it: SentenceTransformer resolves such a path outside the snapshot and
would deserialize an external pytorch_model.bin the gate cannot scan.
On the classifier side, walk the weight set in the exact from_pretrained probe order
(model.safetensors, its index, pytorch_model.bin, its index) so a pickle behind a malformed
safetensors index is no longer accepted as complete, and restrict shard names to the loader-probed
stem/ext pairs so a decoy model-*.bin / pytorch_model-*.safetensors set is not treated as loadable.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Restore scripts/verify_import_hoist.py to main
The offline embedding cache fix does not depend on the __all__ scope
handling that had accumulated in this linter, so revert the file to its
main version and keep the PR focused on the feature. The feature modules
still pass the existing import hoist check unchanged.
* Reuse a shared HF cache skeleton in the offline classification tests
Extract _mk_repo and _activate helpers for the repeated snapshot cache
setup that every per-type builder duplicated, and fold the two
StaticEmbedding missing-asset cases into one parametrized test. Same 125
collected items, all still passing.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reclassify embedding models from the cache on every offline call
is_embedding_model consulted its process memo before the offline branch, so an
online lookup that memoized True from tags (without caching any weights) was returned
unchanged once the session went offline -- the studio flips HF_HUB_OFFLINE in-process
on a dead DNS, and the ungated check-embedding route can populate the memo. Settings
would then accept a repo the offline loader cannot open. Run the offline
cache-marker reclassification ahead of the memo and never record it, so an offline
verdict always reflects the local cache and a later cache materialization is not
masked by a stale negative. Add regression tests.
* Tighten comments on the offline embedding path
Condense the offline-embedding helper docstrings and inline comments added in
this PR to fewer, clearer lines, keeping the non-obvious security and offline
rationale. Comments and docstrings only; no code change.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
* studio: show system-wide VRAM in the multi-GPU System tab view on ROCm
The System tab's per-GPU list comes from get_visible_gpu_utilization. When
amd-smi is unavailable (always on Windows, minimal Linux installs) it fell back
to torch, whose readings are process-local: on Windows WDDM hands each process
its own budget, so a model held by the separate llama-server process read as
~0 VRAM used even with the GPU full (#7072). The primary-GPU endpoint already
compensates with system-wide sources -- Windows Performance Counters (Task
Manager's source) and Linux DRM sysfs -- but the multi-device endpoint never
got those fallbacks.
Add per-GPU variants of both sources and overlay them onto the torch fallback:
_rocm_windows_perf_counter_vram_per_adapter_gb() attributes Dedicated Usage per
physical adapter (phys_<N> in the counter instance name), and
_rocm_linux_sysfs_vram_per_card_gb() reads mem_info_vram_{used,total} per DRM
card. _overlay_system_wide_vram() applies them to the device list, ROCm-only,
best-effort: unmatched adapters and ambiguous card counts keep the torch
figures, and NVIDIA paths are untouched.
Fixes#7072
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: match VRAM overlay sources by device, honor unified memory, unblock the loop
Five review fixes on the multi-GPU system-wide VRAM overlay:
1. Linux: match DRM cards to devices by PHYSICAL index instead of a positional
zip, so a reordering visibility mask (HIP_VISIBLE_DEVICES=1,0) no longer
swaps each card's figures onto the other GPU (which would mislead
auto_select_gpu_ids and the coexistence checks). An index with no matching
card keeps its torch figures.
2. Linux: skip the overlay for a device whose sysfs total is below torch's --
on unified-memory APUs (Strix Halo) mem_info_vram_total is only the small
dedicated slice while torch sees the GTT-backed pool, and
_apply_unified_memory_correction already defines larger-total-wins.
3. Windows: group counter instances by adapter LUID, not the phys_<N> suffix --
separate adapters each read phys_0, which collapsed every GPU into key 0.
LUIDs are mapped to 0-based positions by ascending value as the closest
stand-in for device order.
4. Windows: pair the system-wide usage with the physical capacity from
get_device_properties (as the primary-GPU fallback does) -- under WDDM
mem_get_info's "total" is the process budget, which misreported capacity
and pushed utilization to 100%.
5. Run get_visible_gpu_utilization off the event loop in the /hardware/visible
route (asyncio.to_thread, the repo's convention): the ROCm fallbacks can
shell out to PowerShell with a 5s timeout, which would stall every other
request while the System view polls.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: skip the system-wide VRAM overlay for relative GPU indices
The overlay matches its per-GPU sources (Windows perf counters, Linux sysfs) by
physical device index, but under a UUID/MIG visibility mask the torch fallback
enumerates ordinals and reports index_kind == "relative", where `index` is a
visible ordinal, not a physical id. Applying the overlay there let card/adapter
0's system-wide VRAM overwrite the torch reading of a process that actually
exposes physical GPU 1, misleading auto_select_gpu_ids and the coexistence
checks. Gate the overlay on index_kind == "physical"; relative-index paths keep
the torch fallback.
* studio: drop the unreliable Windows VRAM overlay, keep the Linux one
The multi-GPU system-wide VRAM overlay is now Linux-only. The Windows
per-adapter Performance Counter path could not be made correct: the wildcard
Get-Counter query also returns non-ROCm/iGPU adapters and LUID order is not the
ROCm device order, so an adapter's usage could be overlaid onto the wrong GPU;
and it read only Dedicated Usage, missing WDDM shared memory on unified-memory
GPUs (Strix Halo), overstating free VRAM. Rather than misattribute VRAM and
skew placement decisions, Windows keeps the process-local torch fallback (no
regression vs before this PR); Linux DRM sysfs -- matched by physical index --
still fixes#7072 for the reporter's native-Linux ROCm case.
Removes _rocm_windows_perf_counter_vram_per_adapter_gb and _torch_props_total_gb.
* studio: key sysfs VRAM by DRM card number so filtering can't renumber cards
_rocm_linux_sysfs_vram_per_card_gb dropped cards with a zero total or unreadable
files and then the overlay enumerated the compacted list, so if card0 was
dropped, card1's usage was assigned to physical GPU index 0 (equal-capacity GPUs
slip past the unified-memory total guard). Return {card_number: (used, total)}
and match a device to its card number directly: a hole stays a hole -- device 0
keeps its torch figures when card0 is absent, and card1 maps to device 1.
* studio: key system-wide VRAM by ROCm ordinal, not raw DRM card number
When a non-amdgpu adapter (Intel iGPU, a display-only card) owns an earlier
DRM slot, DRM card numbers stop equalling ROCm device ordinals -- Intel card0
plus AMD card1/card2 gives ROCm devices 0/1, so keying the sysfs overlay by
card number handed ROCm device 1 card1's data (AMD device 0) and left device 0
on stale torch figures, corrupting free-VRAM placement on equal-capacity GPUs.
Only amdgpu cards expose mem_info_vram_*, so the glob already excludes foreign
adapters; order the surviving cards by their PCI address (ROCm/HIP's default
device order, read from each card's device symlink) and key by that position --
the ROCm physical ordinal, which is what the overlay matches against dev index.
An unreadable / zero-total amdgpu card still consumes its ordinal so a later
card is never renumbered onto its slot.
* studio: skip the VRAM overlay under layered HIP-over-ROCR masks
ROCR_VISIBLE_DEVICES filters physical GPUs at the HSA/ROCr layer, and a
HIP_VISIBLE_DEVICES set on top selects WITHIN that already-filtered set
(apply_gpu_ids sets HIP while leaving an inherited ROCR mask in place). When
both are active _get_parent_visible_gpu_spec() prefers the HIP value, so the
reported device index is a ROCR-relative ordinal, not a physical GPU id --
overlaying DRM-sysfs figures by that index would pull another GPU's usage
(e.g. ROCR=2,3 + HIP=1 is physical GPU 3, but the overlay would read card 1),
and equal-capacity cards bypass the total-size safeguard. Detect layered masks
and keep torch's process-local figures there rather than risk misattribution;
a single mask still leaves the index physical and is overlaid as before.
* studio: only overlay whole-card VRAM onto 1:1 ROCm devices
The overlay guard only skipped the case where sysfs total < torch total
(unified-memory APUs), so a partitioned ROCm device (MI300 in CPX mode) --
where HIP exposes several logical devices per physical card but sysfs reports
the whole card's aggregate -- passed the guard: the card total exceeds a
partition's torch total, and the overlay overwrote the partition with
whole-card usage and capacity, letting downstream selection think a partition
had the entire card free. Require the sysfs card total to match the torch
device total (within ~10%) so a mismatch in either direction -- unified memory
(sysfs smaller) or partitioning (sysfs larger) -- keeps torch's figures.
* studio: treat CUDA-over-ROCR as layered, enumerate AMD cards by driver
Two remaining mismatches between the reported device index and the DRM card the
overlay reads:
- On ROCm the HIP layer honors CUDA_VISIBLE_DEVICES as well as
HIP_VISIBLE_DEVICES, so a CUDA mask composed over ROCR layers identically:
ROCR=2,3 with CUDA=1 is physical GPU 3, yet the spec reports the ROCR value
[2,3] and the device was labeled index 2, overlaying card 2's usage onto GPU 3.
The layered check now treats ROCR combined with either HIP or CUDA as layered.
- The ROCm device set is now enumerated by bound driver (device/driver resolves
to amdgpu) instead of by the presence of mem_info_vram_*. An AMD device with
incomplete sysfs support (some APUs expose no VRAM files at all) was omitted
by the glob entirely and shifted every later card down one ordinal, letting a
similar-capacity GPU pass the total guard with another device's usage. Such a
card now consumes its ordinal and simply yields no entry.
* studio: honor GPU_DEVICE_ORDINAL and require an unambiguous card mapping
Two remaining ways the reported device index could be matched to the wrong DRM
card:
- GPU_DEVICE_ORDINAL is a supported ROCm visibility variable that
_get_parent_visible_gpu_spec() never consults, so GPU_DEVICE_ORDINAL=1
surfaces physical GPU 1 as torch ordinal 0 and it was mislabeled index 0,
overlaying card 0's usage onto GPU 1. The mask check now covers it, and is
renamed _rocm_device_index_unreliable() to say what it actually decides.
- driver == amdgpu is only a SUPERSET of the ROCm-visible set: an amdgpu-bound
adapter HIP cannot enumerate (an unsupported older AMD GPU beside a supported
one) still took an ordinal and shifted every real compute device. There is no
torch-side PCI identity to match against, so the overlay now requires the
amdgpu card count to equal the device count -- exactly the condition under
which position-in-PCI-order is a sound 1:1 mapping. Any disagreement keeps
torch's process-local figures: less informative, never misattributed.
* studio: keep the VRAM overlay working for masked GPU subsets
The card-count guard compared the amdgpu card list against the VISIBLE device
list, so any visibility mask disabled the overlay outright: HIP_VISIBLE_DEVICES=1,3
on a four-GPU host gives two devices against four cards. Those masked GPUs then
kept reporting process-local torch usage, hiding VRAM held by llama-server and
letting the training/chat placement checks overestimate free memory -- the exact
problem the overlay exists to fix.
The count check now applies only when no visibility mask is active, which is the
case where the reported devices really are the whole host and a mismatch means an
amdgpu adapter ROCm cannot enumerate is shifting the ordinals. Under a mask the
subset is expected, so each device's physical index is validated individually
instead: the per-card lookup bounds-checks it and the total-size guard rejects a
card whose capacity does not match the device's.
* studio: match GPUs to DRM cards by PCI identity, not by position
Every mapping bug on this PR came from the same root cause: there was no
authoritative link between a reported device index and a DRM card, so the
overlay kept inferring one positionally and each heuristic broke on a new host
shape -- foreign adapters on earlier DRM slots, cards with no VRAM sysfs, and
most recently amdgpu-bound adapters HIP cannot enumerate, which the count guard
could only catch on an unmasked host and therefore missed under any mask.
Use the link ROCm itself enumerates from. KFD topology
(/sys/class/kfd/kfd/topology/nodes/<N>/properties) lists exactly the GPUs HIP
exposes -- GPU nodes in node-id order are HIP's device order -- and each carries
its PCI location, so index N there IS physical device N with a stable identity.
DRM sysfs now supplies system-wide VRAM keyed by that same PCI address, and the
overlay is a join on it.
Every previous skew becomes a failed join rather than a misattribution: an
unenumerable adapter has no KFD node so it never takes an ordinal, a foreign
adapter contributes no entry, and a masked subset resolves each physical index
directly. That removes the count heuristic and its mask exception entirely. With
no KFD topology there is no identity to join on, so the overlay is skipped rather
than guessing positionally.
* studio: require verified host visibility and AMD-only KFD nodes
Three ways the identity map could still be built on a false premise:
- The NVIDIA open kernel module registers KFD topology nodes with a positive
SIMD count, so an earlier NVIDIA node shifted every AMD ordinal and ROCm
device 1 resolved to AMD GPU 0. GPU nodes now require vendor_id 4098 (0x1002),
the same filter install.sh already applies for this exact reason.
- A GPU node with an unreadable properties file or no location_id was skipped,
which silently shifted every later ordinal. Both now fail the whole map
closed, so the overlay is disabled rather than misattributing.
- A container exposing only some render devices through device cgroups sets no
visibility variable, yet torch compacts what it can see to ordinals from zero
while the host-mounted KFD and DRM trees still list every GPU. Nothing in the
reported payload distinguishes that from a full host, and torch exposes no PCI
id to check against, so the overlay now runs only when host visibility is
positively verified: no visibility mask AND device count equal to the host GPU
count. That also subsumes the previous layered-mask and GPU_DEVICE_ORDINAL
checks, so _rocm_device_index_unreliable() is gone.
This trades coverage for correctness: masked subsets and filtered containers now
keep torch's process-local figures instead of a mapping that cannot be verified.
* Fix the multi-GPU VRAM overlay docstring for PR #7216
The docstring claimed a reordering mask keeps each card on the right GPU,
but the overlay skips any active visibility mask and keeps torch's figures.
State the actual gating instead.
* Tighten comments in the multi-GPU VRAM overlay and its tests
Collapse the verbose docstrings and inline explanations added for the Linux
ROCm system-wide VRAM overlay to succinct one-liners, keeping the non-obvious
rationale (fail-closed KFD mapping, PCI-identity join, mask gating, the 10%
whole-card guard). Comments only, no behavior change.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* Improve unsloth start runtime lifecycle
* Remove speculative Gemma prompt override
* Polish model download progress output
* Refine unsloth start status output
* Clarify unsloth readiness banner
* Clarify model reuse and switching output
* Queue model switches behind active inference
* Tighten unsloth start model switching
* Reduce model switch bookkeeping
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix Studio re-exec compatibility
* Recheck sidecar reservation after inference drain
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pass start marker through child environment
* Fix key redaction, switch-waiter ordering, and stop/messaging gaps for PR #7313
- Redact minted sk-unsloth keys from the startup-failure log tail: the early
key marker lands in the server log before the model load finishes, so a
load-phase crash printed a live key to the terminal
- Deregister a finished switch waiter before releasing the swap gate so a
swap on another event loop cannot count it as still queued and unload the
model the finished request is about to generate against
- Warn on same-repo quant switches: an explicit variant replaces the resident
weights for every attached session, but the repo ids match so no switch
warning was printed
- Note the agent exit code when it is nonzero so the server keep-alive
message does not read as a successful session
- Use taskkill /T in unsloth studio stop so llama-server children stop too
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments in start, studio, and inference changes
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Fix resume training crash recovery and MLX checkpoints
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: preserve interrupted stop-and-save output_dir, verify MLX checkpoint
- finish_run: add clear_output_dir flag; preserve output_dir for stopped/error
unless cancel explicitly clears it (fixes pump finalization wiping persisted path).
- training pump: pass interrupted stop-and-save context into finalize_run_in_db.
- MLX stop-and-save: verify resumable checkpoint exists before sending complete;
return bool from _write_mlx_stop_checkpoint and add regression tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address Codex review: MLX current-step checkpoint and cancel error finalize
- Only skip MLX stop checkpoint write when checkpoint-{current_step} exists;
stale periodic checkpoints no longer mask missing stop saves.
- Pass clear_output_dir through error-event finalization so Stop-without-save
cannot leave a persisted output_dir that still offers Resume.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review
* Address more reviews
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* more reviews
* clear in-memory output_dir on interrupted cancel
* allow resuming errored runs at the final step
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Clear persisted output_dir in cancel watchdog path
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Write MLX stop checkpoint in stop path, keep output_dir on crash finalize
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): harden resumable run finalization
* fix(studio): defer safetensors checkpoint import
* fix(studio): reject stale training cancellation
* fix(studio): replay null resume targets
* fix(studio): serialize terminal cancellation
* Harden resume checkpoint validation and fix stop-save cleanup
- Reject unrecognized shard formats and keep indexed shard paths inside the checkpoint dir
- Require a non-empty tensor record when validating .pt/.bin optimizer and model state
- Always finalize TensorBoard and W&B on stop-save-failure exits
- Refuse writing an MLX stop checkpoint through a symlinked directory
- Clarify the resume rejection message to cover errored runs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten resume/checkpoint comments
* Recover resumability when a valid stop checkpoint landed
- Re-validate the current-step checkpoint in the dead-worker and error finalization paths so a stop-and-save that actually wrote a valid checkpoint is not wrongly marked error/resume_blocked
- Accept a valid tensor-free optimizer state (e.g. SGD without momentum); the model-state check still requires real tensors
- Include errored runs in the frontend resume rejection message
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lyxot <longyixing331@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* refactor(studio): move chat model picker into features/model-picker
Relocate model-selector + its support files from components/assistant-ui
into a self-contained features/model-picker feature (own barrel), mirroring
the modular Hub layout. Pure move + import repoint; no behaviour change.
* feat(model-picker): add per-model config persistence layer
Superset PerModelConfig (customContextLength, kvCacheDtype, speculativeType,
specDraftNMax, tensorParallel, chatTemplateOverride, trustRemoteCode) persisted
to localStorage (unsloth_model_configs) with schema versioning + LRU budget.
KV-dtype and speculative value sets match main's sidebar (no q4_0/ngram-simple).
Reuses features/hub/lib/model-identity for normalization; adds storage-key layer
and applyPerModelConfigToRuntime (sets tensorParallel, which the old PR omitted).
* feat(picker): modular backend for chat-template validate + default fetch
New studio/backend/picker package (schemas/service/routes) mounted at /api/picker:
- POST /api/picker/validate-chat-template (Jinja syntax validation, no false positives)
- GET /api/picker/chat-template/{model_name} (default template from tokenizer_config.json,
reusing get_cache_path/resolve_cached_repo_id_case; graceful null, no model-code exec)
Frontend api/templates.ts client + hooks/use-model-defaults lazy cache. No backend
changes to the existing inference load route (per-model load fields already supported).
* feat(model-picker): bind picker on-device list to shared hub inventory
Picker now sources cached + local models from useHubInventory (the Hub's shared
store) via a thin adapter, replacing its own /api/models/* fetchers + module
caches. Hub, download manager, and picker now share one source of truth, so
completed downloads reflect in the picker automatically. Partial/live-download
rows are filtered from the cached lists (unchanged rendering). Local naming/search
preserved via additive LocalInventoryRow modelId/displayName. Variant expander,
scan-folder management, recommended-fit, search, external providers untouched.
Known minor: cached 'Downloaded date' sort tiebreak degrades to alphabetical
(hub cached rows carry no mtime); default 'recent' (load-time) sort preserved.
* feat(model-picker): per-model config step inside the picker
Picking a (non-external) model now opens an in-picker config view built from
main's current load controls (context length, KV cache dtype, speculative
decoding, draft tokens, tensor parallel) plus a chat-template editor backed by
the picker validate/default endpoints. 'Remember for this model' persists the
config per model+variant; Run forwards the config to the existing load flow via
meta.config. External models bypass the step. Two-view orchestration lives in
model-selector (single interception point); pickers.tsx call sites untouched.
trustRemoteCode dropped from PerModelConfig to preserve main's per-load consent.
* feat(chat): apply/persist per-model config through the load flow
handleCheckpointChange threads meta.config into the selection; stageOrLoad and
the autoload/Hub-run paths now apply the picker config (explicit pick or saved
remembered config) via applyPerModelConfigToRuntime before staging/loading, with
keepSpeculative set so a remembered speculative mode survives the model switch.
Replaces the old remembered-load-settings seeding (resolveInitialConfig now the
single source). SelectedModelInput carries config.
* refactor(chat): remove per-model load config from the right sidebar
The load knobs (context, KV cache, speculative, draft tokens, tensor parallel)
and the chat-template editor now live only in the picker config step. The sheet's
Model section keeps the staged Load/Cancel flow (config is applied at pick time);
sampling params, system prompt, and RAG are unchanged. Deletes the superseded
remembered-load-settings module + the store's applyRememberedLoadSettings action,
removes the now-dead sheet state/imports, and points the settings reset at
unsloth_model_configs. Delete-cleanup deferred (stale config is LRU-capped).
* fix(model-picker): remove leftover sidebar-staging cogwheel + empty Model section
The downloaded-variant gear (ModelLoadSettingsAction) staged a model straight
into the right-sidebar Run-settings flow -- the old 'configure before load' path
now fully replaced by the in-picker config step. Removed the gear + its component.
Also gate the sheet's 'Model' section to staged picks only (pendingSelection):
after the load-knob strip its content is staged-only, so it was rendering an
empty section header whenever a model was merely loaded.
* chore(chat): remove dead per-model-config setters + modelControlsDisabled
After the load-config UI moved into the picker, the store's per-model setters
(setKvCacheDtype/setSpeculativeType/setSpecDraftNMax/setTensorParallel/
setCustomContextLength/setChatTemplateOverride) had zero callers
(applyPerModelConfigToRuntime writes via setState), and the sheet's
modelControlsDisabled was unreferenced. Verified dead across the whole tree.
* fix(chat): config-step Load actually loads (ignore Load-on-selection)
Root cause: with Settings > Chat > 'Load on selection' turned OFF, the config
step's load went down the deferred-staging path -- opening the right sidebar with
'<model> is staged, not loaded yet / Choose Load model'. The in-picker config step
IS the deliberate load action, so its Load now loads immediately (or downloads +
auto-loads when not cached) regardless of the toggle. Renamed the button
'Run model' -> 'Load model' to match. Native/dropped picks still honor the toggle.
* refactor(chat,hub): retire 'Load on selection' — config step is the only load flow
The in-picker config step (and the Hub Run button) now fully supersede the old
stage-to-sidebar flow, so the Load-on-selection toggle is removed everywhere:
- chat stageOrLoad: every pick loads immediately, or downloads + auto-loads when
not cached (the previous default behaviour, now universal).
- hub Run: drops the stage branch; downloaded GGUFs load directly with their saved
per-model config (no collision with the chat config step — both end at selectModel).
- store: removed loadOnSelection field/setter/key/default; Settings>Chat toggle and
its settings-reset entry removed.
- staged sidebar section is now a download-progress view (auto-loads on completion).
No manual staging remains; stageModel is used only for background auto-load downloads.
* feat(model-picker): default chat template from GGUF + thread variant through config flow
Read the embedded tokenizer.chat_template from GGUF files (read_gguf_chat_template
in gguf_metadata) and use it as the per-model default. Plumb gguf_variant through
the picker service, /api/picker/chat-template route, frontend templates API, and
use-model-defaults so the right variant's template is fetched.
Also refine the picker config-page/model-selector wiring, drop the dead
ggufNativeContextLength runtime path, and add the per-model-config storage keys to
the settings prefs export.
* feat(model-picker): read safetensors chat template + hide editor where it has no effect
Resolve the default chat template for safetensors models: prefer the modern
chat_template.jinja, fall back to the tokenizer_config.json chat_template field,
then chat_template.json (multimodal processor), then the GGUF embedded template.
Applied to local dirs, the HF cache snapshot scan, and the HF remote fetch.
Hide the chat-template editor in the picker for safetensors models — the override
is only applied at load by the GGUF/llama.cpp backend, so editing it on safetensors
currently has no effect. GGUF keeps the editor. Nothing removed; the dialog stays
for when the safetensors apply path is wired up in a later branch.
* fix(model-picker): set legacy-migration flag only after the write succeeds
Set unsloth_model_configs_migrated only once writeMap confirms the migrated
map persisted, so a quota/storage failure no longer marks migration done and
silently drops the user's pre-existing remembered settings — the next load retries.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* MVP model picker fixes
* MVP picker config fix
* MVP safetensors config
* MVP max seq config
* MVP max seq fix
* Fix static max tokens cap ignoring model context
* Fix picker GGUF scan parity
* fix(studio): harden model picker config loading
Apply remembered per-model configs consistently from picker and Hub loads, keep default configs from overriding standing speculative settings, add config access for direct local GGUF files, and support saving or forgetting active model settings without a reload.
* Fix model picker config flow
* Fix model picker config loads
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Avoid recursive per-model config migration reads
* Apply the displayed context length when loading a GGUF
* Fix template validation, cached template lookup, and failed load rollback
- Validate chat templates with the loopcontrols extension so templates
that use break or continue tags pass the picker validator, matching the
inference renderer that already accepts them.
- Read the default chat template from the newest cache snapshot rather than
an arbitrary iterdir order, so an older cached revision no longer prefills
a stale template.
- Capture the runtime per-model config before a load and reapply it when the
load fails, so a failed switch leaves the active model context, KV cache,
template, and speculative settings as they were.
* Make chat template view only for safetensors models
Custom chat template overrides are applied at inference only for GGUF
models, which pass the template to llama-server. The safetensors backend
renders with the model built-in template and ignores the override, so
editing it would save a value that never loads. For safetensors the
config page now opens the template as a read-only preview with a note
that editing is not available yet. This can become editable once
inference support for custom safetensors templates lands in main.
* Fix model picker config edge cases
- Restore prior runtime config when a load no-ops for the active model
- Cap the picker validator request body via the protected prefixes
- Keep the GGUF context slider max above the loaded context
- Fetch subfolder chat templates for uncached Hub repos
- Show the compare side config when reopening the picker
* Keep saved GGUF context above the fallback ceiling
* Show the model config in the run settings sidebar
* Fix model config sidebar reset and context slider
- Stack the remember toggle and action buttons in the sidebar
- Reset the config to defaults instead of the loaded values
- Fetch the native context so the slider max is not the loaded value
* Fix model picker config and download regressions
- Run picker chat template routes off the event loop
- Depth and root guard local template directory scans
- Restore download manager flow for uncached hub picks
- Apply per model context length on reload
- Import model picker symbols from the feature barrel
* Fix model picker config and cached download sorting
- Restore load settings when a Hub run is rejected mid load
- Reuse one NumericValueInput instead of a duplicate copy
- Fix double decode of the model name in the template route
- Remove the unused reset-to-loaded settings action
- Fix cached model download sorting
* Fix model picker per-model config edge cases
Honor a saved or typed max seq length above the model's native context so
RoPE extended values are no longer clamped and silently overwritten. Allow
typing past native while the slider keeps native as a soft ceiling.
Guard the fetch success paths in use-model-defaults against an aborted
signal, and refetch when the HF token changes.
Hash the chat template content in the sidebar remount key instead of its
length. Enable reset for a GGUF whose native context is unknown, and floor
the context slider max so it can never fall below the min.
* Fix GGUF context auto-fit and gated model config token
Stop forcing a 32768 context when a GGUF native context is unknown so the backend auto-fits to VRAM again, while still honoring an explicit context edit.
Send the HF token as a query param so gated safetensors models resolve their max position embeddings.
Derive model default state during render to drop the set-state-in-effect calls.
* Fix native GGUF context ceiling and guard picker template reads
Restore the native context store field so the sidebar slider keeps the
full ceiling for drag and drop GGUFs. Limit local chat template reads to
the browse allowlist, skip malformed repo ids, and drop unused model
picker exports.
* Fix model picker lint boundaries
* Fix model picker review findings
Chat template editor never seeded its draft. Radix only calls onOpenChange
from internal events, so the seed in the nextOpen branch was dead and a model
with a saved override opened empty. Saving then cleared the override. Drop the
dead branch, treat draft as an untouched sentinel, and reset it on every close.
Uncached Hub picks could auto load a model after the user left the chat. Main
detached the staged pick on route exit and on chat context change. Carry the
context key on the pending pick and skip the load when it no longer matches.
Also clear configTarget when the picker closes, restore the onUpdated ref so
variant rows stop resubscribing on every parent render, skip the LRU write when
the entry is already most recent, import NumericValueInput relatively, and drop
the unused ModelUpdateAction barrel export.
* Preserve GGUF context on active reload
* Fix model picker per-model config regressions
- Stop reloading the already loaded model on re-pick
- Hide infra models from the chat picker
- Detect vision support on cached GGUF repos
- Honor saved maxSeqLength on auto load
- Restore default chat template for local GGUFs
- Warn on save failure and revert config on cancel
- Refetch picker inventory on open
- Persist read only per model config safely
* Fix stale model auto load
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix model picker numeric input sizing and constraints
Size value inputs to their content so long context lengths are not clipped,
restrict them to numeric characters, and stop the speculative decoding label
from truncating in the sidebar.
* Fix picker CI tests and harden chat template resolution for PR #6647
- tests: point the descender guard at the moved model-selector.tsx path
- tests: exclude the disabled Reload model button from the regenerate locator so .first targets the real Regenerate
- picker/service.py: reject symlinked template/gguf leaves that resolve outside the browse allowlist (HF cache reads unchanged)
- compare mode: resolve each pane's own remembered chat template instead of inheriting the other pane's from the store
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Protect future-schema per-model configs from deletion for PR #6647
savePerModelConfig already refuses to overwrite a stored config whose schema version is newer than this client understands, but deletePerModelConfig did not. Unchecking Remember on an older client therefore silently destroyed a newer client's saved config. Apply the same guard on delete and surface the blocked case through the existing saveFailed toast.
* Protect future-schema per-model configs from quota eviction for PR #6647
The save and delete guards already refuse to touch a stored config whose schema version is newer than this client understands, but the quota-eviction path did not, so a full store on an older client could still evict a newer client's config. Skip future-schema entries when evicting and fail the save if the budget cannot be met without them.
* Fix GGUF context persistence, compare context, and rollback settings for PR #6647
Persist a GGUF context override from the user's intent instead of collapsing it against the loaded context, which reintroduced the context-reset (f4838782cb reverted the native-baseline fix). model-config-page now collapses the saved value against native, and use-chat-model-runtime and chat-adapter retain the requested context on load so re-saving another setting keeps the override; a null request stays null so a VRAM auto-fit never becomes a stored override.
shared-composer: a compare pane with no explicit GGUF context now loads at native (0) like single-view, not the session maxSeqLength that silently shrank the shown context.
use-chat-model-runtime: restore the previous model's KV cache dtype and chat template on a failed-load rollback so it runs as it was, not with backend defaults.
* Preserve native path token when reloading the active model for PR #6647
handleReloadActiveModel rebuilt the selection without the store's activeNativePathToken, so reloading a file-picked GGUF after a settings change validated the display label as a repo/path and failed. Thread the active native token through the reload selection so native-loaded models reopen correctly.
* Make picker template validation resilient and accept HF generation tags for PR #6647
Import Jinja lazily inside validate_chat_template so a backend without the optional jinja2 package (GGUF-only installs) still starts instead of raising ModuleNotFoundError at import time. Register a no-op extension for the Transformers {% generation %} assistant-mask tag so pasting a valid HF chat template validates, matching the renderer, rather than being rejected as an unknown tag.
* Honor remembered compare config and parse processor chat_template.json for PR #6647
* Fix failed-load rollback context and processor template map fallback for PR #6647
* Restore speculative decoding config on failed-switch rollback
When a model switch fails after the previous model was unloaded, the
rollback reload restored tensor_parallel, KV cache dtype and the chat
template override, but omitted speculative_type and spec_draft_n_max and
cleared their loaded shadows to null. The previous model therefore came
back running at backend defaults (speculation off) while the UI still
showed it enabled, and the status resync confirmed the off state. Resend
the previous model's speculative settings in the rollback load and keep
the store's active and loaded speculative fields in sync with them.
* Reset max sequence length when a model has no saved config
applyPerModelConfigToRuntime reset every per-model field except
maxSeqLength, which it only wrote when the incoming config had one.
maxSeqLength is the sole field carried on store.params, so selecting a
model with no remembered config left the previous model's value in place
and later loaded the new model at that leaked length. Fall back to the
standing default so an unremembered model loads at its own default.
* Surface a message when a variant update cannot start
startManagedUpdate handled the conflict and error start outcomes but let
busy fall through as if the update began, so the confirm dialog closed
with no job created and the cached variant stayed stale. Show an info
message when the repo is busy with a sibling transfer so the click is
not silently dropped.
* Keep per-model speculative choices out of the global default
A staged load with a per-model or one-off config sets keepSpeculative,
which already skips reading the global speculative preference. The
matching save still ran unconditionally, so the model-specific choice was
written to the global unsloth_chat_speculative_type and a later model with
no saved config started from it instead of Auto. Skip saveSpeculativeType
when keepSpeculative so the per-model choice stays isolated.
* Seed non-active model settings from the app default max length
The Run settings page captured initialMaxSeqLength from the loaded
model's runtime params and fell back to it for a model with no saved
config. Opening settings for a different, unloaded model and clicking
Load then sent the active model's context (for example 64k) instead of
the 4096 default, risking validation failures or OOMs. Seed the default
for non-active models and keep the runtime value only for the active one.
* Prefer sidecar tokenizer chat template over the GGUF copy for variants
_chat_template_from_dir returned the embedded GGUF template first when a
variant was selected, reversing the tokenizer-first precedence of the
no-variant path. A model whose chat_template.jinja or tokenizer_config.json
supersedes a stale embedded template then got the wrong template on
variant selection. Keep tokenizer files first regardless of variant; the
variant only picks which GGUF is the fallback. Adds regression tests for
both the tokenizer-wins and gguf-fallback cases.
* Keep per-model speculative choices load-local in autoload and compare
The interactive load path treats a per-model speculative choice as
load-local and skips writing it to the global default. Autoload and
generalized compare still called saveSpeculativeType unconditionally, so a
remembered off or ngram setting leaked into unsloth_chat_speculative_type
and later models with no saved config inherited it. Persist the global
preference only when the value came from the global settings.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: record the compare pane's loaded context in runtime state so the active model's settings and any reload or save use it, not the previous context
* Studio: notify the user when a Hub autoload can't start because another download for the model is already running, instead of silently dropping it
* Studio: drop the merge's orphaned staged-model store helpers and unused alert imports
The main merge left isPendingGguf and pendingSelectionMatches referencing the
removed PendingModelSelection type, and the alert-dialog/alert imports unused
after the permission-mode dropdown replaced the bypass dialog, so tsc -b failed.
* Studio: cache a null default chat template so the viewer stops re-fetching it
A model with no sidecar or embedded template resolves to a terminal null, but
that result was never cached, so reopening the template viewer re-ran the
backend and Hugging Face lookup every time.
* Studio: detect direct-file GGUFs in run settings so Max Tokens uses their context
A GGUF loaded from a local file or custom folder has no variant label, so the
run-settings panel treated it as non-GGUF and clamped Max Tokens to the session
max_seq_length instead of the loaded GGUF context. Detect it via the reported
GGUF context and the .gguf checkpoint suffix, matching the chat page.
* Studio: prompt to re-select a local model file when its lease expired before reload
A file-picked GGUF is reachable only through a native path token that the
desktop host prunes after a TTL. Reloading reused that token blindly, so a
reload long after the initial load failed with an opaque error. Track the
token's expiry and, when it has passed, ask the user to re-select the file
instead of attempting a doomed reload.
* Fix descender-clipping test to tolerate sidebar layout utilities
The sidebar account-block div carries layout utilities (min-w-0, flex-1)
between 'flex' and 'flex-col', so the descender-clipping guard's regex,
which required 'flex' immediately followed by 'flex-col', no longer matched
and the test failed to locate the account-block div. Generalize the prefix
to allow intervening flex utilities while still capturing the leading-*
class before the collapsible visibility utility and asserting leading-tight,
so the guard against clipped glyph descenders is fully preserved.
* Harden picker chat-template resolution
Enforce the 64 KiB chat-template contract at the validate endpoint's request
model so a direct caller cannot submit a template far larger than the frontend
allows (MaxBodyMiddleware only bounds the whole request body, not this field);
oversized templates now return a clean 422.
Apply sidecar-over-GGUF template precedence globally across cached snapshots
instead of per snapshot. A repo with multiple cached revisions previously
returned the first snapshot's template, so a newer GGUF-only revision could
win over an older revision's maintained chat_template.jinja sidecar, which
contradicted the documented intent that sidecars supersede the embedded copy.
* Guard per-model config against future-schema and lossy migration
Two forward-compatibility gaps in the versioned per-model config store:
- The load/apply path returned and normalized a stored record without checking
its schema version, so a record written by a newer client was reinterpreted
under the current schema and applied to a live model load, even though save,
delete and eviction all refuse to touch future-schema records. Reject
future-schema records on load too.
- The one-time legacy migration enforced the storage budget without protecting
the entries it had just migrated and set the completion flag unconditionally.
When storage was already full of future-schema records (which are unevictable
by an older client), the migrated entries were the only evictable ones and
could be dropped while migration was still marked complete. Protect the
migrated keys during eviction and only mark migration complete when they
survive, so it retries once space frees up.
* Discard chat-template validation results after the dialog closes
Server-side template validation is async, but closing or cancelling the editor
did not abort it, so a late-arriving valid response still called onSave and
applied a template the user had already dismissed. Track a validation token
that is bumped on close and ignore any validation result whose token is stale.
* Record native lease expiry when loading a picked GGUF from the chip
The pending-native-model chip loaded via stageOrLoad directly, bypassing
loadNativeModelIntent, so activeNativePathExpiresAtMs was never recorded for a
chip-loaded file. A later reload then either skipped the lease-expiry guard
entirely (expiry left null) or compared against a previously loaded file's
stale expiry, so reload could reuse an already-pruned token or wrongly block a
still-valid one. Route the chip through loadNativeModelIntent, which builds the
same selection and records the expiry.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Prefer sidecar template for a directly selected local GGUF file
A direct .gguf file path read its embedded chat template without checking the
parent directory for a maintained sidecar (chat_template.jinja /
tokenizer_config.json), while directory and variant selections already prefer
the sidecar. That let the config editor preview or save a stale embedded
template for the same model depending on how it was selected. Check the parent
directory sidecars first, then fall back to the embedded copy, and cover both
paths with tests.
* Resolve cached chat template per revision, newest first
The earlier change searched every cached snapshot for a sidecar before
considering any snapshot's embedded GGUF template, which let an obsolete sidecar
from an older revision override the newest revision's template. Restore
per-snapshot resolution (newest first): a revision's sidecar still supersedes
its own embedded GGUF copy, but a newer revision is no longer overridden by an
older revision's sidecar.
* Preserve autoload transport conflicts and surface background busy downloads
- When a Hub autoload hits a transport conflict, keep pendingHubAutoLoad bound
instead of clearing it. Clearing it re-keyed the download surface and its
cleanup cancelled the conflict the toast tells the user to resolve, so the
Hub resume affordance was gone the moment it appeared. Return early on
conflict, mirroring the started branch, so resolving it from the Hub still
auto-loads on completion.
- The background-download branch handled started and conflict but silently
dropped a busy outcome, leaving the user with no feedback when a peer variant
of the same repo was already downloading. Surface the same busy toast the
autoload path uses.
* Fix context length, GGUF template, fetch state and lease expiry bugs
Keep explicit context length values instead of collapsing to null at
native. The collapse made the slider jump back at the native maximum
and made Reload load the previous context instead of the chosen one.
Prefer the first split when resolving a GGUF without a variant. Later
splits carry no chat template metadata, so picking the largest file
could return no template for a sharded model.
Clear stale fetch state when template and metadata lookups retry, so
a previous terminal error is not shown while a new fetch is running.
Record native path lease expiry together with the token when a load
commits. The expiry was written by only one load path and even when
the load did not start, so a reload could be blocked with an expired
file message for a still valid token.
* fix(model-picker): resolve review findings across config, inventory, and templates
- Apply remembered per-model config in the training-compare chat handoff so a
prior model's customContextLength no longer leaks into the next load
- Match GGUF variant labels with the inventory extractor too, so cached
no-quant-token files resolve their default chat template
- Show "Auto" instead of a fabricated 32768 when native context is unknown
- Reuse the identical staged auto-load object on same-pick so a re-pick during
download pre-flight no longer disarms auto-load via "busy"
- Union supports_vision when deduping cross-cache inventory rows
- Serve hidden-model needles from a new GET /api/hub/hidden-models endpoint and
merge them client-side, covering runtime-configured RAG embedders
- Clamp GET chat templates to MAX_CHAT_TEMPLATE_BYTES (route + jinja sidecar),
matching the validate endpoint's contract
- Lower-clamp stored customContextLength to shared CONTEXT_LENGTH_MIN
- Wipe unsloth_chat_load_on_selection in Settings "Reset all"
- Drop stale pendingHasContext comment describing deleted staging machinery
* Fix stale defaults cache, token in query string and rounded up context ceiling
Refresh cached chat template and max position data when a model update
completes. Send the HF token for model config requests in the dedicated
header instead of the URL. Snap the native sequence length ceiling down
to the nearest step so the slider cannot exceed the declared maximum.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix compare pane reverting active checkpoint on non-GGUF load
Re-read runtime params after setCheckpoint so the fresh checkpoint is
kept instead of being overwritten by the pre-setCheckpoint snapshot.
* Send the HF token via header for the vision and embedding checks
checkVisionModel and checkEmbeddingModel still passed the HuggingFace
token as a ?hf_token= query parameter, so it landed in server access
logs, proxy logs, and browser history. Move them to the
X-Unsloth-HF-Token header like getModelConfig already does, and accept
the header on the check-vision and check-embedding routes with the
existing query parameter kept as a fallback for older clients.
* Cap the chat template on the model load path
The load endpoint accepted an unbounded chat_template_override, so a
direct caller could hand llama.cpp an arbitrarily large Jinja template
even though the frontend, the validate endpoint, and the read paths all
enforce the 64 KiB limit. Reuse MAX_CHAT_TEMPLATE_BYTES in the
LoadRequest validator, rejecting oversized templates with a fast
character-count check before the exact UTF-8 byte check.
* Protect existing per-model configs during legacy migration
When the one-time legacy import pushes the store over budget, eviction
now protects the entries the user already has and drops only the
just-migrated legacy entries, so importing old load settings can never
discard a newer per-model config.
* Reset clears the context override instead of pinning the native value
Reset wrote the discovered native context into customContextLength for
GGUF models, but isDefaultConfig treats any non-null customContextLength
as an explicit pin, so Reset with Remember enabled persisted a fixed
context and future loads stopped using the native auto context. Reset
now restores the full default (customContextLength null); the native
value is still shown through the existing display fallback.
* Bound chat-template sidecar reads to a size limit
The chat_template.json, tokenizer_config.json, and Hub-downloaded sidecar
readers decoded and json-parsed the whole file before the extracted
template hit the 64 KiB response cap, so an oversized metadata file could
exhaust memory. Read them through a bounded reader (4 MiB envelope) that
returns None when the file is larger, matching the existing chat_template.jinja
size guard. Adds tests for oversized tokenizer_config.json and chat_template.json.
* Keep the native-path token and lease expiry in sync
Rollback after a failed reload restored the previous token but left the
failed load's expiry in the store, so a later reload could be falsely
blocked as expired (token A paired with load B's lease). Restore the
previous lease alongside the token, and clear the expiry wherever the
token is cleared on a non-GGUF transition, so the two never diverge.
* Clear the native file lease on compare-pane loads
* Studio: add regression tests for the model-picker per-model-config
Guard the specific regressions that reverted the predecessor change:
- backend pytest (studio/backend/tests/test_model_picker_regression.py):
infra-model hiding, HF token via header with query fallback, and the
chat-template byte caps.
- source contracts (tests/studio/test_model_picker_contracts.py): the token
stays out of the URL, the context ceiling is floored, the native lease is
cleared on compare-load and restored on rollback, the default caches key on
the inventory version, and the hidden needles stay present.
- Playwright E2E (tests/studio/playwright_model_config.py) wired into
studio-ui-smoke.yml on port 18898: Context Length persists across a reload,
Reset clears the stored override, and infra models are absent from the picker.
- optional GPU-gated inference smoke (tests/studio/test_gpu_inference_smoke.py)
that auto-skips on GPU-less CI and stays short on a GPU.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: model pinning, row menus, hub inference settings, and inventory filters
Pinning
- Add a pinned models store (localStorage) with repo and per-quant pins
- Pinned section in the model selector's On Device list and the hub inventory,
with newest pins first so Pin to top lands on top
- Deleting a repo drops its pins
Row menus
- Replace loose row icons with a shared 3-dots menu (pin, reveal in file
manager, copy identifier, copy path, delete) on picker rows, hub quant rows,
the hub run bar, and on-device inventory rows
- Menus only render for models actually on disk; platform-aware reveal labels
- Backend: cached-model-path and reveal-cached-model endpoints resolving
managed HF-cache repos only
Hub inference settings
- Gear in the GGUF run bar opens an Inference settings dialog reusing the chat
page's controls: model config (context length, KV cache, speculative
decoding, chat template), system prompt, reasoning, sampling, tools and
retrieval
Inventory
- Model-type filter (text, vision, embedding, STT, TTS, diffusion) beside the
sort pill, both with a sort icon, capped widths and truncation so the
On device heading never wraps
- Unsloth-owned repos without an upstream provider logo fall back to the
Unsloth mascot avatar
- Discover / On Device tabs widened; hub search bar narrowed to match
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: revert the Unsloth mascot avatar fallback
Unsloth-owned repos without an upstream provider match go back to the
colored-initial tile, and unslothai is no longer a relabeled owner.
* Studio: run-bar options on single models, and aligned type/capability filters
- Give single-model (non-GGUF) run bars the same 3-dots options menu and
settings gear as GGUF, at repo level
- Drop Pin to top from the run-bar menus; pinning stays in the On Device list
- Add an Image to text (diffusion) capability with detection, and surface it
in both the hub Discover capability filter and the On Device type filter
- Align the On Device type filter with the Discover capability options and
share the same detection so both dropdowns match
* Studio: apply hub inference config on reload, eject action, and run-bar polish
- Fix inference settings not applying: the hub dialog now writes the config to
the runtime before reload, matching the chat page (selectModel reads runtime
state, not the selection)
- Order the settings gear before the 3-dots menu in the run bars
- Replace the loaded-model run-bar action (New Chat) with Eject, wired through
the inspector to the hub's ejectModel
- Truncate the results heading so a long search query clips instead of
overlapping the header pills in split view
- Use a plain magnifying-glass icon for the no-results empty state
* Studio: fix GPU settings loss, load guards, pins, filters, and cached paths
Reloading a model from the chat sidebar or the hub gear dialog rebuilt the
per-model config without the GPU memory fields, so manual GPU layers, MoE
placement, and the GPU pick were reset on every reload and could be saved
over a remembered config. The active config now comes from a shared
useActiveModelConfig hook that carries the GPU fields for GGUF models, and
the sidebar remount signature tracks them through a shared gpuFieldsSignature
helper.
The in-flight load guard lived in a ref inside each useChatModelRuntime
instance, so the chat page, hub page, and gear dialog could not see each
other's loads. A load started from the gear dialog left the hub page free to
eject the model mid-reload or start a second concurrent load. The runtime
store now records the loading pick, selectModel checks it across instances,
and ejectModel refuses with a toast while any load is in flight.
The cached-model-path endpoint matched GGUF files by basename and excluded
only mmproj, so Copy path and Reveal could return an MTP drafter for a quant
and returned 404 for directory layouts like BF16/model-00001.gguf. Variant
files are now resolved from snapshot-relative paths with the same drafter,
mmproj, and big-endian exclusions as the load path, shared through a new
_main_variant_gguf_label helper.
Hub and picker fixes:
- rename the diffusion capability label from "Image to text" to
"Image generation", since it detects image generators
- validate pinned quants through the cached variant listing, keep the last
verified set while revalidating, and drop deleted quants immediately
- pass a measured scroll margin to the on-device virtual list so rows past
the overscan stay visible below the pinned block
- keep the delete menu for stopped partial safetensors downloads
- give the inventory type filter a reset in Clear filters, a truthful empty
state with a Show all types action, and hide it on the datasets view
- order picker pinned rows by pin recency, include pinned matches in the
empty-state check, and sync pins across browser tabs
- count only the visible rows in the On device list header
Tests: contract checks for each fix in test_model_picker_contracts.py and a
backend test for the variant label selection.
* Studio: reveal cached models in Windows Explorer under WSL
The reveal endpoint only branched on macOS, Windows, and generic Linux.
Under WSL the Linux branch spawned xdg-open, which is missing on a stock
distro without a Linux desktop, so the request failed with a 500 and the
UI showed a failed to open file manager error.
WSL is now detected with the existing helper and the path is converted
with wslpath before opening explorer.exe, selecting the file the same
way native Windows does. Directories open directly. When interop is
unavailable the old xdg-open fallback still runs. The macOS, native
Windows, and native Linux branches are unchanged, and the Tauri app is
covered since its hub reveal calls this same local endpoint.
Tests: platform guards for the WSL reveal, the interop fallback, and
the unchanged native Linux behavior in tests/studio/test_reveal_file_manager.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Adjust model picker row spacing and cogwheel hover consistency
* Studio: exact hidden model ids and newest revision cached paths
A custom RAG embedder repo was published to the frontend as a basename
substring needle, so a generic name like org/model could hide unrelated
models in the pickers. The hidden-models endpoint now sends full repo ids
that are matched exactly.
Copy path and Reveal picked a GGUF variant from an arbitrary cache
revision when the same file existed in more than one. The newest revision
now wins, matching the whole repo lookup.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix model picker GPU config, metadata, and cache selection
Load each compare model with its saved GPU memory mode, GPU layers, CPU MoE layers, and selected GPU IDs. Reconcile saved GPU IDs with the current hardware. Include the active native GGUF path token in metadata checks. Search all Hugging Face cache roots when resolving cached models and select the largest visible cache entry. Remove obsolete barrel exports and the staging-only GPU memory helper.
* Studio: hide hub inference settings gear for now
The cogwheel in the hub download cards is out of scope for this PR. The
dialog component stays in place and a TODO marks where the button
returns in a future PR.
* Refresh hidden model matchers
* Fix GGUF detection, compare context pin, and picker delete staleness
Treat any pick with a GGUF variant as GGUF in selectModel so the first
load after downloading an uncached quant validates and sizes with the
right GPU settings instead of unloading the current model on a wrong
preflight. Variant picks now also set isGguf on their selection meta.
Stop compare panes from inheriting the active model's context pin when
their own saved config says Auto. Null context in a remembered config
now means no pin, matching how the pane settings are shown.
Route picker deletes through the hub inventory client, which
invalidates the HF cache scan and the variants cache. The legacy
delete route left the scan cache warm, so deleted models reappeared
in the picker until the TTL expired. Removed the now unused legacy
delete client and updated the contract test to match.
* Studio: fix stale GGUF load-marker ordering test
The load-in-flight marker still precedes the hub-download guard and the
unload, but the llama_extra_args inheritance that used to sit between the
marker and the guard now runs ahead of the GGUF branch, so it is no
longer a landmark inside the sliced source. Drop it from the ordering
assertion and keep the marker -> guard -> unload invariant.
* Studio: fix per-model config edge cases in compare loads and saved defaults
- chat-settings-sheet: gate the MTP fallback note and context/VRAM warning on
the broader isGguf (variant, loaded gguf context, or .gguf suffix) instead of
isLoadedGguf, so direct-file and custom-folder GGUF loads still surface
those diagnostics.
- shared-composer: a compare pane's context now comes from its own config only
(a saved pin, else null for Auto/native). It no longer inherits the active
model's shared snapshot, which resolveFitMaxSeqLength treated as an explicit
pin and could load a pane at another model's context (VRAM/OOM), matching the
single-model load path.
- model-config-page: when an auto-fit GGUF is saved with fixed GPU layers
(Manual) and Remember, pin the displayed fitted context so a later fresh load
keeps the placement instead of sending native/0 and recreating the OOM.
- per-model-config: treat Auto GPU memory mode and Auto/default speculative type
as follow-global defaults; do not persist them as per-model overrides so later
global preference changes keep applying.
* Studio: gate vision capability on GGUF projectors and bound remote template downloads
- cache_inventory: only mark a cached repo vision-capable when it holds an actual
GGUF mmproj projector, not any file whose name merely contains "mmproj" (e.g.
mmproj_config.json), matching the runtime's GGUF-only projector detection.
- picker/service: pre-check the remote file size before downloading an uncached
repo's chat template / tokenizer config, so a maliciously large sidecar is
skipped instead of fetched and retained in full, mirroring the size gate the
local-file path already applies.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add source-contract guards for the per-model-config edge-case fixes
Guard the four per-model-config fixes against silent regression in CI:
- local GGUF diagnostics gate on the broad isGguf, not the variant-only isLoadedGguf
- fixed-layer GGUF saves pin the displayed context
- Auto GPU mode and Auto/default speculative are not persisted as per-model overrides
- a compare pane's context comes from its own config, not the active model's snapshot
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: clear manual GPU knobs on Default and resolve local embedders before repo-id
- model-config-page: switching GPU Memory back to Default now clears the Manual-only
knobs (gpuLayers/nCpuMoe/selectedGpuIds); otherwise a remembered config kept stale
pins that a later load re-applied when the global GPU preference was Manual, despite
the page showing Default.
- routes/models hidden_model_matchers: resolve an existing local path before the repo-id
regex, mirroring is_hidden_model, so a local embedder shaped like "models/embedder" is
hidden by exact path instead of leaking as a chat model.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add _is_mtp_drafter to the model_config stub in the export-paths test
routes/models.py imports _is_mtp_drafter from utils.models.model_config at module
load, but the lightweight stub in test_export_absolute_paths.py did not provide it,
so loading the module under the stub raised ImportError on Backend CI. Add the stub.
* Studio: read a picked GGUF's chat template through the native path lease
The picker chat-template GET has no native-path-lease plumbing, so a
desktop-picked (drag-drop) GGUF could not show its default chat template
in Run Settings until the model was loaded: the endpoint only receives
the display label, not the leased file path.
Read the embedded template through the existing lease-aware
/api/inference/validate probe instead. A new include_chat_template flag
resolves the granted canonical path and returns the GGUF's own embedded
template, never a sibling sidecar (the grant authorizes just that one
file); it skips the training guard like include_context_length and is
bounded by MAX_CHAT_TEMPLATE_BYTES. The frontend fetch mints a one-shot
validate-model lease when a native token is present and keeps the plain
GET path for HF and allowlisted local models.
Adds backend and source-contract regression tests.
* Studio: call worker.direct_wheel_url in the ROCm wheel-url test
The ROCm Mamba/SSM test referenced worker.py's private _direct_wheel_url,
but the worker imports the wheel helper under its public name
direct_wheel_url (utils.wheel_utils). When the worker module loads (its
imports resolve in CI), worker_mod._direct_wheel_url raised AttributeError;
the test only masked it by skipping when the worker could not be imported.
Call the name that actually exists so the assertion runs; it still returns
None for an empty cuda_major (ROCm).
* Studio: reset max sequence length to the app default, not the loaded value
For a non-GGUF active model, the per-model config seeds maxSeqLength from
the loaded runtime value so the panel opens showing the running context.
Reset set config.maxSeqLength to null, but the null fallback resolved back
to that captured runtime value, so the field kept showing the old custom
length and the config saved/reloaded it again. A remembered or active
max-length override therefore could not be cleared from Run settings.
Fall the null/default case back to the app default (clamped to the model's
native ceiling) instead of the active runtime snapshot, so Reset actually
clears the override. The initial view is unaffected: an active model's
config.maxSeqLength is already non-null, so it still shows the loaded value.
Adds a source-contract regression guard.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: persist default max length, refresh deleted quants, hide non-chat locals
Three follow-up fixes from review of the per-model-config picker:
- Max sequence length: the persisted per-model record now keeps config's
maxSeqLength (null after Reset) so isDefaultConfig can clear a remembered
override; the resolved app-default is substituted only into the load
request, never the saved record. Previously Reset saved the concrete
default and left the model pinned/remembered.
- GGUF variant expander: deleting a downloaded quant from a repo that still
has other cached quants now bumps the expander refresh key, so the removed
quant stops showing as downloaded and clickable (which would try to reload
the deleted file) until the repo is collapsed and reopened.
- Local picker rows: require capabilities.canChat before listing a local
models-folder / LM Studio row. A weightless folder (only config.json) is
classified non-chat, and toLocalModelInfo drops capabilities, so selecting
such a row would try to load a path the inventory already marked non-chat.
Adds source-contract regression guards for all three.
* Fix compare-pane and Reset context defaults in model picker
Two related per-model-config default regressions:
- A non-GGUF compare pane with no saved maxSeqLength fell back to the
active model's shared runtime snapshot, so comparing a saved 128K model
against an unconfigured pane loaded the latter at 128K and could OOM. It
now falls back to the shared app default (DEFAULT_MAX_SEQ_LENGTH), the
same fallback the single-model config path uses.
- contextAtDefault treated an explicit customContextLength equal to the
native ceiling as a default, which wedged the Reset button disabled for
a deliberate pin-to-native. It now counts as default only when there is
no override at all.
DEFAULT_MAX_SEQ_LENGTH becomes a single exported constant in
per-model-config.ts so the single-model config and the compare path share
one source of truth. Adds source-contract guards for both fixes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Skip over-cap remote Jinja templates so the tokenizer template wins
The remote chat-template resolver bounded raw chat_template.jinja downloads
only by MAX_TEMPLATE_METADATA_BYTES (4 MiB), then returned the first
non-empty Jinja unconditionally. The picker route drops any template larger
than MAX_CHAT_TEMPLATE_BYTES (64 KiB), so an uncached repo whose
chat_template.jinja sits between 64 KiB and 4 MiB returned no template at
all, even when a valid smaller tokenizer_config.json template existed. The
local path already skips oversized .jinja files and falls through.
Gate the extracted Jinja on MAX_CHAT_TEMPLATE_BYTES and continue searching
when it exceeds the cap, matching _chat_template_from_jinja_file. The 4 MiB
download bound stays for JSON files that merely embed a small template. Adds
a regression test that a big Jinja plus a valid tokenizer config resolves to
the tokenizer template.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard legacy per-model-config migration idempotency
The v1->v2 localStorage migration (unsloth_load_settings ->
unsloth_model_configs) runs on every store read, so it must migrate exactly
once and never re-run, duplicate, or clobber a newer per-model config on a
reload or restart. That was covered only by a manual proof, so add durable
guards:
- Source-contract test pinning the three idempotency layers (the in-memory
legacyMigrationChecked guard, the persistent unsloth_model_configs_migrated
flag set in every terminal branch, and the non-overwriting Object.hasOwn
merge-skip) plus the readMap invocation. Reddens if any layer is dropped.
- Playwright model-config E2E: promote the legacy-migration step to a gating
check (soft_fail, which gates under the CI STUDIO_UI_STRICT=1) that the
migrated value is preserved and the flag is set, then reload again with a
fresh legacy seed present and assert the stored key set is unchanged, so a
second reload cannot re-migrate, duplicate, or clobber.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Note the migration E2E now gates idempotency under STUDIO_UI_STRICT
* Tighten model-picker per-model-config code comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: shimmyshimmer <shimmyshimmer@users.noreply.github.com>
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
* Studio: validate Hugging Face tokens before use
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep token validation failures non-blocking
* Studio: harden Hugging Face token preflight
* Studio: make token validation effect lint-safe
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: Data settings tab, uploaded files manager, quant pinning, image preview fix
Settings
- New Data tab in the settings sidebar, under Connections. Chat data
management (archived chats, confirm before deleting, exports, import,
clear all) moved there from the Chat tab.
- New Archive all chats action with confirmation. Archives every chat in
Recents and Projects; compare pairs count as one chat.
- New Uploaded files manager listing RAG documents (chats, projects,
knowledge bases) and chat message attachments with location, size and
date. Files can be opened in a new tab or deleted. Deleting a chat
attachment keeps the message text.
Backend
- GET /api/rag/documents lists all uploaded RAG documents with file size
plus KB and project names.
- GET /api/chat/attachments lists chat message attachments; per
attachment file and delete endpoints included.
Model selector
- Downloaded GGUF quants can be pinned from the quant row (next to the
settings and delete actions). Pinned quants show at the top of On
Device under a Pinned heading as model name plus a grey quant chip and
load directly with one click. Non GGUF cached repos pin as a whole.
- Toned down the green of the downloaded label.
Fix
- Clicking an image attachment in chat now opens the preview overlay.
The tooltip trigger wrapper called preventDefault before composed
handlers ran, which made Radix DialogTrigger skip opening.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: image previews and file type chips in uploaded files list
Image attachments now show a small thumbnail (lazy loaded from the
stored bytes, object URL revoked on unmount) and every row shows a grey
uppercase type chip derived from the extension or content type. Non
image rows keep a file icon. Name cell floors its width and clips
overflow so narrow dialogs stay aligned.
* Harden attachment serving, add tests, and polish pinned rows and previews
- Strict base64 decoding for attachment files: corrupt payloads now return
422 instead of silently serving empty or garbled bytes; whitespace,
missing padding, the URL-safe alphabet, and RFC 2397 percent-encoded
data URLs are all handled
- New backend test suite covering attachment listing, size accounting,
malformed rows, deletion semantics, and every file-serving edge case
- Pinned quant rows show a Loaded tag when that exact quant is active,
and reveal unpin, settings, and delete actions on hover
- Uploaded files dialog is wider and chat locations link straight to the
thread the attachment belongs to
- Chat image preview is now a chrome-free lightbox: dimmed backdrop,
rounded image, corner close button, click outside to dismiss
- File opens go through a synchronous window.open so Safari and Firefox
popup blockers do not eat them
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Uploaded files: click a file to jump to its chat, square thumbs, new Data icon
- Clicking a file row (thumbnail or name) now goes straight to the chat it
belongs to; files without a chat open directly as before
- File thumbnails pin a small 7px radius: the theme scales rounded-md up
to a near circle at this size
- Settings Data tab now uses the database-setting icon
* Uploaded files is now a Data tab subpage instead of a popup
- Manage swaps the tab body for an inline Uploaded files page with a back
header, matching the rest of settings navigation
- Size column header and values are left aligned like the other columns
- Column widths tightened so the table fits the settings panel
* Lightbox polish and Data tab row order
- Image preview close button is transparent until hovered
- Preview image no longer rounds its corners
- Import chats now sits below Clear all chats in the Data tab
* Data tab: export chats as fine-tuning data and open them in Recipes
- New Fine-tuning section in Settings > Data converts every chat into a
JSONL dataset in the OpenAI messages format, one conversation per line
with string-only system/user/assistant turns
- The Train tab detects this file as chatml natively: no column mapping
and no standardization pass, and it works with train on completions
since every assistant turn sits behind the chat template response marker
- Consecutive same-role turns merge, trailing turns without an assistant
reply drop, and reasoning, tool calls, and images are excluded so chat
templates format the data cleanly
- Open in Recipes stages the JSONL as a local seed upload, creates a new
Data Recipe with the seed block preconfigured, and jumps to the editor
* Data tab: load chats straight into the Train tab, row moved to the top
- New Load in Train tab button uploads the fine-tuning JSONL through the
training dataset endpoint, selects it in the training config store, and
opens the Train tab with the dataset loaded and format-checked
- Use chats as training data now sits at the very top of the Data tab
- The Chats subheading is gone; chat rows flow directly under it
* Address review findings on the uploads manager and quant pins
- Deleting the last attachment stores '[]' instead of NULL: a NULL reads
back as a missing field and triggers the legacy IndexedDB backfill,
which resurrected the deleted attachment on the next chat load
- The attachment file endpoint now serves audio: adapter parts store
{data, format} raw base64 and compare chats store a bare base64 string;
media type comes from the attachment contentType or the format
- Compare-chat uploads live in message content parts, not attachments;
the uploads list now includes those blobs via synthetic content-part
ids that the same get and delete routes resolve
- Deleting a quant from the expanded repo row also unpins it so a pinned
row cannot try to load a file that no longer exists
- Thumbnails in the uploads list fetch their blob only once the row is
visible, so a long screenshot history does not download everything
- Nine new backend tests cover audio serving, content-part listing,
serving, deletion, and the empty-list delete behavior
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Data tab: single action dropdown with format choices for chat training data
- The three fine-tune buttons collapse into one dropdown plus a run
button; pick Load in Train tab, Open in Recipes, or Export JSONL,
then click the arrow to run it
- The dropdown's Format section adds ShareGPT and Alpaca alongside the
default OpenAI messages format, ticked like a checklist; all three
shapes are auto-detected by the Train tab's format check
- Alpaca is single-turn, so each user to assistant pair becomes its own
record with the system prompt and earlier turns carried in the input
column
- Shorter description on the training data row
- Uploaded files rows show the size under the file name instead of a
separate column, matching the tighter layout
* Polish the training data action control
- Run button is a true circle (icon-sm plus rounded-full) with a
heavier arrow stroke
- Dropdown trigger uses the shared standard chevron and a fixed width
so switching actions no longer resizes the control
* Shorten the training data row description
* Use the standard chevron for the run button and enlarge the ticks
- Run button uses the shared standard right chevron so it matches the
dropdown chevron instead of the hugeicons arrow
- Dropdown ticks bumped up a size for legibility
* Reword the training data row description
* Shorten Data Recipes to Recipes in the training data description
* List Export JSONL first and rename the default format to Chat Completions
* Handle legacy string content in fine-tune exports and gate Train on chat-only hosts
- messageToPlainText now accepts plain-string message content, the shape
legacy and imported histories store, so those conversations export
instead of being skipped as having no exchange
- The Load in Train tab action is disabled on chat-only hosts the same
way the sidebar gates Train; the default action falls back to Export
JSONL there so the run button never uploads a dataset that /studio
would immediately redirect away from
* Narrow the training data action dropdown slightly
* Drop the format picker from the training data dropdown
Chat Completions (OpenAI messages) is the only export format we ship, so
the ShareGPT and Alpaca options and the Format section are removed. The
export always uses the OpenAI messages shape.
* Address the second round of review findings
Security
- Chat attachment data URLs no longer echo their embedded media type:
anything that is not a plain raster image serves as octet-stream, so
imported text/html or SVG payloads cannot render under the app origin
- Uploaded .html/.htm RAG documents serve as text/plain for the same
reason; the preview sheet only uses the file URL for PDFs
Uploads manager
- Remote image URLs in imported chats are no longer listed as stored
uploads (nothing to serve, and delete would strip the chat reference);
the delete guard mirrors the same data:-only rule
- Deleting a content-part upload refetches the list since the remaining
parts re-index, keeping sibling row ids current
- Deleting a project document from the Data tab invalidates the project
sources cache like the sources panel does
- Data-tab deletions now patch the loaded thread's in-memory copy via a
small event, so a later repo sync cannot write the attachment back
Fine-tune export
- Branch siblings from retries stay out of the exported conversation;
only the selected chain converts (full exports still keep everything)
- Assistant turns before the first user turn drop, preserving leading
system prompts, so no unconditioned assistant targets are emitted
Four new backend tests cover the media type clamp and remote-URL rows;
two existing tests updated for the clamped types
* Fix uploaded file lifecycle and model state
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make archived chats a Data settings subpage
* Studio: fix attachment route tests and pinned quant edge cases
- test_chat_attachments: drop asyncio.run around the synchronous
/attachments routes (list/get/delete are plain def, so asyncio.run
raised 'a coroutine was expected' and failed the Repo tests CI job).
- test_chat_attachments: align compare-chat content-part assertions with
the stable content-hash id scheme (content-part-sha256-...) instead of
the removed array-index ids; resolve ids from the listing.
- pickers: pass disabled={deleteDisabled} to the pinned-quant delete
action so a quant cannot be deleted mid model-load, matching the
expanded variant rows.
- pickers: build the pinned-quant existence set from the query-unfiltered
cached GGUF repos (format filter still applied) so a pinned quant stays
findable when the search term matches only its quant name.
* Fix Studio review regressions
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard fine-tune export content blocks
* Add Export button for archived chats
Adds an Export action to the Archived chats view in Settings > Data that
downloads only the archived chats as a JSON backup (their threads, messages
and projects). The button sits in the archived header row and appears only
when archived chats exist.
* Refactor archived export into pure, testable units
Split the archived-chats export into a dependency-free filter
(archived-chat-export.ts) and a shared JSON download helper
(download-json.ts). Skip the download when nothing is archived so a
stray call never drops an empty file. No behavior change to the button.
---------
Co-authored-by: shimmyshimmer <shimmyshimmer@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: persist llama.cpp KV cache across idle auto-unload (slot save/restore)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address KV persistence review feedback
* Studio: guard KV restore on launch config
* Studio: fix KV resume purge race, fingerprint requested ctx, purge on disable
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: re-check idle/keep-KV settings after slot save, ns file identity
* Studio: shard-aware KV guard, honor user --no-cache-prompt, early save cap
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: honor LLAMA_ARG_CACHE_PROMPT env in slot-save guard
* Studio: derive prompt-cache state from final argv for slot saves
* Studio: stat LoRA/control-vector sidecars in KV restore fingerprint
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: parse csv and FNAME:SCALE sidecar syntax in KV fingerprint
* Studio: address codex review on idle-unload KV resume
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden slot-save cleanup, cap accounting, stale-KV guard, save timeout
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: treat unavailable KV estimate as full-cap for slot-save disk check
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: GPU memory dropdown — llama.cpp --fit on and manual gpu-layers/cpu-moe
* Studio: simplify GPU memory changes (reuse ParamSlider, GPU_LAYERS_ALL, loadedGpuMemoryFields helper)
* Studio: GPU picker — choose which GPUs a GGUF model loads on (gpu_ids)
* Studio: simplify GPU picker (share /api/system fetch, validate gpu_ids)
* Studio: GPU picker review fixes (gate relative indices, no cross-model leak, validate, types)
* Studio: group GPU controls under a collapsible GPU section
* Studio: GPU feature review fixes (fix fit-ctx test, behavior-test the floor, comment accuracy)
* Studio: make GPU a top-level settings section (not nested under Model)
* Studio: flatten GPU controls into the Model section, group by GPU/context/generation
* Studio: move GPU Memory to the bottom of Model with its dependent controls beneath it
* Studio: move GPU Memory below Tensor Parallelism and GPUs below GPU Memory
* Studio: tighten GPU Memory and GPU Layers tooltip copy
* Studio: fix fit-mode context slider track-click, restore GPU Memory tooltip, shorten fit dropdown label
* Studio: GPU Memory tooltip one mode per line, briefer
* Studio: note HIP_VISIBLE_DEVICES (ROCm) in the GPUs picker tooltip
* Studio: narrow the GPU Memory dropdown to fit the shortened label
* Studio: use 'llama.cpp --fit' in the GPU Memory tooltip for consistency
* Studio: allow Tensor Parallelism in Manual GPU mode
* Studio: graduated MoE-on-CPU offload (--n-cpu-moe) replacing the all-or-nothing toggle
* Studio: size the MoE-offload slider for staged (deferred-load) models
* Studio: share one GGUF header walk for the context-length and MoE-count readers
* Studio: size the GPU Layers slider for staged models (one staged-header read)
* Studio: move Tensor Parallelism below the GPUs picker
* Studio: GPU split (--tensor-split) per-GPU model share in Manual mode
* Studio: tolerate whitespace in GPU split input, move it below GPU Layers
* Studio: rename the GPU split control to "Split ratio"
* Studio: Split ratio sends explicit even input; fix blank=free-VRAM (not even) copy
* Studio: tighten llama.cpp --fit VRAM margin with --fit-target 512
* Studio: GPU memory review fixes (rollback re-baseline, single-GPU TP gate, accurate copy)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: move Split ratio below MoE Layers on CPU
* Studio: address PR review (fix GPU-info hydration race, share fit context-length across load paths)
* Studio: address codex review (manual single-GPU TP guard, GPU-aware spec defaults in fit/manual, GGUF-only context/preference)
* Studio: address codex review round 2 (gpu_present seed, single-GPU tensor-split guard, staged manual-knob reset, strip inherited offload flags)
* Studio: address codex review round 3 (strip inherited --n-cpu-moe, CPU-fallback warning in Manual mode)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address codex review round 4 (preserve pinned fit context across a later Apply)
* Studio: address codex review round 5 (honor GPU picker for diffusion GGUFs, clear fit pin on cross-model switch)
* Studio: preserve the pending GPU Memory mode when staging a model
* Studio: pin diffusion GPU device order and reset GPU-memory state for diffusion loads
* Studio: address codex review round 6 (fit-Auto rollback context, preserve manual non-tensor split modes, persist GPU mode on load not select)
* Studio: persist the applied GPU Memory mode, not the requested one (skip diffusion loads)
* Studio: replace Manual-mode split-ratio field with per-GPU layer sliders
* Studio: clarify per-GPU layer split hint for tensor-parallel mode
* Studio: address codex review round 7 (allow GGUF gpu_ids past the legacy guard, replay GPU-memory fields on respawn)
* Studio: address codex review round 8 (size the validate preflight like the load in fit mode, across both load paths)
* Studio: skip the training-OOM guard for llama.cpp --fit GGUF loads (they spill to RAM)
* Studio: drop the now-redundant compare-path validate sizing (the --fit guard skip makes it moot)
* Studio: address codex review round 9 (keep the training guard for fit loads, forward gpu_ids to validate, strip inherited manual tensor-split)
* Studio: address codex review round 10 (gate GPU-memory adoption on is_gguf, record manual knobs only in Manual mode)
* Studio: handle diffusion GGUFs symmetrically in the GPU Memory controls (preserve the standing mode preference, hide the inapplicable mode/TP controls)
* Studio: remember the GPU Memory settings per model
* Studio: consolidate --fit mode and Manual mode into a single Manual mode
* Studio: preserve the per-GPU layer split across GPU Layers changes
* Studio: trim overly long GPU Memory comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* address GPU memory config review comments
* trim redundant GPU memory tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reconcile manual-mode TP drops with the #6659 drop-site invariants
* Preserve quantized KV in manual --fit, charge GGUF companions in full, reconcile GPU pick on load
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Clear stale GPU baseline on non-GGUF loads so it can't read as dirty
* Fix no-context-shift test for the conditional -c flag
* Credit manual GPU-layer offload for cached HF GGUFs
* Reset per-model load knobs on GGUF quant switch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Strip inherited tensor-split when manual ratio is cleared
* Match auto-load validation to safetensors placement
* Reset editable manual knobs after Auto GGUF loads
* Record a single device for diffusion GPU picks
* Reset per-model GPU knobs before applying saved settings
* Address review comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard manual tensor splits and keep remembered context on auto-load
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Snapshot compare knobs, seed splits from free VRAM, flag zero-offload loads
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Exempt CPU-only loads from the guard floor and harden compare and reseed paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reach full offload from the layers slider and charge extras drafters in the guard
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Warm the GPU device cache before pick reconciles and disable staged GPU controls
* Align the training guard with inherited extras, spec mode, and compare targets
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Hide GPUs from companion-less zero-offload loads
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Size diffusion picks per device, own manual offload flags, reject XPU picks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop tensor flags at zero layers and exempt CPU-pinned drafters
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Allowlist the zero-layer tensor parallel drop site
* Keep validate and load guards on the same extras and refresh stale baselines
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop mismatched manual tensor splits before launch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate XPU picks on the real backend field and harden split and hydration paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Weight full GPUs as zero, clamp split shares, and refine the zero-layer mask gate
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Carry fit context across mode changes and align drafter and picker gates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Catch variant switches, uncached diffusion repos, and text-only mmproj skips
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Check companions on the first device and size native and remote zero-layer loads
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Replace the training guard's precise VRAM modeling with a conservative bound
* Baseline context pins on non-GGUF hydration and reprobe list-seeded staged GGUFs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Size manual splits by their largest share and preserve resolved context from Default
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Default-deny unsized required companions and price KV at the effective cache dtype
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reserve MTP draft KV and MLA target-copy in the training guard
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Size tensor-parallel loads per device and show GPU controls for native GGUFs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reserve MTP overhead for uncached remote GGUFs and the mmproj runtime factor
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop the training-coexistence VRAM estimation this PR added
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate remembered load settings to GGUF picks
* Lock the remaining load-time controls during a staged load
* Clear the stale native-path token on compare loads
* Drop a stale guard reference from the zero-offload masking comment
* Seed GPU baselines from the rollback response and drop never-emitted offload flags
* Match validate's training guard to load and keep the native reload token
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim verbose GPU-memory comments
* Thread the variants header walk off the event loop, honor device pins on zero-offload, and hold staged GPU edits
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor manual placement and classify pinned zero-offload loads
* Close diffusion admission and status hydration gaps
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Check the actual diffusion GPU during training
* Align staged baselines and manual reload dedupe
* Fix GGUF placement and rollback state
* Harden manual GGUF placement boundaries
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove unused resolve_tensor_parallel import in llama_cpp.py
The name is used only in llama_server_args.py, routes/inference.py, and tests,
not in llama_cpp.py; the unused hoisted import trips the import-hoist verifier
in the source-lint CI job.
* Fix diffusion GPU dedup and training guard for non-numeric device tokens
The diffusion runner drives only its single lowest device and the backend
records that one device (self._gpu_ids = [sorted(gpu_ids)[0]]), but the reload
dedupe compared it against the full requested list, so a multi-GPU pick that
resolves to the same device forced a needless reload. Normalize the request the
same way for a loaded diffusion model in both _already_in_target_state and the
route _request_matches_loaded_settings.
The chat-during-training coexistence guard called int() on the single-device
token and hard-rejected when it could not parse. A non-numeric token (a CUDA
UUID / MIG handle) now sizes against the whole visible pool like the GGUF guard
instead of falsely blocking the load, and an empty token (a CPU-only runner such
as a CPU diffusion GGUF) is allowed outright since it uses no GPU VRAM.
* Tighten comments added by the GPU memory config changes
* Harden GGUF placement from independent review: VRAM sizing, diffusion TP reset, tensor_split validation
- Training coexistence guard: a single-device runner pinned through an
unresolvable UUID/MIG token was sized against the aggregate visible-VRAM pool,
so a load could pass on capacity it cannot use and then OOM active training.
Size against the worst-case visible device (min free) instead, keeping the
guard's documented default-deny contract. The empty-token (CPU-only runner)
allow path is unchanged.
- Diffusion startup: _start_diffusion_server now resets self._tensor_parallel to
False alongside the other placement resets. A prior tensor-parallel chat load
(process killed but not fully unload-reset) otherwise left /status misreporting
tensor parallelism and made an identical diffusion re-Apply reload against the
stale state.
- tensor_split: reject negative / non-finite / all-zero splits up front. They
were dropped at launch but still compared raw in the reload dedupe, so an
identical Apply reloaded indefinitely.
- Tests: the shared httpx stub was incomplete and, installed via setdefault
before real httpx loaded, broke a combined pytest run (collection errors on
httpx.Response). Import the real installed httpx instead.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothshared@gmail.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Studio: hide infra models from the hub cached inventory
The hub inventory scans behind /api/hub/cached-gguf and /api/hub/cached-models
returned the llama.cpp install validation probe (ggml-org/models) and the RAG
embedder (unsloth/bge-small-en-v1.5[-GGUF]) as on-device models. Share the
hidden-model check from routes/models.py via utils/models/hidden_models.py and
apply it in both scans. A GGUF infra repo stays visible when the user
explicitly downloaded a variant through the Hub, since variant manifests only
exist for user-initiated downloads.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make On Device trust the hub inventory, match repo ids exactly, lighten the hidden-model import
Follow-up on the hub cached-inventory hidden-model change, addressing the review.
On Device now trusts the Hub inventory API for cached rows. The backend already
hides the RAG embedder and the llama.cpp probe and re-includes a GGUF infra repo
once the user downloads a variant through the Hub, but the frontend was
re-hiding it by repo id, so the user-downloaded variant never appeared in the On
Device list or the count. isVisibleInventoryRow now short-circuits cached rows
(kind === "cache") to visible and keeps client-side needle hiding only for local
filesystem rows and Discover.
is_hidden_model matches Hub repo ids exactly (case-insensitive) against the probe
plus the effective embedder and its GGUF companion, instead of substring
matching the configured-embedder basename. A custom embedder with a generic
basename like org/model no longer hides unrelated cached repos such as
user/model-chat or org/model-instruct. The probe filename and local-path
embedders keep exact matching.
The helper moves to utils/hidden_models.py and is imported at module scope in the
hub cache scanner, so it no longer pulls in utils/models/__init__ (the eager
model-config/checkpoint stack) and a broken import fails at startup instead of
being swallowed per-repo and silently emptying the inventory. routes.models
keeps the _is_hidden_model and _safe_resolve aliases and drops the unused
_HF_REPO_ID_RE re-export that was failing source lint.
Tests: exact repo-id matching with a custom embedder, the cached-models scan
keeping an unrelated repo, and a clean-interpreter check that the helper imports
without the model-config stack.
* Studio: match the llama.cpp probe filename on both path separators
The hidden-model check compared the probe's on-disk filename with
Path(value).name, which on a POSIX interpreter does not split a Windows-style
path ("...\stories260K.gguf") and would let the probe through. Split on both
separators so the probe is matched regardless of which OS produced the path,
matching the tolerance of the previous substring check. Adds a Windows-path
assertion to the probe test.
* Studio: harden hidden infra model handling
* Fix hidden cache row confirmation
* Fix hidden local rows and confirmed hint merges
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Handle snapshot-configured hidden models
* Hide basename-only default embedders
* Fix dynamic embedder inventory filtering
* Studio: hide the configured RAG embedder from Discover and feed rows
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothshared@gmail.com>
Co-authored-by: Daniel Han <23090290+danielhanchen@users.noreply.github.com>
* Replace standalone Studio wording with Unsloth
Replace the single word Studio with Unsloth wherever it is used as
shorthand for Unsloth Studio in docs, CLI output, UI strings, i18n
locales, workflow display names, comments and docstrings.
Kept unchanged: the full name Unsloth Studio, third party product
names (LM Studio, Visual Studio, Mac Studio), feature names
(Recipe Studio, Fine-tuning Studio and its translations), and all
identifiers such as env vars, commands, paths and filenames.
* Address review feedback on the Studio wording rename
Use "an" before Unsloth where the rename left the article as "a".
Restore the split brand where Unsloth and Studio render as two halves
of the full product name: the onboarding sidebar subtitle and the
IPv6 localhost warning. Scope two messages to the full name Unsloth
Studio where plain Unsloth was misleading: the AMD README bullet and
the CLI studio setup error.
* fix(studio): add MLX adapter state control
* fix(studio): honor MLX adapter comparison state
* fix(studio): keep enabled MLX adapters permissive
* Studio: preserve public error message on MLX compare-mode adapter failures
generate_with_adapter_control raised a plain RuntimeError, which the compare
route handled with the generic handler that drops the operational message.
Raise GenStreamErrorRaised(public=chunk.public) instead and catch it in the
streaming and non-streaming consumers, matching the safetensors tool loop, so
errors like 'model is being unloaded' surface their real message.
* Studio: re-emit VLM think prefill inside the adapter context
The compare-mode merge dropped _generate_vlm's upfront yield of the prefilled
<think> block. Restore it as the first snapshot inside the lock+adapter context
(matching _generate_text) so the UI renders the thinking block during prefill
and a cancel/error before the first token does not drop it. Adds a regression
test asserting the prefill is emitted first, after entering the adapter context.
---------
Co-authored-by: danielhanchen <unslothshared@gmail.com>
* Studio: stream live tool output with SSE heartbeats and fix web page extraction
Server-side python/terminal tools now stream incremental stdout to the chat
UI while running (new tool_output SSE event), and every blocking tool
execution emits heartbeat keepalives so reverse proxies (Cloudflare tunnels
cap idle streams at ~100s) cannot drop the connection mid-turn. The tool
loop routes also emit a stall keepalive during silent prompt prefill between
tool iterations. The final role=tool message the model sees is byte-identical
to before, so tool-call parsing, nudging, and healing are untouched.
web_search page fetches now extract main content: GitHub repo root pages are
rewritten to the README API (with HTML fallback), hidden/aria-hidden client
error placeholders are dropped, conversion scopes to article/main, and known
boilerplate fragments are stripped. Non-HTML responses are returned raw
instead of being run through the HTML converter.
The frontend renders live-scrolling tool output inside running python and
terminal cards, and a chat stream that ends without a terminal signal now
surfaces an explicit interrupted state with a Retry action instead of
silently ending the turn.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix content-type sniffing, unlimited-timeout drain, and env parity in tool streaming
Content-Type sniffing: get_content_type() defaults to text/plain when the
header is absent, so the sniffing fallback never fired and header-less HTML
came back as raw markup. Report an empty type for a missing header and sniff
the body whenever the declared type is not HTML, so mislabeled text/plain
HTML pages are converted like before the extraction change.
Unlimited timeout drain: with tool_call_timeout disabled the old path used
communicate(timeout=None) and waited for EOF, but the streaming drain capped
the post-exit drain at a 5 second join, truncating output from a grandchild
that holds stdout open. When timeout is None, drain until EOF or the cancel
event fires; finite timeouts keep the bounded remaining-budget join.
Env parity: drop the PYTHONUNBUFFERED=1 injection on the streaming path so
the child invocation is byte-identical with and without streaming (the env
var was model-visible via os.getenv). Live streaming granularity now depends
on the child flushing; unflushed output arrives in ~8 KB chunks or at exit
and the final result is unchanged, with SSE heartbeats covering the gaps.
* Studio: stream tool-call arguments while the model writes them
A model writing a large tool call (a full python game is minutes of
generation) produced nothing on the stream: the structured path
accumulated delta.tool_calls fragments silently after the provisional
card, and the text path's DRAINING state consumed everything until
stream end. The user saw a dead Running spinner while the model was in
fact writing code, and the byte-silent SSE segment was also the window
where proxies drop the connection.
New tool_args SSE events stream the arguments as they generate. The
structured path forwards each fragment once a provisional card exists
(backlog first, so the card starts from the top of the call). The text
path sniffs the drained call for an enabled tool name and streams the
raw call text under the id the stream-end parser assigns its first call
(call_0), so the final tool_start reconciles the same card; the sniff is
gated on enabled names plus the provisional size floor, and prose or
ordinary JSON answers never spawn a card. The safetensors loop streams
the drained render_html call to its existing provisional card the same
way.
The chat adapter accumulates the raw stream per card and feeds a partial
JSON parse (call envelopes and stringified arguments unwrapped) into the
part's args, so the python and terminal cards render the code live and
the render_html canvas builds while streaming; both cards now say
Writing code / Writing command during this phase via useToolArgsStatus.
Display only: the parser input, the executed call, and the conversation
the model sees are byte-identical, covered by new loop-level tests for
the structured path, the text path, and the no-tool JSON answer.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep full tool output visible past the model cap; heal /mnt/data habits
Live testing surfaced two issues in the tool streaming UX.
First, a long python stdout ended in '... (truncated' in the finished
card: the model-visible result is capped by tools._truncate
(_MAX_OUTPUT_CHARS, previously 8000 chars) to protect the context
window, and the card rendered that capped text even though the live
stream had already shown everything. The cap stays (raised to 16000,
overridable via UNSLOTH_TOOL_RESULT_MAX_CHARS) but display and model
concerns are now split: the adapter preserves the accumulated live
stream on tool_end whenever it captured more than the final result, and
the finished python/terminal cards prefer it. The live-stream ceiling
rises from 16 KB to 400 KB (chunks batch per poll, so SSE stays cheap),
and both the live pane and the finished card render only the last 2000
lines with a Show all control so a huge output cannot jank the DOM. The
truncation notice now tells the model the user saw the full output and
that written files persist in the working directory. The final result
string remains byte-identical with and without streaming.
Second, models trained on ChatGPT code-interpreter transcripts write to
/mnt/data, which does not exist here (the sandbox CWD is a per-thread
persistent dir). Three layers, all identical across streaming and
non-streaming paths: the python/terminal tool descriptions gain one
sentence saying to use relative paths in the persistent CWD; a failed
execution whose output shows a missing-file error on a known
code-interpreter prefix (/mnt/data, /mnt/outputs, /home/sandbox,
/workspace) gets a model-visible retry hint appended after truncation
so it always survives; and a sitecustomize shim on the sandbox
PYTHONPATH remaps those prefixes onto the CWD in open()/os.makedirs()
with a one-line stderr notice, covering the python tool and any Python
launched from the terminal tool without touching the exec wrapper (so
tracebacks keep their line numbers). Bash-level file operations cannot
be redirected without root or mount namespaces, so they rely on the
description and the hint.
* Studio: fix hidden-element parsing, heartbeat gaps, and tool output id collisions
Review follow-ups on the tool streaming work:
- _html_to_md: treat any present hidden attribute value as hidden (it is an
enumerated attribute whose invalid value default is the Hidden state, so
hidden="false" is still hidden), and implement HTML5 optional end tags so
an unclosed <p hidden> or <li hidden> ends at the next sibling start tag
instead of swallowing every following sibling until the parent closes
- tool_stream_exec: keep heartbeats flowing after the live-output cap; a
tool that keeps printing past the cap kept the queue non-empty, so neither
tool_output nor heartbeat events were emitted and the SSE stream went
silent past proxy idle timeouts
- routes/inference: forward tool heartbeats before the
disable_parallel_tool_use drop window swallows events, so a dropped call
that executes server-side cannot leave the Anthropic stream silent
- llama_cpp: close the provisional text tool card with a tool_end when the
drained call fails to parse (DRAINING false-positive path), so the card
cannot spin forever while the text is delivered as content
- tools: decode terminal output as utf-8 with errors=replace like the python
tool; invalid bytes used to raise UnicodeDecodeError from communicate() on
the non-streaming path and silently truncate the streaming reader, so the
two paths diverged
- sitecustomize: patch io.open alongside builtins.open; pathlib Path.open,
read_text and write_text call io.open directly and bypassed the remap
- frontend: scope the toolLiveOutput/toolFullOutput store keys by pane
(modelType and pairId) and clear stale entries on tool_start; backend ids
like call_0 repeat across turns and across concurrently streaming panes
(compare mode), so a later turn or another pane could display the wrong
preserved output, and run-end cleanup now clears only its own keys
Each backend fix carries a regression test that fails on the previous code;
the byte-identity tests between streaming and non-streaming stay green.
* Studio: keep tool failure status visible and truncation/remap notices truthful
Finished python/terminal cards preferred the fuller live stream by length
alone, so a tool that printed a lot then timed out or exited non-zero showed
the captured stdout but dropped the final result's status (timeout notice,
Exit code N). preferFullToolOutput now shows the stream when the result is
just its truncated prefix, and appends the result otherwise so the failure
tail always survives and the copy button copies both.
The result truncation notice claimed the user was shown the full output, but
the same wrapper serves non-streaming chat/API and direct execute_tool()
callers where nothing is streamed to anyone. The notice is now mode-neutral
and stays byte-identical with and without an output_callback, keeping the
streaming vs non-streaming invariant intact.
The sandbox sitecustomize shim now remaps /tmp/outputs into the working
directory only while it does not already exist, so a real /tmp/outputs the
user's own code created is never shadowed; /tmp/outputs also joins the
missing-path retry-hint list.
* Studio: suppress hidden void elements and keep live output scroll pinned only when at bottom
* Studio: drop capped tool output without concatenating; remap pathlib mkdir
Past the live-output cap stream_tool_execution built item + _drain_pending()
(the current chunk joined with every queued sibling) only to discard it in the
capped branch, so a chatty tool (yes, a tight print loop) could enqueue far
more than one poll interval of text and blow past the memory/CPU ceiling the
cap exists to enforce. Drain and drop queued items without building a combined
string, still counting each drain toward the heartbeat cadence so the SSE
keepalive survives.
Generated code often prepares code-interpreter paths with
Path('/mnt/data').mkdir(parents=True, exist_ok=True); pathlib drives that
through os.mkdir (not the patched os.makedirs) per component and, on
FileExistsError, probes the unpatched os.stat via Path.is_dir(), so the setup
raised before open() ever ran. Patch os.mkdir with the same remap and patch
Path.mkdir so the whole parents/exist_ok dance lands on the mapped working
directory and stays idempotent; real paths still pass through.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: generalize sandbox write remap and hint to any hallucinated absolute path
Models invent absolute paths from seeing their CWD (a DeepSeek run did
open('/home/ubuntu/Sandbox/flappy_bird.html', 'w') and died with
FileNotFoundError). A prefix list cannot enumerate these, so the sitecustomize
shim gains a write-mode fallback in open()/io.open(): when a write/create-mode
open targets an absolute path outside the CWD whose parent directory does not
exist, redirect it to the basename in the CWD and emit the same one-line stderr
notice, echoing the original path. The prefix remaps still run first (they cover
reads and preserve subpaths); read modes never hit the fallback so real system
files fail or succeed truthfully; bytes paths pass through. The fallback is not
applied to mkdir/makedirs/Path.mkdir, since creating an arbitrary absolute
directory can legitimately succeed on the host, and that decision is documented
in a comment.
The model-visible retry hint now echoes the real failing path (parsed from the
traceback tail) instead of the canned /mnt/data example, and fires for any
absolute path outside the working directory, not just the enumerated prefixes,
while a relative miss still gets no hint.
The shim wrapper still adds one frame to tracebacks that surface open() errors;
suppressing only our frame has no clean standard mechanism (a wrapper always
adds a frame), so the frame is left as an accepted compromise.
Tests: hallucinated absolute write remaps to the CWD basename across w/a/x/w+;
reads of a missing absolute path pass through untouched; writes to an existing
external dir pass through; prefix subpaths still preserved; end-to-end write
fallback lands the file in the sandbox workdir identically with and without
streaming; the hint echoes the actual path for convention and non-convention
absolute paths alike.
* Studio: kill exited process groups on drain; bound the over-cap output batch
_drain_process_output killed the process only via _kill_process_tree, which
short-circuits once the parent has exited, so a grandchild that inherited
stdout and outlived the parent was never signaled: a finite-timeout run could
return while it kept holding the pipe, and a timeout=None cancel left it
behind. Capture the setsid process group before waiting and SIGKILL that group
at both give-up points so the whole tree is torn down.
The streaming wrapper's first over-cap batch joined the current chunk with the
entire pending backlog before enforcing the live-output cap, so a chatty tool
could allocate far past the cap on the crossing batch. Bound the drain to the
remaining budget and drop the surplus in place, keeping the truncated output
byte-identical to joining everything.
* Studio: harden sandbox path healing and process/generator cleanup
Sandbox sitecustomize shim:
- Make the generalized write fallback collision-safe: never redirect an
invented absolute path onto an already-present CWD file (refuse and let the
original open raise FileNotFoundError, preserving the workspace file).
- Only w/a/x create a file; r+/rb+ are read-update modes that require the
target to exist, so a bare + no longer trips the write fallback.
- Gate every convention-prefix remap (/mnt/data, /mnt/outputs, /home/sandbox,
/workspace) on the prefix root being absent, so a real host mount is never
shadowed; a miss under an existing real prefix passes through.
- Patch os.open so Path.touch and other low-level creators heal convention
paths too, matching the Path.mkdir patch.
Local code execution (tools.py):
- Capture the setsid process group right after Popen (before any watcher can
poll/reap the leader) and thread it through the cancel watcher and drain.
- Kill the captured group in the non-streaming python/terminal timeout branch
so an exited leader no longer leaks a stdout-holding grandchild (matches the
streaming drain path).
- Guard os.getpgid/os.killpg by platform so streamed execution no longer
raises on Windows; fall back to single-pid kill.
- Judge missing-path hints against the executor's real workdir so a legitimate
miss inside a project workspace outside the sandbox root is not mislabeled.
Tool streaming routes (routes/inference.py):
- Drain a pending next(gen) worker before closing the generator in the
safetensors and Anthropic tool streams, so a disconnect no longer races
gen.close() (generator already executing) or leaks the thread/generator.
HTML to markdown:
- Only drop boilerplate lines composed entirely of known furniture phrases so
real prose that merely quotes one (for example "we use cookies to
authenticate requests") is preserved.
Adds hermetic tests for each change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep aside callouts, contain sandbox path remaps, and keepalive dropped Anthropic tool events
_html_to_md: stop dropping <aside> unconditionally. Documentation pages
render notes/warnings/examples as aside admonition callouts; those inside
the selected article/main scope are real content. A furniture aside outside
the scope is already excluded by the main-content pass.
sitecustomize: contain the code-interpreter path remap under the sandbox
CWD. A hallucinated habit path such as /mnt/data/../other_session/file no
longer escapes the per-conversation workdir; parent-traversal components in
the suffix are dropped and a '.'/'..' write-fallback basename is refused.
routes/inference: emit a rate-limited comment keepalive when the Anthropic
Messages stream drops tool_output/tool_args events. A chatty tool keeps the
generator busy so the stall keepalive never fires and the tool wrapper emits
heartbeats only while idle, which left the SSE stream silent past proxy idle
caps; the OpenAI passthrough paths forward these events, this path now keeps
the connection alive.
* Studio: bound the tool-output chunk that first crosses the live cap
_drain_queue joined the entire chunk that first crossed the live-output
cap before dropping the rest, so a single multi-megabyte line (or any
chunk dequeued once the budget was already met at max_chars <= 0) was
materialized in full only to be truncated away, defeating the memory
ceiling the cap enforces. Slice the crossing chunk to one character past
the budget: that preserves the caller's overflow signal and its
byte-identical truncation while dropping the arbitrarily large remainder
in place.
* Studio: scope missing-path hint to the failing line, keepalive dropped-call output, and preserve truncated tool streams over byte length
- tools._missing_path_hint: the code-interpreter convention-prefix trigger
scanned the whole output, so a convention prefix mentioned only in a
traceback frame (a /workspace project root) or printed by the user's code
would add a misleading 'use a relative path' hint even when the actual
FileNotFoundError was a relative or in-workdir path. Scope the convention
test to the failing-path error line(s), matching _extract_missing_abs_path.
- _anthropic_tool_stream: the tool_output/tool_args rate-limited keepalive sat
after the drop_until_tool_end skip, so under disable_parallel_tool_use a
chatty second-or-later tool call was dropped whole with no keepalive, letting
an idle proxy kill the SSE stream. Check the keepalive branch before the drop
skip (like the heartbeat branch) so dropped-call output keeps the stream alive.
- preferFullToolOutput / chat-adapter: a truncated result can be longer than
the live stream by byte count once its footer, an 'Exit code N:' notice, or an
__IMAGES__ base64 tail is appended, so the length-only gate discarded the full
stream and the finished card fell back to the truncated text. Add a shared
truncation-aware shouldPreserveFullOutput used by both the write and read
sites: preserve the stream whenever the result carries the truncation footer.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: skip the habit-path hint for real project paths under a convention prefix
* Kill captured process group on streamed wait-timeout
The streamed drain path's proc.wait() timeout branch only called
_kill_process_tree(proc). If the leader exits in the narrow window between
the wait timing out and _kill_process_tree sampling its pgid, that helper
short-circuits on the reaped leader and a stdout-holding grandchild in the
same group survives. Also kill the captured pgid there, matching the
non-streaming communicate() timeout path. Adds a hermetic regression test
that models the reaped-leader race by stubbing _kill_process_tree.
* Fix 3.10 pathlib write_text remap and honor cancel in finite drain
On Python < 3.11 pathlib routes Path.open / read_text / write_text through
a module-level accessor singleton whose open attribute captured the original
io.open at import time (_NormalAccessor.open = io.open). Patching io.open in
the sandbox shim therefore never reached that captured reference, so a
Path('/mnt/data/x').write_text(...) raised FileNotFoundError on 3.10 while
passing on 3.11+ (which dropped the accessor and calls io.open at call time).
Repoint _NormalAccessor.open at the same io.open wrapper via a staticmethod,
guarded so it is an idempotent no-op on 3.11+. Keep the test save/restore
helpers symmetric so the accessor is restored too, and add a hermetic
write_text/read_text remap test that covers every version.
Also honor cancellation while draining inherited stdout after the leader
exits. Once the leader is reaped the cancel watcher returns (its loop is
while proc.poll() is None), so the finite-timeout drain did one blocking
reader.join(timeout=remaining) that ignored cancel_event and kept draining a
chatty grandchild for the whole budget after a disconnect/Stop. Poll
cancel_event in 0.5s slices against a deadline like the timeout=None branch
and kill the captured process group promptly on cancel. The normal path still
reaches EOF on its own, so the streamed vs non-streamed result is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: port no-tool stream keepalive/drain and fix subprocess/queue/extraction asymmetries
Streaming no-tool paths now match their tool twins:
- _anthropic_plain_stream, safetensors/MLX no-tool stream, and standard GGUF
no-tool stream run next(gen) in a worker with a timed SSE keepalive loop so a
long prompt prefill cannot leave the stream idle past a proxy cap.
- The Anthropic plain and safetensors/MLX no-tool teardowns now drain the
pending next(gen) worker and close the generator on disconnect instead of
leaking the suspended generator.
Other asymmetries:
- Non-streaming _python_exec/_bash_exec always drain via _drain_process_output
(output_callback may be None) so a cancelled run reaps a stdout-holding
grandchild that outlived the leader instead of blocking in communicate(). The
joined bytes are identical to communicate(), so streamed vs non-streamed
results stay byte-identical.
- _build_bypass_env installs the sitecustomize path shim on PYTHONPATH (prepend,
keeping the operator's entries) so /mnt/data remap works in bypass mode too.
- GGUF forwards output_callback to execute_tool only when the callable accepts
it (shared accepts_output_callback), matching safetensors and preserving
legacy monkey-patched signatures.
- tool_stream_exec bounds accepted live output at the producer boundary so a
chatty tool cannot grow the queue without limit under consumer backpressure
and cannot keep the drain spinning and starve heartbeats.
- html_to_md implicit-close now searches past unclosed inline descendants so a
hidden <p>/<li> is closed by a following block; main-content scoping gates on
the largest single <article>/<main> so a swarm of tiny cards cannot pass the
threshold in aggregate and displace the real main.
- preferFullToolOutput re-attaches the "Exit code N:" prefix to the fuller
stream instead of appending the still-prefixed result, so a failed truncated
tool no longer duplicates its stdout in the finished card.
Adds hermetic tests for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Preserve short live output on timed-out tools; strip inline-CSS-hidden subtrees and score truncated main-content scopes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten chat tool streaming comments and docstrings
* Keep HTML READMEs from the GitHub API and preserve interrupted tool output
Convert a 200 HTML README body from the GitHub README API to Markdown
instead of discarding it and falling back to the repo page chrome, and
promote captured live stdout to full output when a tool never reaches
tool_end (stream interrupted or cancelled) so the partial diagnostics
stay on the finished card.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: anchor HTML sniff, keep repeated sandbox writes, reuse textual tool ids
Anchor _looks_like_html to the leading doctype/tag so a Markdown README that
opens with a fenced HTML example stays Markdown (no html_to_markdown
corruption), while bare HTML fragments (<body>/<article>/<section>) are still
detected and converted on a missing/wrong Content-Type.
Let the sandbox write fallback re-serve a target it already healed for the same
invented absolute path, so iterative overwrites of a generated artifact stop
failing with FileNotFoundError while the anti-clobber guard still refuses
unrelated same-basename files.
Reconcile the first textual tool call carrying an explicit id onto the open
provisional TEXT card instead of spawning a duplicate card under that id.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: run implicit-close before skipping tags and keep leading README tables as Markdown
A skipped block (<nav>/<footer>) is an HTML5 optional-end-tag closer of an
open <p>, but handle_starttag returned before the implicit-close bookkeeping,
so a never-closed <p hidden> kept its hidden mark and swallowed every following
sibling. Run _close_implicit before the skip decision so the hidden mark is
released and trailing content renders.
Drop <table> (and its <thead>/<tbody>/<tr>/<td>/<th> children) from the
_looks_like_html leading set: Markdown READMEs routinely open with a raw HTML
<table> badge/layout row, and sniffing that as HTML collapsed the whole
Markdown body through html_to_markdown, exactly like the already-excluded
<div align>/<p align> layout headers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make bypass-permissions Popen double faithful to the unified drain path
The non-streaming _python_exec/_bash_exec now share _drain_process_output,
which reads proc.stdout in a reader thread and calls proc.wait(); the test
double only implemented communicate(), so bypass-mode bash returned an
AttributeError instead of the faked output. Give _FakeProc a readable stdout
pipe (yields the fake line then EOF), wait()/poll()/pid, so the test exercises
the real drain path on both the python and bash bypass branches.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: persist sandbox path heals across runs and suppress nested hidden lists
* Studio: run tool Python child unbuffered (-u) so unflushed prints stream live
A long-running snippet doing bare print() without flush=True never reached
the live-output pane: CPython block-buffers stdout when writing to a pipe, so
_drain_process_output's readline() saw nothing until the buffer filled or the
process exited. Launch the child with the interpreter -u flag so stdout is
unbuffered and each print streams as it is produced.
-u is applied unconditionally on both the streaming and non-streaming path, so
the child invocation stays byte-identical with and without streaming and the
final joined result is unchanged (buffering/timing only). Unlike the earlier
PYTHONUNBUFFERED=1 env injection that was removed, -u does not pollute the
child's os.environ and is not visible via os.getenv.
* Render only the selected main-content subtree in html_to_markdown
The main-content heuristic sized each <article>/<main> candidate
individually to pick the largest subtree, but then rendered every
matching tag in the document. A page with one real article plus
sibling related-post cards or comment threads passed the size gate on
the real article yet still emitted the unrelated siblings.
Size and render the same chosen subtree so only the selected
main-content subtree reaches the output.
* Studio: tighten chat-tool-streaming fix comments
* Studio: store tool-output-scope separators as unicode escapes
The pane-scope and tool-output-key separators were literal NUL (0x00) bytes, which made git treat the file as binary and hide its diff and blame. Write them as \u0000 escapes instead; the runtime key value is unchanged.
* Studio: bound tool-stream teardown when the client disconnects
stream_tool_execution ran its yield loop with no try/finally, so a gen.close() on client disconnect (GeneratorExit at a yield) skipped the worker join and never signalled cancellation. A tool that does not poll cancel_event mid-flight (web_search, MCP, search_knowledge_base) then kept request teardown blocked until the tool's own timeout. Thread the request cancel_event into the wrapper, set it only on the abnormal-exit path so a clean multi-tool turn is unaffected, and bound the worker join to a few seconds; the daemon worker cannot outlive the process.
* Studio: sandbox path remap no longer masks missing reads
The sandbox sitecustomize shim remapped code-interpreter prefixes (/mnt/data, /workspace, ...) onto the working directory for every open mode, including reads. A read of a path that truly did not exist was silently redirected onto a same-basename workdir file instead of raising on the path the model used, hiding real missing-input errors. Remap writes and creates as before, but remap a read only when the mapped workdir target already exists (re-reading a just-written artifact); otherwise keep the original absolute path so the failure stays truthful.
* Studio: bound web fetch with one overall deadline and cancellation
The web fetch applied timeouts per network operation, so a GitHub README API attempt plus its HTML fallback plus up to five redirect hops could run well past the tool timeout, and nothing aborted once the client had disconnected. Add a single wall-clock deadline shared across the API attempt, the fallback, every redirect hop and the body read, cap each hop's socket timeout at the time left on the budget, and poll cancel_event. SSRF host pinning, per-hop redirect revalidation, the five-hop cap and the size cap are unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep tool-stream teardown off the event loop on disconnect
The bounded worker join added for disconnect safety still ran on an abnormal close, so a client disconnect could wait the full join timeout; and the safetensors and Anthropic tool streams closed their generator synchronously on the event loop, unlike the GGUF path. On abnormal exit the daemon worker is abandoned, so join with a zero timeout instead of waiting; offload the safetensors and Anthropic gen.close to a thread to match GGUF; and surface a heartbeat as soon as cancel_event is set while the worker is silent so the route regains control at once instead of after a heartbeat interval.
* Studio: extend the web-fetch deadline to DNS, the body read, and search
The overall fetch deadline did not cover host resolution or the response body read, and query-mode web_search ignored cancellation. Resolve hosts (initial and every redirect) on a budget-polled helper so a slow or pre-cancelled getaddrinfo aborts on time; read the capped body in chunks with the budget re-checked between them so a slow-drip server cannot stretch a single read past the deadline; and gate the blocking DDGS query on cancel_event on both sides. SSRF host pinning, per-hop redirect revalidation, the five-hop cap and the size cap are unchanged.
* Studio: defer the sandbox remap notice and tighten os.open create flags
The one-shot remap notice fired while computing the mapping, so a read that kept its original path emitted a false notice and spent the notice a later genuine remap needed. Only emit it once _remap_open commits to the redirect. Separately, os.open classified O_TRUNC / O_APPEND without O_CREAT as creating, but those cannot create a missing file, so a missing target now stays truthful (only O_CREAT maps to the creating mode).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: convert only genuine HTML README bodies, not Markdown with a leading block tag
The GitHub README API returns the raw file, almost always Markdown. _looks_like_html classified a Markdown README opening with a block tag (<ul>, <ol>, <dl>, <pre>, <blockquote>) as HTML, so _fetch_page_text ran it through html_to_markdown and collapsed its headings, lists and fenced code into a single line. Sniff the README body with a stricter document-level check (doctype or a leading <html>/<head>/<body>) so only a real .html README is converted; the general page path is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Surface unclassified mid-stream Anthropic errors as SSE error events
The local Anthropic tool-stream and plain-stream paths called
_anthropic_stream_error_event(e) with force defaulting to False, so an
unclassified mid-stream failure (llama-server crash, decode OOM, a
dropped upstream socket) returned no event. The except block then fell
through to emitter.finish(), emitting a normal message_delta and
message_stop that masked a truncated turn as a clean finish.
Pass force = True at both fall-through sites so an unclassified failure
emits a 500 SSE error event and returns, matching the Anthropic
passthrough path that already forces it. Add regression tests covering
both stream paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: give each tool run a unique part id so finished cards keep their own output
Backend tool ids restart at call_0 every assistant response, and the
transient toolLiveOutput/toolFullOutput store maps were keyed by pane
scope plus that bare backend id. Two turns in the same pane therefore
shared one key: the stale-clear at tool_start only guards the forward
direction, so when a later call_0 finished and wrote its preserved full
output, every earlier still-mounted finished card reading the same key
re-rendered and displayed the newer tool's output instead of its own.
Mint one per-run-unique part id per backend id (call_0:<uuid>) and route
tool_start/output/args/end through a single resolver so all events for a
call resolve the same id. The durable part carries the unique id, so the
finished-card readers derive a collision-free key with no change, and the
awaiting-confirmation path keeps its own synthesized id. Outbound replay
stays paired (the assistant tool_call id and the role=tool result
tool_call_id both come from the part id) and gains unique ids across
turns, which strict providers require.
* Studio: tighten PR comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* Studio: exclude /api/export/status from request access logs
The frontend polls /api/export/status every 5s to detect export start, so it
fires continuously even when idle. Each poll emitted an info request_completed
access line, making up most of the server access logs. Add it to _EXCLUDED_PATHS
alongside /api/train/status. The endpoint is unchanged; export state is still
logged by the export modules and streamed over SSE, so no signal is lost.
* Studio: collapse hub download-progress polls in the access log
download-status and gguf-download-progress (plus the dataset equivalents)
are polled about twice a second for the whole download, so each emitted an
info request_completed line. Add them to _QUIET_POLL_PATHS so they collapse
to one heartbeat line per 10s instead of one per poll.
* Studio: log hub download progress at 10% steps
The access log carried no real progress, only poll pings. Emit one
hub_download_progress line per 10% step from the shared snapshot progress
reader, so an active download shows actual percentage without a line per
poll. Throttled per job and resynced if the same download restarts.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: drop successful chat thread/project CRUD from the access log
A single chat turn fans out about twenty requests under /api/chat/threads
and /api/chat/projects (list, fetch, per-message forks, and the message
writes) that only reflect the UI re-rendering. Suppress their 2xx access
line so the log keeps the signal (generation, tool calls, code execution,
engine stats) and errors. Non-2xx on these paths still log.
* Studio: silence transformers torch_dtype deprecation warning
transformers logs "`torch_dtype` is deprecated! Use `dtype` instead!" once at
model-config load via logger.warning_once (logging, not warnings), so a warnings
filter cannot catch it. Attach a small logging.Filter in setup_logging, which
runs before any model config is parsed, to drop that record on the transformers
loggers that emit it.
* Studio: quiet inference load-progress polls and log throttled load progress
The frontend polls /api/inference/load-progress about twice a second for the
whole model load, so each emitted a request_completed line. Add it to
_QUIET_POLL_PATHS (heartbeat) and emit one inference_load_progress line per 10%
step from the load-progress route, so a load shows real percentage instead of a
line per poll.
* Studio: fully suppress download/load progress poll access lines
The download-status, download-progress, gguf-download-progress, active-downloads
and transport-status polls (model and dataset), plus inference load-progress,
fire ~2x/s for the whole download or load. Their progress is now reported by the
hub_download_progress / inference_load_progress events (and the viewer's progress
line), so the per-poll access line adds nothing. Drop it on 2xx and keep it on
errors, instead of the prior 10s heartbeat. Chat CRUD suppression is folded into
the same _is_quiet_success helper.
* Studio: suppress training-tab model/dataset download-progress polls
The training tab polls /api/models/download-progress and
/api/datasets/download-progress about twice a second for the whole prep phase.
These are separate routes from the /api/hub equivalents and only scan the cache,
so their 2xx access line adds nothing (on Windows they always read 0 since the
bytes live in snapshots/, not blobs/). Suppress the 2xx line and keep errors,
alongside /api/models/gguf-download-progress.
* Studio: drop transient pre-auth 401 on chat thread/project polls
On first load the SPA fires chat thread/project GETs before the initial token
refresh, so they 401 until /api/auth/refresh runs and the retries succeed. That
pre-auth 401 is a bootstrap artifact, not an error; suppress it alongside the
already-quiet 2xx line. Genuine 4xx/5xx on these paths, the download/load poll
401s, and all /api/auth/* still log.
* Studio: quiet tab-switch list polls and per-poll scan/reconnect logs
Switching between the Train, Export, and Chat tabs refetches list endpoints on a
timer, and each hit re-logs internal detail. Heartbeat /api/train/runs,
/api/models/checkpoints, /api/models/local and /api/rag/knowledge-bases (10s
window, first hit and errors still log), and downgrade two per-poll INFO lines to
debug: the checkpoints scan summary ("Found N training runs") and the
per-reconnect SSE resume line. The meaningful "replayed N missed steps" line,
logged only when steps were actually replayed, stays at info.
* Studio: enable tokenizer parallelism for dataset prep on Windows/macOS
TOKENIZERS_PARALLELISM was forced off everywhere to stop datasets' forked map()
workers from deadlocking, but that fork only happens on Linux. On spawn platforms
(Windows/macOS) dataset.map() runs in-process (dataset_map_num_proc returns None),
so disabling tokenizer parallelism leaves the fast tokenizer single-threaded and
dataset prep runs serially on one core. Keep it off on Linux (fork safety) and on
for spawn platforms, where there is no fork to deadlock. Measured ~7x faster
tokenization (12.5s -> 1.7s for 20k rows on a 32-core Windows box).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: log throttled training status to the server log
Training step/loss/epoch only went to the UI via SSE, so the server log showed
inference engine_stats and train/runs heartbeats but nothing about the actual
run. Emit one throttled training_progress line (step/total, percent, loss, epoch,
eta) from the CUDA event pump: the first step, then at most every 30s, plus the
final step, resyncing when a new run restarts the counter. Per-step UI streaming
is unchanged.
* Studio: quiet llama.cpp update-status polls and log throttled update progress
The prebuilt llama.cpp update polls /api/llama/update-status about twice a second
for the whole download and install. Suppress its 2xx access line (errors still
log) and emit one throttled llama_update_progress line per 10% step from the
status route, so the update shows progress without a line per poll. The existing
"llama update: installing" and "llama update: success" events still bracket it.
* Studio: quiet the export log-tail poll
The Export tab polls /api/export/logs about once a second to stream the export
subprocess output into the UI panel. Suppress its 2xx access line; the real
progress is already logged as event-driven "Export subprocess status: <phase>"
lines plus the subprocess start and checkpoint-loaded events, and errors still log.
* studio: keep errors and mutations visible in access-log suppression
Make the quiet-success access-log suppression GET-only so chat thread/project
mutations (POST/PUT/DELETE) still log; only their list-poll 2xx and the
transient pre-auth 401 are dropped.
Suppress /api/export/status 2xx only (move it out of the all-status exclude
set) so a 401/403/500 on it stays visible.
Legacy /api/models and /api/datasets download-progress polls emit no
hub_download_progress events, so heartbeat them via the 10s quiet-poll window
instead of suppressing outright, keeping download visibility (notably on
Linux). The event-emitting /api/hub download polls stay fully suppressed.
Update and extend the middleware tests to cover GET-only suppression, the
export-status error path, and the legacy download heartbeat.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten access-log and training-progress comments
Comment-only pass: collapse the multi-line explanations in the logging
middleware and the throttled training-progress logger to fewer lines while
keeping the rationale. No behavior change.
* studio: log structured export_progress phases
Emit a structured export_progress event per phase (consolidated in the server
log like training and download progress) instead of a plain status string, and
add a phase milestone at the start of the heavy export step so the
merge/save/convert is visible in the server log, not only in the forwarded
stdout panel.
* Studio: reset training-progress log throttle on each new run
start_training rebuilds the per-run progress state but left _last_progress_log_ts/_last_progress_log_step at their prior values. A run started within 30s of a previous one whose last logged step matched the new run's first step would hit the step == prev short-circuit and drop the promised first training_progress line, then stay suppressed until the old 30s window expired. Reset both fields when a new job is accepted.
* Studio: keep post-bootstrap chat 401s visible in the access log
The chat thread/project 401 suppression dropped every GET 401 on those prefixes, so a genuine expired-session 401 vanished alongside the transient pre-auth race. Gate the 401 drop on a per-middleware bootstrap latch that flips once /api/auth/refresh first succeeds: before that the 401s are the pre-refresh race and are suppressed; after it any chat 401 is a real failure and logs. Add a test for the post-refresh case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: limit chat access-log suppression to the exact list polls
The chat thread/project suppression matched by startswith, so it also dropped the 2xx access line for detail and message reads (/threads/{id}, /threads/{id}/messages, /threads/{id}/messages/{id}, /projects/{id}) that are not the high-frequency list polls, losing their access and latency logging. Match the two list paths exactly instead, so only the intended list polls (and their pre-auth 401 race) are suppressed while detail and message reads keep their access line. Add a regression test.
* Studio: reset inference load-progress throttle for each load
The load-progress throttle (_last_load_progress_step) is a module global that persisted across loads, so a cached or small load whose first sampled /api/inference/load-progress response already reported fraction=1.0 hit step == prev (10) from a prior completed load and emitted no inference_load_progress line, while that endpoint's access log is suppressed, leaving the new load with no progress signal. Arm the throttle at load initiation in _load_model_impl so each load's first step always logs. Add a regression test.
* Studio: tighten logging comments
Collapse a few verbose comments (tokenizer-parallelism note, torch_dtype filter, legacy download-poll heartbeat, chat list-path suppression) to fewer lines without changing intent or code.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The edge fades toggle in Settings > Appearance let users swap the panel
edge gradients for thin divider lines. It added little on top of the
default look, so this removes the setting and all of its wiring while
leaving the default edge fades in place.
- drop the edgeFades field, default, and no-edge-fades class from the
appearance customization store
- remove the settings row, switch, and search entry
- drop the html.no-edge-fades rules from index.css and hub.css
- remove the edgeFades label and description from all locales
- drop the edgeFades field from the personalization backend model and
its test references
* Studio: make the Cloudflare tunnel opt-in (off by default)
A wildcard bind (`-H 0.0.0.0`) auto-started a public trycloudflare.com
tunnel, so exposing Studio on the LAN also published it to the public
internet. Flip the default so the tunnel is opt-in.
- `--cloudflare` is now tri-state (Optional[bool], default None = off),
mirroring the existing --enable-tools/--disable-tools handling. Pass
--cloudflare to expose a public HTTPS link for a wildcard bind; --secure
still implies the tunnel.
- --secure + --no-cloudflare is still rejected as a contradiction.
- Update the parent-command guard, re-exec forwarding, startup-banner
wording, the colab comment, README, and tests.
* Studio: update installer/setup launch hints for opt-in Cloudflare
The post-install launch hints only mentioned --secure for a public link.
Now that the tunnel is opt-in, clarify that -H 0.0.0.0 exposes the raw
port on the LAN (not a public URL), and surface --cloudflare as the
explicit opt-in for a public HTTPS link (--secure keeps the raw port
private). Applied to install.ps1, install.sh, and studio/setup.sh.
* Studio: address review - keep cloudflare tri-state + harden run re-exec
Two review points from the bots:
- Gemini: keep `cloudflare` as Optional[bool] in run_server instead of
casting None -> False, so the startup banner can distinguish "OFF (default)"
(unset) from "OFF (--no-cloudflare)" (explicit). `_cloudflare_flag` and the
banner branch now carry the tri-state.
- Codex (P1): `unsloth studio run` re-execs the studio venv's console script,
which can be an older build whose --cloudflare defaulted on; omitting the
flag let it re-enable the tunnel. That path now forwards the default polarity
explicitly (--no-cloudflare, or nothing under --secure since --secure implies
the tunnel). The plain `unsloth studio` path runs the same-version in-tree
run.py (resolved via _find_run_py), so it keeps forwarding only an explicit
polarity and still shows the accurate "(default)" banner.
Tests updated for the tri-state banner labels, the None gate cases, and the
new re-exec forwarding.
* Studio: forward --no-cloudflare on plain re-exec too (mixed install)
Codex follow-up: _find_run_py falls back to STUDIO_HOME/.../studio/backend/
run.py when the package copy is absent, so the plain `unsloth studio` re-exec
can land on an older run.py whose --cloudflare defaults on. Forward the default
polarity explicitly there too (--no-cloudflare, or nothing under --secure),
matching the run subcommand. The common in-venv launch skips the re-exec and
still shows the tri-state "(default)" banner.
* Studio: fix launch hint - --cloudflare needs the wildcard bind
Codex P3: the launch hint listed --cloudflare next to the loopback
`unsloth studio -p 8888` command, but the tunnel only starts for wildcard
binds, so `--cloudflare` alone on 127.0.0.1 does nothing. Show
`-H 0.0.0.0 --cloudflare` in the hints (install.ps1, install.sh,
studio/setup.sh) and clarify the same in the README.
* Studio: cross-platform masked terminal password prompt helper
Per-keystroke '*' echo (POSIX termios cbreak / Windows msvcrt.getwch),
backspace editing, Ctrl-C abort, EOF handling, confirmation loop with
re-prompt on mismatch or policy failure. Pure should_prompt gate for the
--secure/--cloudflare exposure paths.
* Studio CLI: force a terminal password change before public tunnel exposure
When a launch will start the Cloudflare tunnel (--secure, or --cloudflare on
a non-api-only wildcard bind) and the admin account still has its seeded
bootstrap password, prompt for a new password in the terminal (masked with
'*', confirmed, re-prompting until valid) before any re-exec or server
exists. The change is committed in the parent so it never crosses argv or
the environment and older studio-venv children see it immediately. Without
a terminal, warn and fall back to the backend bootstrap shutdown timer.
Mirrors backend update_password semantics in one transaction: rehash,
rotate the JWT secret, clear must_change_password, revoke refresh tokens,
drop the desktop secret, then remove the stale credential files.
* Studio: terminal password gate before the public tunnel (backend backstop)
Never publish a trycloudflare URL while the seeded admin password is
active: run_server now runs a terminal password-change gate after the
tunnel decision and strictly before start_studio_tunnel. Interactive
refusal fails closed (shutdown + exit 1, mirroring the secure gate);
without a tty it warns and keeps the bootstrap deadline. Success applies
the same effects as the change-password route (update_password +
revoke_user_refresh_tokens) and drops the stale
app.state.bootstrap_password. MIN_PASSWORD_LENGTH centralised in
auth/storage.py and referenced by the HTTP schema. terminal_prompt.py
carries the pure gate helper (interactive loop stubbed; supplied by the
masked-input module). Also migrates the studio/setup.ps1 launch footer
that still showed the bare wildcard hint.
* README: reconcile remote-access section with opt-in Cloudflare tunnel
* Studio: harden the terminal password gate after review
- run.py: run the gate BEFORE the uvicorn socket binds. On a wildcard
--cloudflare launch the served HTML injects the bootstrap credential
for first login, so a pre-gate listener would hand the default
password to anyone who reaches the raw port while the operator is
still typing. The gate now also seeds the admin row itself (it can
run before lifespan startup).
- Headless launches that nothing would protect now fail closed: the
bootstrap deadline never arms for api-only serving and
UNSLOTH_STUDIO_BOOTSTRAP_TIMEOUT=0 disables it, so warn-and-proceed
would have promised a shutdown that never comes. Both the CLI and the
backend refuse to publish in that case; the ordinary headless path
still warns and relies on the 1h deadline, and no longer auto-fills
the default credential into HTML served on a public URL.
- storage.update_password gains revoke_refresh_tokens to delete the
user's refresh tokens in the SAME transaction as the password commit;
the change-password route and the backend gate use it (a separable
follow-up delete could fail after the commit and leave a stale
refresh token able to mint access tokens under the rotated secret).
- clear_bootstrap_password is best-effort: a locked/undeletable file
must not surface as a failed password change.
- CLI masked reader: disable ISIG like the backend so Ctrl-Z cannot
suspend the process with the shared terminal stuck in no-echo mode;
handle Ctrl-C/Ctrl-Z as characters; treat stream EOF mid-line as an
abort instead of submitting a partial password. Both readers restore
terminal attrs from a SIGTERM/SIGHUP handler since a finally block
cannot run when a default-disposition signal terminates the process.
- Backend reader: decode byte-at-a-time through an incremental UTF-8
decoder so multi-byte characters split across read boundaries are no
longer dropped; isatty checks tolerate closed/None streams.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: persist bootstrap suppression through lifespan startup
The pre-bind password gate nulled app.state.bootstrap_password, but the
FastAPI lifespan runs after it and re-reads the bootstrap password into
app.state on both admin paths, so a headless public launch could still
serve the injected credential in HTML. Carry a persistent
suppress_bootstrap_injection flag that the lifespan honors instead.
Also drop the quoted Tuple annotation on _terminal_password_gate that
tripped the import-hoist lint (the typing import looked unused).
* Studio CLI: keep the pre-exec auth DB private (0700 dir, 0600 db)
On a fresh install the pre-exposure password gate creates auth/ and
auth.db through the CLI before the backend ever runs, and
sqlite3.connect leaves the DB 0644 under a 022 umask. Mirror backend
storage.get_connection's chmod so the committed password hash and JWT
secret are never world-readable, even if the launch aborts before the
backend applies its own modes.
* Tighten pre-exposure password gate comments
* Studio: delete seeded bootstrap password before headless public re-exec
The headless warn-and-proceed path returns with the default admin
password still active, then re-execs a child Studio process. An old
studio-venv child (mixed-version install) predates the pre-bind gate and
its injection-suppress flag, so its lifespan reads .bootstrap_password
and injects the seeded credential into the public HTML for up to the
bootstrap deadline. A CLI-flag handshake cannot fix this uniformly: the
studio run path uses ignore_unknown_options and an old in-venv child
runs in-process, so it would never reject the flag.
Delete the seeded .bootstrap_password file in the parent before re-exec
so a fresh child of any version reads None and never serves it. This
covers both re-exec paths and both child versions. must_change_password
stays set, so the login page still forces a change and the bootstrap
shutdown timer still arms; only the plaintext-on-disk copy is removed.
Recovery is via a terminal-attached run or reset-password. Backend gate
and CLI warnings updated to match.
* Studio: commit the seeded admin before headless public re-exec
The headless-warn path deletes the seeded .bootstrap_password so a
re-exec'd child cannot inject it, but _ensure_cli_default_admin's INSERT
was never committed and rolled back on conn.close(). On a fresh
STUDIO_HOME an old studio-venv child then found no admin, regenerated a
fresh bootstrap password + file, and injected THAT into the public page,
defeating the deletion.
Commit the seeded admin right after _ensure_cli_default_admin so any
re-exec'd child sees the existing account and does not regenerate.
Regression tests cover both re-exec paths on a fresh (unseeded) DB.
* Studio: fail closed when the bootstrap password file cannot be removed
On the headless public path, deleting .bootstrap_password is the
protection against an old re-exec'd child injecting the seeded
credential. If unlink fails (locked file, read-only auth dir) the file
is still on disk, so warning and proceeding would still leak it for the
bootstrap-timeout window. Abort with a clear error instead. Regression
test covers the unlink-failure fail-closed path.
* Studio: hold no-echo for the whole password line, not per keystroke
The POSIX masked reader set cbreak/no-echo inside _getch_posix and restored
the terminal to echo-on in a finally after every single keystroke, because
_read_password calls _getch once per character. Between one char returning and
the next call re-entering cbreak, ECHO was on, so a keystroke arriving in that
window echoed the password in cleartext.
Move the terminal mode into a _prompt_raw_mode context that _read_password
holds around the entire line (mirroring unsloth_cli/commands/_password_prompt.py,
which already did this), restoring once when the line completes or aborts.
_getch_posix now only reads, since the mode is held by the caller. The context
is a no-op when stdin is not a real terminal, keeping the _getch test seam.
Add a regression test asserting the raw-mode context wraps the read exactly
once and every keystroke is read while it is active.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strip the seeded bootstrap password when the auth DB check fails
The pre-exposure gate returned early on two auth-DB inspection failures and
proceeded to re-exec without removing the seeded .bootstrap_password:
- _connect_auth_db() failure: a seeded credential from a prior run may still
be on disk.
- the must_change_password read-back failure: worse, _ensure_cli_default_admin
had already seeded the admin and the code committed it (writing
.bootstrap_password) right before the failing SELECT.
In the mixed-version case (a new outer CLI re-execing an old studio-venv child
that predates the pre-bind gate), that child would read the file back and
inject the default admin credential into the public Cloudflare page. The
sibling headless branch already deletes the file for exactly this reason, so
these returns were an inconsistent gap.
Factor the delete-or-fail-closed logic into
_strip_seeded_bootstrap_password_or_exit and call it on both inspection
failures (and reuse it in the headless branch): strip the seeded file first
(version-independent protection), failing closed if the removal itself fails.
must_change_password stays set, so the login page still forces a change and the
bootstrap shutdown timer still arms.
Add tests for both new paths (connect failure and post-commit read-back
failure strip the file and proceed; a failed strip fails closed).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fail closed when the seeded admin cannot be committed before exposure
The pre-exposure gate wrapped _ensure_cli_default_admin (the INSERT), its
conn.commit(), and the must_change_password read-back in one try, and the
except recovered by stripping .bootstrap_password and proceeding to re-exec on
the assumption the admin was already committed. That assumption only holds when
the failing statement is the SELECT. When the INSERT or the commit itself fails
(e.g. a write lock held past the busy timeout on a fresh install), no admin row
is committed: it rolls back on conn.close(), and a re-exec'd old studio-venv
child (no pre-bind gate) then finds no admin, regenerates a fresh bootstrap
password + file, and serves that default credential on the public Cloudflare
page. Stripping the file cannot stop a regeneration.
Split the seed+commit into its own try that fails closed (refuse the public
launch, best-effort removing any half-written seed file) since we cannot prove a
committed admin; keep the separate read-back failure on the strip-and-proceed
path, where the admin is committed so an old child finds it and will not
regenerate. Add a test for the seed-commit-failure path.
* Studio: decode the CLI masked password reader with errors="replace"
The CLI reader read keystrokes with text-mode sys.stdin.read(1), which raises
UnicodeDecodeError on a pasted non-UTF-8 password (e.g. Latin-1 bytes), or under
PYTHONUTF8 yields a lone surrogate that later crashes the pbkdf2 encode -- either
aborts the launch with a traceback. The backend mirror (terminal_prompt.py)
already reads raw bytes through an incremental decoder with errors="replace".
Mirror that here: read with os.read and an incremental decoder so invalid bytes
map to U+FFFD, iterating over each emitted char (one byte can complete a
replacement plus the next char).
* Studio: resolve the child launcher before the pre-exposure gate
The gate strips the seeded .bootstrap_password on a headless public launch, and
it ran before the re-exec launchability check (studio venv / run.py / console
script present). So a headless launch with an incomplete studio setup would seed
the admin, delete the bootstrap password, then abort because the child could not
be found, leaving the admin at must_change_password=1 with no password ever
shown or injectable: locked out until `unsloth studio reset-password`.
Resolve and validate the child launcher first, in both `studio` (studio_default)
and `studio run`, and only then run the gate, so an unlaunchable setup exits
before anything is stripped. Add a regression test that a missing venv exits
without removing the seeded file.
* Studio: fail closed when the auth DB cannot be opened before exposure
The connect-failure branch of the pre-exposure gate stripped .bootstrap_password
and proceeded, on the assumption a committed admin from a prior run made an old
child find it and not regenerate. But on a fresh public launch whose
_connect_auth_db() itself fails (transient lock during the schema/seed step, or
an unwritable home), no admin is committed, so a mixed-version re-exec child that
predates the backend gate can find no user, generate a fresh bootstrap password,
and serve it on the public Cloudflare page. Stripping a file we cannot vouch for
cannot stop a regeneration.
Make this branch fail closed like the seed/commit failure path: we only continue
past the DB inspection once a committed admin is confirmed. The existing file is
left untouched so a retry (after a transient lock clears) can still prompt.
Update the connect-failure test to assert fail-closed, and give the in-venv
--secure flag test a real STUDIO_HOME with an already-changed admin so the gate
is a no-op rather than relying on a DB-open failure.
* Studio: invalidate seeded bootstrap files before deleting auth.db on reset
reset-password deleted auth.db first, then best-effort unlinked the seeded
.bootstrap_password and desktop secret. unlink() only ignores
FileNotFoundError, so a locked or read-only file (Windows AV, read-only auth
dir) survived while auth.db was gone. The next server start then re-seeded
from that stale plaintext and re-validated the exact credential the reset was
meant to revoke.
Invalidate the credential files first, truncating any that cannot be
unlinked, then delete the DB, so a surviving file can never carry a reusable
secret. clear_bootstrap_password now truncates on unlink failure for the same
reason, and its warning says the contents were cleared rather than claiming
the stale password is already invalid.
* Studio: require a servable frontend before the pre-exposure gate can strip the seeded password
A headless public launch strips the seeded .bootstrap_password before the
re-exec'd child starts. If the child then cannot serve the login page (the only
in-band way to change the seeded password) the admin is locked out
(must_change_password=1, no file, no UI) until reset-password.
Add _require_servable_frontend_or_exit and call it before the gate on both
`unsloth studio` and `unsloth studio run` public launches: fail closed if a
non-api-only public launch has no built frontend dist, before anything is
stripped. A user-supplied --frontend is validated to contain index.html so a
bad path cannot silently bypass the check; an auto-resolved dist is trusted
(_find_frontend_dist already requires index.html) and forwarded to the child.
Model-load aborts on `studio run` remain a residual: the parent must strip for
mixed-version safety (an old studio-venv child has no pre-bind gate) and model
loadability cannot be proven before exec, so that path stays recoverable via
reset-password.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden reset-password ordering and validate the in-venv backend before the strip
Three follow-ups to the pre-exposure hardening:
reset-password now deletes auth.db FIRST and proves it is gone before touching
the seeded credential files. If the DB cannot be removed (a running Studio or
Windows holds it open, or a read-only auth dir) it aborts with the credential
files untouched, so a forgotten-password reset is not left half-done with the
recovery credentials deleted while an un-resettable must_change_password=1 DB
survives. After the DB is gone it invalidates the stale credential files
(unlink, else truncate) and fails closed if a file can be neither removed nor
truncated, since a surviving plaintext would be re-seeded by
generate_bootstrap_password() and re-validate the revoked password.
The in-venv (in-process) launch path had no analogue of the re-exec launcher
check: a headless public launch would seed the admin and strip the seeded
.bootstrap_password in the gate before _load_run_module() later failed on a
broken/partial venv, leaving must_change_password=1 with no password to log in.
Add _validate_inproc_backend_before_strip, called on the in-venv path (both
`unsloth studio` and `unsloth studio run`) before the gate on the headless
public path, so a broken backend fails cleanly before anything is stripped. It
is scoped to the headless path so an interactive prompt is not delayed behind a
full backend import.
* Studio: validate the frontend and tunnel before the strip on every public path
Five follow-ups closing the remaining pre-exposure-strip lockouts:
The in-venv (in-process) paths of both `unsloth studio` and `unsloth studio
run` validated the backend but not the frontend before the gate, so a headless
public launch with a missing/bad dist would strip the seeded .bootstrap_password
and then abort in run_server() during frontend setup, leaving
must_change_password=1 with no login page. Both now validate a servable frontend
before the strip (cheap check first, backend import after) and serve the
resolved dist in-process.
The `studio run` re-exec discarded the dist that satisfied the pre-strip check
and only forwarded a user-supplied --frontend. In a shadowed install where the
parent finds a built dist the child cannot, it stripped and exec'd without the
path, and the child aborted during frontend setup. It now forwards the resolved
dist, matching `unsloth studio`.
On a headless --secure launch the bind is loopback, so the Cloudflare tunnel is
the only public exposure. If cloudflared is provably unavailable (found nowhere
and undownloadable) the tunnel cannot start, so stripping the recovery
credential would just lock the user out with no public URL ever served. Add
_tunnel_binary_confirmed_unavailable and, on --secure only, refuse the launch
with the credential preserved rather than strip. Wildcard --cloudflare binds
0.0.0.0 publicly regardless of the tunnel, so it still strips; any uncertainty
(helper not loadable) also still strips, since a possible credential leak
outweighs a recoverable lockout.
clear_bootstrap_password no longer claims it cleared the file's contents when
both unlink and truncate failed; it now reports the stale password is still on
disk and asks the user to remove it manually.
* Studio: fix cloudflared probe path and skip the bootstrap strip for a self-suppressing child
Two follow-ups to the --secure pre-exposure hardening:
The cloudflared availability probe loaded cloudflare_tunnel by file path but not
its backend deps: ensure_cloudflared() -> _cache_path() lazily imports
utils.paths.storage_roots, which only resolves when studio/backend is on
sys.path. From the outer CLI it is not, so the probe saw ensure_cloudflared()
return None (cache unresolvable) and wrongly treated the tunnel as unavailable,
refusing --secure even when cloudflared was cached or downloadable. Add the
backend dir to sys.path for the probe (and remove it after) so the cache path
resolves as it will in the child.
A headless --secure launch stripped the seeded .bootstrap_password before the
child proved the tunnel could actually connect, so a cloudflared that is present
but cannot establish the tunnel (blocked connectivity, Cloudflare outage) left
must_change_password=1 with no recovery credential. But the strip is only needed
when the re-exec'd child is an OLD studio-venv backend with no pre-bind
suppression: this install's own run.py sets app.state.suppress_bootstrap_injection
before binding and never serves the seeded credential publicly. Add
_child_self_suppresses (true in-process, or when the re-exec target is this
install's own run.py by path identity) and skip the strip in that case, keeping
.bootstrap_password as a local recovery credential; the strip stays fully in
force for the studio-venv console-script path and any venv-fallback run.py, where
an old child is actually possible.
* Studio: reword the pre-exposure terminal password prompt
* Studio: warn when -H is overridden by --secure; align pre-exposure prompt wording
- --secure/--secure run: emit a Note (not an error) when -H is a non-loopback
host, since --secure forces the loopback bind and would otherwise discard -H
silently.
- Reword the pre-exposure terminal prompt to 'exposed on the public internet'
in both the backend gate and the CLI mirror.
- Align the CLI success line with the backend ("Password updated for '<user>'.").
- Tests for the new -H warning (present when overridden, absent on loopback).
* Studio: add non-interactive --password to set the initial admin password
Headless hosts (CI, containers, systemd units) have no TTY, so the forced
first-exposure password change could not be completed unattended. Add a
non-interactive way to set the INITIAL admin password before the server binds:
- --password <value>, the UNSLOTH_STUDIO_PASSWORD env var, or --password -
(read one line from stdin). Off by default; unset falls back to the normal
interactive terminal prompt / browser setup.
- Applies on any launch (public --secure/--cloudflare or a headless -H 0.0.0.0
bind), only when the account still has its seeded bootstrap password. An
already-set password is a hard error, never an override; an invalid value
(too short, or equal to the bootstrap) fails closed before bind.
- The CLI applies the change in the parent, never forwards --password to the
re-exec child, and strips UNSLOTH_STUDIO_PASSWORD from the child env so the
secret never crosses to the child. run.py does the same on the direct path and
strips the env var so spawned subprocesses (cloudflared, llama-server, tools)
cannot inherit it.
Mirrors resolve_supplied_password across the CLI and backend, documents the
option in the README (including the argv-visibility caveat), and covers all
flows (env/stdin/literal, fail-closed cases, no-forward, env-strip,
reset-password roundtrip) in the CLI, backend, and unit suites.
* Studio: truncate the stale bootstrap file when unlink fails on a CLI password change
The post-change cleanup in _cli_update_password only warned when
.bootstrap_password could not be unlinked but was still writable (locked file,
read-only auth dir), leaving the old plaintext on disk. If auth.db is later
reset or removed, generate_bootstrap_password() reads that file back and
re-validates the revoked bootstrap password. Truncate the file on unlink
failure so its stale plaintext cannot be re-seeded, mirroring the backend
clear_bootstrap_password(); the password change is already committed, so this
never rolls it back. The warning now states truthfully whether the contents
were cleared or the file must be removed manually.
* Studio: tighten comments
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: permission levels for chat tool calls (Ask, Approve for me, Off, Full access)
Replace the Bypass permissions on/off toggle with a four level permission
selector, available in Settings > General (new Permissions section above
Notifications), the chat settings panel, the composer plus menu, and a new
always visible composer pill.
Levels:
- Ask for approval: every local tool call pauses for allow/deny.
- Approve for me: only calls detected as potentially unsafe pause; the
python/terminal sandbox stays on.
- Off: never pauses; sandbox stays on (previous default behavior).
- Full access: never pauses and the sandbox is disabled. Still requires
the danger confirmation and is never restored across reloads.
Backend adds permission_mode to the OpenAI compatible and Anthropic
passthrough payloads and threads it through both tool loops. Auto mode
uses a fail closed classifier in tools.py: terminal commands must be on
a read only allowlist with no redirection or substitution, python code
is AST scanned for writes, exec, process and network use, MCP tools
auto run only with read only style names. Unknown tools always ask.
Legacy bypass_permissions and confirm_tool_calls keep their exact
behavior for existing API callers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio permissions: Off is a plain toggle below Full access
Off moves to the bottom of the level menu with a short description and
acts as the feature-off state: the composer pill is hidden entirely
while Off, and reselecting the active level toggles back to Off.
* Studio permissions: higher contrast composer pill text
The permission pill uses a foreground based grey instead of the shared
muted pill color, so it reads darker in light mode and lighter in dark
mode. Full access keeps the danger yellow.
* Studio permissions: panel dropdown layout and shorter tooltip
Chat settings panel: the Bypass permissions label sits on one line with
a full width dropdown underneath, styled like the other panel selects.
Tooltip shortened and wording uses Unsloth instead of Studio.
* Studio permissions: harden auto-mode unsafe detection
Extend the Approve for me classifier to catch write and exec paths that
slipped through:
- terminal: sort -o, tree -o, xxd -r, find -exec/-execdir/-ok/-delete
and find -fprint/-fprintf/-fls now ask; plain read-only forms still
auto-run. awk is no longer allowlisted since its program can write and
call system().
- python: from-imports of mutating names (from os import remove [as rm])
and star imports now ask.
Found by a fuzz and edge-case simulation matrix; pinned in
test_permission_mode.py.
* Studio permissions: split multi-line terminal commands in auto detection
A shell runs each line as its own command, but shlex reads newlines as
whitespace, so "ls\nrm -rf x" demoted rm to argument position and
auto-ran. Normalize newlines and CR to separators, and treat any all
separator token as a command boundary so runs of blank lines still
split. Found by the simulation matrix; pinned in tests.
* Studio permissions: address review feedback on auto-mode detection
Auto-mode (Approve for me) safety classifier hardening:
- Python: flag any reference to a mutating attribute, not only direct
calls, so indirect refs (f = os.remove; f(x)) and aliases ask. Detect
Path.open(mode) write modes and wrap the AST walk to fail closed.
- Terminal: match attached short output flags (sort -o/tmp/out) and keep
find context across grouping parens so find ( -delete ) asks.
- Both: ask before reads that escape the sandbox workdir via parent
traversal or hit credential paths (.ssh, .aws, id_rsa, .pem, etc.).
permission_mode plumbing:
- Fold permission_mode=full into bypass_permissions at the request model
so route-level confirm-gate guards see it as bypass.
- Reject ask/auto on the Anthropic Messages server-tools path, which has
no confirmation channel (mirrors the confirm_tool_calls rejection).
- Keep forced RAG autoinject in auto mode: the safe search_knowledge_base
retrieval never gates, so derive the skip from the real confirm need.
- Reset all local preferences now also clears the legacy confirm key so a
reset restores the fresh default instead of the old level.
Regression tests added for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio permissions: close auto-mode classifier gaps from review round 2
Auto mode ("Approve for me") let a few mutating calls through as safe:
- os.open(...) always creates/writes a descriptor, so treat it as unsafe
even though builtin open in read mode stays safe.
- fd -x/--exec/-X/--exec-batch runs a command per match; scan for these
alongside find's -exec/-delete.
- tempfile writes artefacts and hands back writable handles, so importing
it now asks.
- Calling the result of a call (getattr(os, "remove")("x"), partials) is a
dynamic target the AST can't vet, so fail closed.
- An MCP tool whose name pairs a read verb with a mutating one
(get_or_create_issue, read_and_delete_file) no longer auto-runs on the
read prefix alone.
Also fold permission_mode="off" into confirm_tool_calls=False on both
request models so the non-stream route guard sees the disabled gate, and
drive the Confirm tool calls toggle off permission_mode="ask" so auto no
longer shows it on.
* Harden auto-mode classifier and normalize bypass to full for PR #7079
Approve for me now asks for a few cases it previously auto-ran:
- os.open via an os alias (import os as o; o.open(path, O_CREAT))
- pathlib symlink_to / hardlink_to / link_to
- importlib.import_module dynamic imports
- os.mkfifo / os.mknod / os.utime
Also fold bypass_permissions into full when a stale ask/auto permission_mode
is sent alongside it, so the Anthropic route guard no longer 400s those legacy
callers. Adds classifier and request-model regression tests.
* Close more auto-mode classifier gaps for PR #7079
Approve for me now asks for cases the review surfaced:
- builtin open aliased to a name (f = open; from builtins import open as w)
or looked up dynamically (globals()['open'])
- pickle / marshal / shelve / dill deserialization
- io.FileIO write handles
- sort --compress-program (runs an external program)
- MCP names carrying save/archive/submit/commit/push/sync/register verbs
Also refine the attribute open() write check so an explicit read mode
(ZipFile.open(name, "r")) stays auto while os.open flags still ask. Adds
test coverage for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close three more auto-mode gaps for PR #7079
- rg runs an arbitrary program per file via --pre / --hostname-bin, so
"Approve for me" now asks for those flags (rg is on the read-only
allowlist).
- A path-qualified command token (./ls, /tmp/cat) is an arbitrary
executable, not the trusted utility its basename matches, so it asks
before running.
- A direct /chat/completions caller that sets permission_mode ask/auto
but omits the legacy confirm_tool_calls flag now self-enables the
confirmation gate, so tools can no longer run ungated on that path.
Adds classifier and request-model tests for each case.
* Close auto-mode classifier gaps from review round 3 for PR #7079
Approve for me now asks for cases the latest pass surfaced:
- short-option clusters bundling a write flag (sort -uo out => -u -o)
- procfs reads that leak a process env/args/memory
(cat /proc/self/environ, /proc/PID/cmdline, maps)
- env-assignment prefixes that change command lookup/loading
(LD_PRELOAD=x ls, PATH=. ls, IFS=x ls); benign FOO=1 cmd stays auto
- os.open imported as a bare callable (from os import open as o)
Also drops ps from the safe terminal allowlist: its BSD environment
flags (ps auxe, ps eww) dump a parent process's unscrubbed env and
cannot be flag-parsed reliably, so ps always asks now. Adds classifier
tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 4 for PR #7079
Terminal (Approve for me now asks for these):
- cd dropped from the safe allowlist: cd /; cat etc/passwd moves the
shell out of the session workdir so a later relative read escapes it
- env -C/--chdir (workdir escape) and -S/--split-string (builds a fresh
command line); wrapper flags are now checked
- /etc//passwd and /etc/./passwd normalize to /etc/passwd before the
sensitive-path scan
- a sensitive path split across an assignment and an argument
(p=/etc; cat $p/passwd) via best-effort NAME=value expansion
Python:
- builtins.exec / builtins.eval attribute calls (dynamic code execution)
- destructured open aliases (f, _ = (open, print); f('out', 'w'))
- a sensitive path composed from literals (os.path.join('/etc','passwd'),
'/etc' + '/passwd')
- ZipFile/TarFile write modes (ZipFile(name, 'w')); the reader stays auto
Adds classifier tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 5 for PR #7079
Terminal (Approve for me now asks for these):
- procfs reads hidden by shell quotes (cat /proc/$PPID/enviro''n) or
quoted/nested-variable assignments (p="/proc/$PPID"; cat $p/environ):
quotes are stripped and NAME=value prefixes expanded before the scan
- LESSOPEN/LESSCLOSE, which make less run an input preprocessor command
Python:
- os.chdir / os.fchdir, which move the cwd so a later relative read
escapes the sandbox workdir
- sensitive paths composed via a pathlib / chain (Path('/etc') / 'passwd')
or an f-string of literals (f'/proc/{pid}/environ')
- runpy (import) and runpy.run_path / run_module, which run arbitrary code
Adds classifier tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 6 for PR #7079
Approve for me now asks for these:
- a mutating callable reached through a getattr alias
(rm = getattr(os, "remove"); rm("f")): calls through a getattr-bound
name fail closed
- compound MCP tool names carrying clone/checkout/comment/fork/tag/
invite/share, which start with a read verb but still mutate
- a sensitive path hidden behind a glob (cat /e??/passwd,
cat /e[t]c/passwd): a ? / * / [..] token is matched against the
sensitive-file set and bracket classes are de-obfuscated; benign
globs (ls *.py) stay auto
Also run first-pass RAG retrieval in off mode: like auto, off never
prompts, so a direct caller passing a stale confirm flag should not lose
document retrieval (both tool loops).
Adds classifier tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 7 for PR #7079
Approve for me now asks for these:
- __builtins__.exec / __builtins__.eval (dynamic code via the dunder)
- terminal reads that hide a credential path behind a backslash escape
(cat /et\c/passwd)
- read-named MCP filesystem calls pointed at a credential path
(mcp__fs__read_file {"path": "/etc/passwd"})
- compound MCP names carrying append / prepend
- open aliased through a subscript or builtins attribute
(f = globals()["open"]; f = builtins.open) then called to write
- open(..., **{"mode": "w"}) where a kwargs splat hides the write mode
- a sensitive path with a dynamic segment (open(f"/etc/{name}"),
os.path.join("/etc", name)); /tmp/{name} stays auto
- urllib3 networking
Also stop folding permission_mode ask/auto into confirm_tool_calls for
external-provider requests: that branch rejects confirm_tool_calls with
tools, and the mode only governs local tool calls. Local requests still
self-gate. Adds tests for each case.
* Close auto-mode classifier gaps from review round 8 for PR #7079
Approve for me now asks for these:
- dbm on the unsafe-module list: dbm.open(file, "c"/"n") creates files,
and importing the family signals a persistence writer
- reads of ~/.azure and ~/.config/gh credential stores (Azure/GitHub
tokens), in terminal, MCP arguments, and Python literals
- compound MCP names carrying upsert / assign
Adds classifier tests for each case.
* Gate secret mounts and fix the composer pill count for PR #7079
- Add Docker/Kubernetes secret mount dirs (/run/secrets,
/var/run/secrets) to the sensitive-path checks, so Approve for me asks
before reading injected credentials (terminal, MCP args, Python).
- Count the always-visible permission pill in the composer's compact
threshold so labels collapse at the intended width instead of
overflowing by one pill.
Adds classifier tests for the secret mount paths.
* Close auto-mode classifier gaps from review round 10 for PR #7079
Approve for me now asks for these:
- qualified pathlib constructors (pathlib.Path('/etc') / name), folded
the same as bare Path(...), so a dynamic sensitive path is detected
- open aliased through an annotated assignment (f: object = open;
f('out', 'w')), tracked like a plain assignment
- recursive searches rooted at an absolute path (grep -R TOKEN /home,
rg TOKEN /, fd pattern /etc), which read host files outside the
sandbox tree; sandbox-relative searches stay auto
Adds classifier tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 11 for PR #7079
Approve for me now asks for these terminal reads, which bash would
expand into a sensitive path only after the classifier had approved:
- a glob that resolves into a secret mount or credential dir
(cat /r?n/secrets/hf_token, cat /root/.s??/id_rsa)
- a recursive search rooted at a tilde home (grep -R TOKEN ~root,
grep -R TOKEN ~/logs)
- a brace expansion that builds a credential path (cat /etc/pass{w,}d)
- a default/alternate parameter expansion that builds one
(cat /etc/pass${x:-wd})
- an input redirection that hides a glob (cat </e??/passwd)
And these python calls:
- a str.format-built sensitive path (open('/etc/{}'.format('passwd')))
- writer methods that persist to disk without open() (numpy.save,
Image.save, plt.savefig, DataFrame.to_csv, json.dump)
Segment-wise directory matching keeps benign globs (ls /home/*/projects)
auto. Adds regression tests for each case and its safe counterpart.
* Close auto-mode classifier gaps from review round 12 for PR #7079
Approve for me now asks for these too:
- a terminal read whose parent traversal hides behind a redirection with
no following space (cat <../../notes)
- a python read whose path is built with str.join
(open(''.join(['/etc', '/passwd']))), told apart from os.path.join
- a dynamic-code builtin reached through an alias
(from builtins import eval as e; e(...); x = builtins.exec; x(...))
Adds regression tests for each case and its safe counterpart.
* Close auto-mode classifier gaps from review round 13 for PR #7079
Approve for me now asks for these too:
- a recursive search whose root is hidden behind an assignment
(p=/; grep -R TOKEN $p): the recursive-root test now runs on the
assignment-expanded tokens as well
- a python read whose sensitive path is split through a literal variable
(base = '/etc'; open(base + '/passwd')), including via an f-string
- numpy ndarray.tofile, which persists without open()
- a sequence brace read (cat /etc/pass{w..w}d), expanded alongside the
comma brace form before the sensitive-path scan
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 14 for PR #7079
Approve for me now asks for these python reads that assemble a sensitive
path in a form the fold did not yet recognize:
- a pathlib object reused through a name (p = Path('/etc'); p / 'passwd')
- old-style percent formatting ('%s/%s' % ('/etc', 'passwd'))
- Path.joinpath ('/etc'.joinpath('passwd'))
- a bytes path literal (open(b'/etc/passwd'))
And these terminal reads, which bash expands into a sensitive path only
after the classifier had approved:
- a substring parameter expansion off an assignment
(p=passwd; cat /etc/${p:0:6})
- an ANSI-C quoted path (cat $'/etc/pass\x77d')
- a glob into an Azure or GitHub CLI config dir
(cat /home/*/.az?re/..., cat /home/*/.config/g?/...)
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 15 for PR #7079
Approve for me now asks for these terminal reads, which bash expands into
a sensitive path only after the classifier had approved:
- a per-thread procfs env alias (cat /proc/$PPID/task/$PPID/environ)
- a recursive root behind a default parameter (grep -R TOKEN ${root:-/home})
- a path built by pattern replacement (p=passXd; cat /etc/${p/X/w})
And these python reads:
- a pathlib .parent/.parents chain that escapes the session workdir
((Path.cwd().parent / 'other' / 'notes').read_text())
- a sensitive path resolved through glob (glob.glob('/e??/passwd')[0])
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 16 for PR #7079
Approve for me now asks for these terminal reads, which bash expands into
a sensitive path only after the classifier had approved:
- a case-modifying parameter expansion (p=PASSWD; cat /etc/${p,,})
- a mutating find action hidden behind an assignment (f=-delete; find . $f)
- a glob assembled through an assignment (g=e??; cat /$g/passwd)
- a POSIX bracket class glob (cat /etc/pass[[:lower:]]d)
And these python reads/writes:
- a glob pattern folded from a literal variable
(base='/e??'; glob.glob(base + '/passwd'))
- a directly imported os.path.join (from os.path import join; join('/etc', 'passwd'))
- a directly imported writer (from numpy import save; save(...))
- an aliased pathlib constructor (from pathlib import Path as P; P('/etc') / 'passwd')
The find/fd and glob scans now run on the assignment/parameter-expanded
command, and pathlib/join/writer import aliases are tracked. Adds
regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode gaps from review round 17 for PR #7079
Two fixes:
- Gate sqlite3 in auto mode. sqlite3.connect(path) creates or mutates a
database file (and runs DDL/DML) with no open()/writer attribute for
the AST checks to catch, so treat the module like dbm and ask.
- Only self-enable confirm_tool_calls for Studio's own tool loop. The
ask/auto fold previously set confirm on every non-provider request,
including a plain client-tool passthrough (client-supplied tools that
Studio does not execute), which then tripped the local-tool
streaming-confirm route guard and rejected the passthrough. Restrict
the fold to requests that actually ask Studio to run tools
(enable_tools / enabled_tools / mcp_enabled).
Adds regression tests for the sqlite3 write and for the passthrough vs
tool-loop confirm behavior.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode gaps from review round 18 for PR #7079
Classifier (auto mode asks for these):
- os.open through a module alias (import os as o; o.open(...)); os/posix
aliases are tracked like the literal module name.
- less/more pagers, whose escapes (+cmd, !shell, -o/--log-file, LESSOPEN)
can run a command or write a file the command-name allowlist cannot
see, so they are no longer auto-approved.
- a read-named MCP tool carrying a mutating query
(query_database {"query": "DELETE FROM runs"}); DML/DDL statements are
matched as whole statements so a natural-language query that merely
contains "delete" stays safe.
- ML persistence helpers (save_pretrained / save_file / save_model /
save_weights / save_lora / save_checkpoint) that export weights to disk.
Route:
- Honor CLI-forced tools when deriving the confirm gate. When a process
policy (unsloth run --enable-tools) opens the local tool loop without a
request-level tool signal, a permission_mode ask/auto request now
derives confirm at the route (GGUF and safetensors paths) so the mode
still gates the call, and a non-streaming ask/auto request is rejected
rather than running unprompted. A plain client-tool passthrough (no
local loop) is unaffected.
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 19 for PR #7079
Approve for me now asks for these too:
- a terminal read whose path is built by indirect parameter expansion
(x=passwd; p=x; cat /etc/${!p})
- a bash /dev/tcp or /dev/udp redirection, which opens a network socket
(cat </dev/tcp/host/port)
- a python read via pathlib's receiver-plus-pattern glob
(Path('/etc').glob('passw?'))
- a python read whose sensitive root passes through a normalizer
(os.path.abspath('/etc'), Path('/etc').resolve())
- a pickle-backed loader that can execute code on load
(torch.load, joblib.load, pandas.read_pickle), tracked through module
import aliases
- compiled code wrapped into a callable (compile(...) + types.FunctionType)
Adds regression tests for each case and its safe counterpart.
* Honor unset permission_mode as ask across the local tool loop for PR #7079
Three gaps where an omitted permission_mode did not behave as the
documented default ("ask"):
- The frontend only sent permission_mode / confirm_tool_calls /
bypass_permissions when a tool pill was on. A process policy
(unsloth run --enable-tools) can open the tool loop with no pill, so
the backend never saw the selected gate. Send the three permission
fields at the top level of every local chat payload instead.
- The backend read payload.confirm_tool_calls directly at the
pre-switch guard and both late per-backend derivations, so an unset
mode fell through as no-gate even for an explicit ask/auto. Add
_permission_mode_confirm(payload): explicit confirm_tool_calls wins,
explicit ask/auto engage the gate, off/full never prompt, and an
unset mode defaults to ask only where realizable (streaming), keeping
the legacy no-gate run for non-streaming unset requests.
- A forced ask/auto tool loop (CLI --enable-tools) with no stream now
400s at the pre-switch guard before evicting the resident model,
matching the existing confirm-without-stream rejection.
Adds test_permission_mode_confirm_derivation covering the derivation
truth table.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Declare permission_mode and bypass_permissions on the local chat request type
The previous change moved permission_mode, confirm_tool_calls and
bypass_permissions to the top level of the local chat payload. They had
lived inside a conditional spread, which is not subject to excess
property checking, so the fields were never declared on
OpenAIChatCompletionsRequest. At the top level tsc flagged
permission_mode as unknown (TS2322), failing the frontend build and
every job whose Studio install builds the frontend.
Add permission_mode and bypass_permissions to the request interface
(confirm_tool_calls was already present).
* Close auto-mode classifier gaps from review round 21 for PR #7079
Auto mode ("Approve for me") now asks for these too:
- a pathlib read built from a concrete constructor (PosixPath, WindowsPath
and their Pure* forms), which the folder previously ignored so
PosixPath('/etc') / 'passwd' lost its /etc root and ran unprompted
- a terminal or python read of the ssh host keys under /etc/ssh, which
the sensitive-path regex only covered for passwd/shadow/sudoers
- a read whose path variable is reassigned: the whole-tree pre-scan kept
the last binding, so base = '/etc'; open(base + '/passwd'); base = 'data'
folded to data/passwd and ran even though execution reads /etc/passwd;
any multiply-bound name now folds to the escape sentinel and asks
Also stop the pre-switch guard from rejecting a plain client-tool
passthrough. permission_mode only implies the confirm gate for Studio's
own local tool loop (enable_tools / enabled_tools / mcp_enabled); a
non-streaming client-tool passthrough that carries permission_mode
ask/auto (confirm_tool_calls left unset by the validator) must forward to
the provider branch. Only an explicit confirm_tool_calls=True still forces
the local-confirm rejection there.
Adds regression tests for each case and its safe counterpart.
* Fix permission-pill compaction count and Full-access confirm sync for PR #7079
Two frontend consistency issues in the permission-level UI:
- The composer collapses tool pills to icons above four, but the count
left out the permission pill, which renders in every mode except off.
With one optional pill also shown the row reached five pills without
collapsing and could overflow. Count the pill when it is visible
(permission_mode != off).
- Entering Full access via setPermissionMode('full') or
setBypassPermissions(true) left confirmToolCalls at its previous value,
so a Full-access run (which sends confirm_tool_calls=false) could still
report confirmations as enabled in response metadata. Set
confirmToolCalls false at both entry points.
* Close auto-mode classifier gaps from review round 23 for PR #7079
Auto mode ("Approve for me") now asks for these too:
- a command using an abbreviated GNU long option that reaches a
write/exec action (sort --out= for --output, env --ch= for --chdir,
fd --base-dir= for --base-directory); a prefix of an unsafe long flag
now fails closed
- printf -v NAME, which assigns to a shell variable, so
printf -v PATH %s .; ls can rewrite PATH and run ./ls unprompted
- fd --base-directory / --search-path, which move the search root
outside the session workdir without any positional slash token
- an MCP tool whose compound read name carries a copy-style mutator
(read_and_copy_file, get_and_snapshot_volume): copy, duplicate,
import, export, download, backup, restore, snapshot, mirror
Also treat an omitted permission_mode as its documented default ("ask")
on the Anthropic Messages server-tool path. That branch has no
confirmation channel and already rejects explicit ask/auto, so an
omitted mode now falls into the same rejection instead of silently
running server tools unprompted, unless the caller opted out with
confirm_tool_calls=false (the legacy equivalent of "off"). off/full and
that opt-out still run; the two routing tests that relied on the old
implicit run now set permission_mode="off".
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Refine permission gating from review round 24 for PR #7079
Four fixes from the latest review:
- Anthropic Messages server tools: an omitted permission_mode no longer
rejects a request that only runs safe server tools (web_search), so
existing Anthropic callers keep working. It still rejects an omitted
mode when a local tool (terminal/python) is selected, and an explicit
ask/auto is still rejected outright. off/full and a
confirm_tool_calls=false opt-out always run.
- Pre-switch confirm-without-stream guard: use
_explicit_studio_tool_loop_requested (the same predicate the
passthrough router uses) instead of the policy-inclusive
_effective_enable_tools, so a process --enable-tools policy no longer
turns a client-tool passthrough into a local-loop rejection.
- Auto mode now asks for `uniq INPUT OUTPUT`: uniq writes its second
file positional, so a second positional (numeric flag values skipped)
is treated like `sort -o`. A lone `uniq file` or piped `... | uniq`
stays safe.
- MCP mutation check now strips SQL comments before matching, so
DELETE/**/FROM and UPDATE/**/users (comment-as-whitespace) no longer
slip past the DML/DDL denylist.
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode gaps from review round 25 for PR #7079
Auto mode ("Approve for me") now asks for these Python cases too:
- a bare archive constructor with a write mode (from zipfile import
ZipFile; ZipFile('out.zip', 'w')), tracked through import aliases like
the zipfile.ZipFile attribute call already was
- a dynamic lookup aliased through getattr (g = getattr;
rm = g(os, 'remove'); rm('file')), not just direct getattr(...) calls
- a callable that wraps open or a writer via functools.partial
(w = partial(open, mode='w'); w('out.txt')), which hides the write mode
Also:
- Always-safe tools (render_html) stream their early provisional canvas
card in auto mode again. The provisional-card guard mirrored the raw
confirm flag, which suppressed the early card under Approve-for-me; it
now reuses the auto-mode safety decision (is_always_safe_tool).
- The assistant-ui composer no longer counts the permission pill toward
its collapse threshold when the level is Off (the pill renders null
there), matching the other composer.
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Align permission-mode confirm guards with the router (review round 26)
Three pre-switch confirm-gate checks disagreed with how the tool
loop actually enters, so a valid request could 400 (or an invalid
one could evict the resident model) at the wrong point:
- The /chat/completions pre-switch guard only looked at explicit
request fields, so a process --enable-tools policy that forces the
loop on (request omits enable_tools, no client tools) slipped past
it and only 400ed after _maybe_auto_switch_model had swapped the
model. It now mirrors the router's own loop-entry gate
(_effective_enable_tools or mcp, tool_choice="none" disabling it
unless explicitly asked) while still deferring to client-tool
passthrough, so the policy-forced case is caught before the switch.
- The ChatCompletionRequest full/off fold treated enabled_tools by
itself as a local-loop request and set confirm_tool_calls=True.
The router never starts the loop on enabled_tools alone (it only
filters which tools run), so a non-streaming passthrough carrying
client tools plus enabled_tools 400ed instead of routing verbatim.
The fold now keys off the same enable_tools / mcp_enabled signals.
- The Anthropic /v1/messages unsupported-mode rejection (ask/auto,
or an omitted mode selecting terminal/python) ran inside the
post-switch server-tools block, so an invalid request evicted the
resident model before the 400. It now runs before the auto-switch,
determined from the requested server tools, like the neighboring
malformed- and mixed-tool guards.
Adds regressions for each: a policy-forced non-streaming ask/auto
guard rejection that never reaches the switch, an enabled_tools-only
passthrough that keeps confirm unset, and an Anthropic rejection that
precedes _maybe_auto_switch_model.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 27 for PR #7079
Auto mode ("Approve for me") now asks for these host-mutating or
host-reading cases it previously ran unprompted (the sandbox does not
jail filesystem reads, and terminal commands can change host state):
- Destructured string literals fold into the scanned path now, so
base, leaf = ('/etc', 'passwd'); open(base + '/' + leaf).read()
resolves to /etc/passwd and asks, like the single-assignment form
already did. The tuple/list unpacking branch tracked only aliases to
open; it now also binds literal and folded-path elements.
- pathlib name rewrites fold to the rewritten path:
Path('/etc/x').with_name('passwd').read_text() (and with_stem /
with_suffix) spell no literal /etc/passwd but resolve to it, so they
are folded and caught. Benign in-sandbox rewrites stay safe.
- hostname NAME (or -F/--file, -b/--boot) sets the hostname, so a
positional or a set flag asks; bare hostname and the display flags
(-f/-i/-I/...) stay read-only.
- date -s/--set STRING and the bare MMDDhhmm... positional set the
system clock and now ask; the display forms stay read-only (+FORMAT,
-u/-R, and -d/-r/-f whose following value is skipped so date -d
tomorrow is not mistaken for a clock-setting positional).
Adds regression rows for each gap and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close more auto-mode classifier gaps from review round 28 for PR #7079
Auto mode ("Approve for me") now asks for these cases too:
- Mapping-style %-formatted paths. '/etc/%(f)s' % {'f': 'passwd'} folds
to /etc/passwd and asks; a dynamic value or a non-literal mapping
leaves the NUL marker so /etc/<dynamic> still fails closed. The path
folder previously handled only tuple/scalar % right-hand sides and
returned None for a dict, hiding the sensitive segment.
- A read-named MCP database tool carrying PostgreSQL COPY. COPY ... FROM
bulk-loads a table and COPY ... TO writes a server-side file, so both
are matched as mutating queries like DELETE/UPDATE already were. A
'copy' substring in a column name stays safe (word boundary).
- logging file handlers. logging.FileHandler('out.log', mode='w') (and
the default append mode, RotatingFileHandler/TimedRotatingFileHandler/
WatchedFileHandler, and the bare from-import form) create or truncate
a file like open(..., 'w'), so they are classified as writer calls.
StreamHandler / NullHandler and logging reads stay safe.
Adds regression rows for each gap and its safe counterpart.
* Fix writer aliases, GraphQL mutations, and auto server tools (review round 29)
- Auto-mode Python: an aliased writer or archive constructor is tracked
like the existing open alias, so from numpy import save; s = save;
s('out.npy', arr) (and z = ZipFile; z('a.zip', 'w'), incl. the
destructured forms) ask instead of running the write unprompted. A
benign builtin alias (x = len) stays safe.
- Auto-mode MCP: a read-named tool carrying a GraphQL mutation now asks.
query_graphql {"query": "mutation { deleteIssue(id: 1) }"} matches a
leading mutation keyword (GraphQL uses # comments, so it scans the raw
payload); GraphQL read queries stay safe.
- Anthropic /v1/messages: permission_mode "auto" no longer 400s a
safe-only server-tool selection. auto only needs a confirmation
channel for an unsafe call, so like the omitted default it runs for
web_search / RAG / render and rejects only when a gate-needing local
terminal/python tool is selected. ask still always rejects (it asks
per call, which this passthrough cannot honor). The rejection stays
ahead of the model auto-switch.
Adds regression rows/cases for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate asyncio spawn, net clients, default-captured open; allow safe-only auto (round 30)
Auto-mode Python now asks for more process/network/write vectors:
- asyncio process spawners (asyncio.create_subprocess_exec/shell and a
loop's subprocess_exec/shell) run an arbitrary program without the
terminal blocklist, so they gate like os.system/subprocess.
- stdlib network clients imaplib / poplib / nntplib / xmlrpc(.client) /
webbrowser open outbound connections the sandbox does not namespace
off, so their import asks like the other network modules.
- a callable captured as a function or lambda parameter default
(def f(o=open): o('out', 'w')) now binds that parameter into the same
alias set, so the later write through it is gated. A benign default
(o=len) stays safe.
Also, permission_mode "auto" no longer 400s a non-streaming local tool
request whose selection is always-safe-only (web_search / RAG / render).
auto only prompts for a classifier-flagged call, so a safe-only auto
request needs no stream, while ask, an explicit confirm_tool_calls=true,
MCP, and an unrestricted or unsafe selection still require it. Applied
via a shared _confirm_gate_needs_stream helper at the pre-switch, GGUF,
and safetensors confirm-stream guards; the loop's per-call confirm flag
is unchanged.
Adds regression rows/cases for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Catch brace-glob paths and attribute writer aliases; unfold auto (round 31)
- Terminal auto mode now runs the glob-sensitive scan over every
expansion candidate, so a brace-expanded glob (cat /e{t,}c/pass?d,
which bash expands to /etc/pass?d and then globs to /etc/passwd) asks.
Brace expansion alone spells no literal /etc/passwd and the glob only
resolves once the brace group is expanded, so scanning both together
is required. A benign brace + glob stays safe.
- Python auto mode now tracks a mutating attribute captured as a plain
name: s = np.save; s('out.npy', arr) binds a writer alias, a captured
.open bound method (p = Path('out').open; p('w')) fails closed on any
call since its mode position varies, and z = zipfile.ZipFile is gated
like the bare import. A benign attribute alias (x = np.mean) stays safe.
- permission_mode "auto" is no longer folded to confirm_tool_calls=true
on the request model. Folding it defeated the safe-only-selection
exception in _confirm_gate_needs_stream (an explicit confirm forces
stream=true), so a non-streaming safe-only auto request was rejected.
Leaving it unset lets the route apply the exception; the mode still
drives the loop's per-call gate. "ask" still folds (it gates every
call).
Adds regression rows/cases for each.
* Harden SQL/GraphQL/writer classification and passthrough guards (round 32)
MCP argument mutation detection (read-named query tools):
- CREATE DDL now matches modifiers and the broader object set, so
CREATE OR REPLACE VIEW, CREATE UNIQUE INDEX, CREATE TEMP TABLE,
CREATE MATERIALIZED VIEW and CREATE FUNCTION ask.
- Stored-procedure invocation (CALL proc(...), EXEC/EXECUTE) and VACUUM
ask; a natural-language "call me back" stays safe via the trailing
"(" / ";" / end lookahead.
- GraphQL # comments are stripped before the mutation match, so
mutation # note\n { deleteIssue(id: 1) } no longer hides the mutation.
Python auto-mode classification:
- numpy.memmap / open_memmap and pandas ExcelWriter / HDFStore create or
truncate a file on construction, so they gate like open(..., "w").
- asyncio networking (asyncio.open_connection, loop.create_connection /
create_server and unix variants) opens outbound connections/listeners
the sandbox does not isolate, so it gates like socket.connect.
Terminal auto-mode: file -C / --compile writes a compiled magic database.
Routing:
- A JSON-schema response_format is guided-decoding passthrough, not a
local tool loop, so a --enable-tools policy no longer 400s a
non-streaming ask/auto structured-output request at the confirm guard.
- An explicit confirm_tool_calls=False opts out of the Anthropic Messages
server-tool gate entirely (it wins over the mode, mirroring
_permission_mode_confirm and the GGUF path), so it runs even under ask.
Adds regression rows/cases for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Track path-ctor aliases, exempt empty selection and safe safetensors card (round 33)
- Python auto mode now propagates path constructor / join aliases, so
assigning Path or os.path.join to another local name is still folded:
P = Path; (P('/etc') / 'passwd').read_text() and j = os.path.join;
open(j('/etc', 'passwd')) ask, while a benign /tmp alias stays safe.
- _confirm_gate_needs_stream now distinguishes an omitted enabled_tools
(None, all tools) from an explicit empty list ([], no tools). An empty
selection runs no built-in tool and cannot prompt, so a non-streaming
auto request with enable_tools=true, enabled_tools=[] is no longer
400ed under a --enable-tools policy.
- The safetensors provisional render_html card now uses permission_mode:
render_html is always safe and never prompts, so its early canvas card
streams under auto (which ships confirm_tool_calls=true) instead of
being suppressed, matching the GGUF path's is_always_safe_tool exemption.
Adds regression rows/cases for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Extend auto-mode classifier: SQLite mutations, more net/xattr/compressed writers
Additional fail-closed gaps found by a fresh adversarial pass, each with a
reproduction and a benign control:
- MCP read-named tools now ask on SQLite-flavored writes the base DML/DDL regex
missed: ATTACH / DETACH DATABASE, a write-form PRAGMA (PRAGMA journal_mode=WAL
/ user_version=42 / foreign_keys(0), while the read-form PRAGMA journal_mode
stays safe), and load_extension() which loads and runs an arbitrary shared
library.
- Python auto mode now gates the remaining asyncio network entry points
(start_server, open_unix_connection, loop.create_datagram_endpoint,
sock_connect), os.setxattr / os.removexattr metadata writes, the gzip / bz2 /
lzma single-stream writers (GzipFile / BZ2File / LZMAFile, mode-gated like
ZipFile so a read stays safe), pandas to_xml, and the websockets client.
Benign controls (SELECT 1, read-form PRAGMA, asyncio.sleep, gzip read, numpy
read, natural-language "attach"/"analyze") stay safe. Regression rows added to
test_permission_mode.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close follow-up auto-mode gaps: SQLite/GraphQL variants, more writers and net
A fresh adversarial pass on the previous round found consistent extensions of
the same fail-closed rules, each reproduced with a benign control:
- MCP read-named tools: DROP / ALTER now cover the same broad object set as
CREATE (DROP FUNCTION, ALTER INDEX, DROP MATERIALIZED VIEW); ATTACH is caught
without the optional DATABASE keyword via its quoted-path form; a
schema-qualified write PRAGMA (PRAGMA main.user_version=1) is matched; and a
GraphQL mutation carrying directives (mutation M @audit { ... }) is treated as
a mutation.
- Python auto mode: os.startfile (Windows program launch), asyncio
start_unix_server, and the socketserver framework now ask; a gzip/bz2/lzma
open imported under an alias (from gzip import open as gopen) is gated like
builtin open; and a dynamic path prefix that can form a sensitive absolute
root (open(chr(47) + "etc/passwd"), open(os.sep + "etc/passwd")) is treated as
sensitive, while a dynamic prefix with a benign suffix stays safe.
Benign controls (read-form PRAGMA, natural-language "attach ... as", "drop the
idea", SELECT dropped_at, query @cached, gzip read alias, dynamic prefix +
data/file suffix) stay safe. Regression rows added to test_permission_mode.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate GNU time -o, basicConfig/methodcaller/fileinput, and more SQL mutations
Another adversarial pass surfaced further consistent fail-closed gaps, each
reproduced with a benign control:
- Terminal: GNU time -o/--output/-a/--append truncate or append to a file with
timing output; time is a wrapper, so the flag is checked before the wrapped
command like env -C.
- Python auto mode: logging.basicConfig(filename=...) opens a log file for
write; operator.methodcaller("write_text"/...) hides a writer method behind a
string and is now treated as dynamic dispatch (like getattr/partial);
fileinput.input(..., inplace=True) rewrites a file in place (the default read
form stays safe).
- MCP read-named tools: UPDATE now matches quoted, bracketed, and
schema-qualified targets (UPDATE "users" / public.users / ONLY public.users /
[users] / `users` SET); SELECT ... INTO OUTFILE/DUMPFILE writes a server file;
and state-changing SQL functions inside a SELECT (pg_terminate_backend,
setval, pg_write_file, lo_export, ...) ask.
Benign controls (time ls / time -p, basicConfig(level=), methodcaller("upper"),
fileinput read, NL "update ... set", setval_col column, PL/pgSQL SELECT INTO
var) stay safe. Regression rows added to test_permission_mode.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten auto-mode classifier comments
Collapse the multi-line rationale blocks in the permission classifier to one or
two lines each without dropping the exploit each branch closes. Comments and
whitespace only (no code change); the classifier tests are unchanged and pass.
* Retry transient SSE stalls in the tool-calling smoke probes
The tool-calling job flaked with a bare "TimeoutError: timed out": the
server-side python/bash probes stream over post_sse(), which (unlike
post()) had no transport-level retry, so a single stalled stream on a
shared CI runner hard-failed the whole step even though function calling
had already passed.
post_sse() now mirrors post(): a transport-level stall (stream open or a
mid-stream read timing out) is retried once with a fresh request capped
at 300s, while HTTP status errors still surface immediately. The
Linux _run_tool_probe caps each attempt at 360s and treats a stall that
outlives the retry as a failed attempt (rotate to the next seed) instead
of raising, and the web_search probe uses the same 360s cap. A genuine
server wedge still fails (the retry also times out), so real regressions
are not masked. Applied to the Linux, macOS, and Windows inference-smoke
workflows, which share the probe.
* Close five more auto-mode classifier gaps from review
Each reproduces with a benign control:
- Path constructor aliased through an attribute (P = pathlib.Path) now folds
like the bare-name alias, so (P('/etc') / 'passwd').read_text() asks while a
/tmp alias stays safe.
- Callable defaults that are not plain names now bind the parameter: an
attribute writer (def f(s=np.save)), an archive constructor, a captured .open,
and partial(open, mode='w') fold like the equivalent assignment; a benign
default (np.mean) does not.
- A dynamic piece inside a sensitive name (open('/et' + chr(99) + '/passwd'),
which folds to '/et\x00/passwd') now asks: the literals around each dynamic
segment are matched against a credential target with the segment as any run of
non-separator chars, so an all-dynamic ('1 + 1') or segment-spanning
(a + '/' + b) path stays safe.
- MCP read-named tools now ask on REFRESH MATERIALIZED VIEW and REINDEX; a
'refresh' column or natural-language 'refresh' stays safe.
- A writer/open alias handed to a higher-order invoker (map(open, names, modes),
starmap(np.save, ...)) is gated even without a direct call site; a benign
map(len, ...) is unaffected.
Regression rows added to test_permission_mode.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Default tool pills off on model load so tool execution is opt-in
resolveToolsEnabledOnLoad turned the web-search and code pills on for
any tool-capable model when the user had expressed no preference. Default
them off instead, so tool execution is enabled only when the person
clicks the pill to turn it on; a saved preference (on or off) is still
honoured, so a user who already enabled tools keeps them on.
* Gate mark/subscribe MCP verbs and qualified higher-order writer invokers
- A read-prefixed MCP tool name carrying mark / subscribe / unsubscribe
(get_and_mark_read, get_and_subscribe) now asks; a 'mark' substring inside
one token (list_bookmarks) stays safe.
- The higher-order writer check now also fires for a qualified invoker
(itertools.starmap(open, ...), functools.reduce(open, ...)), matching the
bare-name map/filter form; the writer-check on the first arg keeps a benign
itertools.starmap(len, ...) or itertools.chain(...) safe.
Regression rows added to test_permission_mode.py.
* Close more auto-mode gaps and align the ask confirm fold across paths
Each classifier change reproduces with a benign control:
- MCP read-named tools now ask on reply / notify verbs (get_and_reply_email,
list_and_notify_users), on catalog writes COMMENT ON / SECURITY LABEL / LOCK
TABLE and CREATE|DROP|ALTER POLICY, and on state-changing PostgreSQL functions
inside a read-shaped SELECT (nextval, set_config, pg_notify, the advisory-lock
family). A 'comment' column, a 'locks' table, and a 'nextval' column prefix
stay safe; the natural-language NOTIFY/SET ROLE statement forms are left out
because SET/NOTIFY overlap ordinary prose.
- Python auto mode now gates loader.exec_module (runs a module's code), archive
extractall (zip-slip file writes), the ensurepip / venv modules (install pip /
build an environment), and pydoc.writedoc. The Hugging Face login token
(~/.cache/huggingface/token and stored_tokens) is now a sensitive path, while
the rest of that cache (model data) stays readable.
- ChatCompletionRequest no longer overwrites an explicit confirm_tool_calls=false
when permission_mode='ask': the fold only self-enables the gate when the flag
is unset, so an explicit opt-out wins on the chat path exactly as it already
does via _permission_mode_confirm and the Anthropic pre-switch guard.
Regression rows added to test_permission_mode.py.
* Gate sort -T, xxd outfile positional, and the legacy HF token path
- sort -T / --temporary-directory writes spill files to a caller-chosen dir,
so it joins -o / --output in sort's unsafe-flag set.
- xxd [infile [outfile]] writes its second positional, like uniq; xxd now uses
the same second-positional-write handling (xxd in.bin out.hex asks, xxd
in.bin and xxd -c 16 in.bin stay read-only).
- The sensitive-path regex now also covers the legacy ~/.huggingface/token
location (optional leading dot), not just ~/.cache/huggingface/token; an
unrelated dir like myhuggingface/token stays safe.
Regression rows added to test_permission_mode.py.
* Catch multi-char SQL mutation targets, globbed credential names, digit outfiles
Three fail-open gaps in the auto-mode classifier, each with a benign control:
- SQL: the trailing word boundary on the MCP mutation regex meant a bare \w
stopped at the first character, so TRUNCATE users, GRANT SELECT ON t, and
REVOKE ALL ON t (multi-character names) slipped through while single-letter
targets matched. Match the whole identifier instead, and accept an explicit
AS alias on UPDATE (UPDATE users AS u SET). The implicit-alias form is left
out because it is indistinguishable from the prose "update <noun> <noun> set".
A truncate_log column and a grants table stay safe.
- A glob that resolves to a credential basename anywhere (cat ~/.huggingface/tok?n
-> token, cat proj/.netr? -> .netrc, cat repo/.aws/cred*) now asks; the fixed
target list only covered a handful of home paths. notes/dra?t.txt and
token_counts.tx? stay safe.
- uniq / xxd counted file positionals but skipped every numeric token to ignore
a flag value, so a file literally named with digits (uniq 123 out) hid the
output positional. Track each command's value-taking flags and consume only
the value, so uniq -f 2 in stays safe while uniq 123 out asks.
Regression rows added to test_permission_mode.py.
* Isolate the permission-mode loop tests from process-global state
The loop-driving tests (auto/off/full/bypass) drove run_safetensors_tool_loop
against a process-global approval registry (state.tool_approvals._pending)
keyed by a single shared session id, and read os.environ. Other backend test
modules mutate both, some at import time, so in the full-suite ordering a stale
pending approval or a leaked env var could make the loop deny or skip a call
these tests expect to run. It passed when the file ran alone but failed only in
the complete tests/ run on CI.
Add an autouse fixture that snapshots and restores os.environ and the approval
registry around each test, and give every _drive call a unique session id so a
leaked approval can never collide. Attach a compact event-stream dump to the
loop assertions so any residual full-suite-only failure reports what the loop
actually did instead of a bare diff.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden auto-mode classifier for recursive listers, sort file lists, aliased invokers, single-member extract
Close four fail-open gaps in is_potentially_unsafe_tool_call:
- terminal: tree/du (always recursive) and ls -R rooted at an absolute or
tilde path now ask, matching the existing grep/rg/find recursive-read gate;
relative walks stay safe.
- terminal: sort --files0-from=F reads the file list named in F, so it can
read arbitrary host files indirectly; added to sort's unsafe flags.
- python: track aliases of the higher-order invokers (m = map;
from itertools import starmap as sm) so an aliased invoker handed open/a
writer is still gated; a benign callable (map(len, ...)) stays safe.
- python: single-member archive extract (ZipFile/TarFile.extract) writes to
disk like extractall and is vulnerable to a crafted member path, so gate it.
Also update the stale _FakeExecuteTool in test_permission_mode.py to accept
the thread_id keyword that run_safetensors_tool_loop now forwards to
execute_tool after the main merge, which had broken the five tool-loop tests.
Adds regression rows covering each gap plus benign controls.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: normalize unknown permission_mode to 'ask' instead of a 422
The request models validated permission_mode with Literal[ask, auto, off,
full], so an unrecognized value from a newer UI/client was rejected with a 422
before the tool loops could apply their unknown -> ask fallback
(safetensors_agentic.py:464, llama_cpp.py:9001). That made the intended
forward-compat degradation unreachable at the API boundary for both Chat
Completions and the analogous Anthropic field.
Accept a plain string on both ChatCompletionRequest and AnthropicMessagesRequest
and normalize in a before-validator: None stays unset, the four known modes pass
through, and any other value degrades to the safest gate ('ask'), matching the
loops. Adds a regression test covering unknown/None/known across both models.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: close five more auto-mode classifier gaps
- terminal: xargs is no longer a safe wrapper. It appends arguments read from
stdin that the scan never sees, so `echo -o out /etc/passwd | xargs sort`
forwards to `sort -o out /etc/passwd` (a write + sensitive read) while only
the allow-listed literals are visible. Any xargs command now asks.
- terminal: ionice -p/-P/-u change the I/O priority of an already running
process / group / user instead of forwarding to a wrapped read-only command,
so `ionice -c 3 -p <pid>` now asks. ionice -c 3 <cmd> stays safe.
- MCP: gate ALTER SYSTEM, which persists PostgreSQL server configuration and was
not one of the DDL objects the mutation detector matched.
- MCP: a credential noun in a read-named tool (read_secret, list_tokens,
get_credentials, fetch_api_key) is a sensitive disclosure, so it asks even
without a mutating verb or a path/SQL argument. Scoped *_key nouns keep a
primary_key / keyboard lookup safe.
- render_html: no longer unconditionally safe. A static canvas still auto-runs,
but one whose HTML/JS reaches the network (fetch/WebSocket/remote script) asks,
since it can egress under the canvas CSP when artifact network access is on.
Its early provisional card is suppressed under the auto confirm gate, and the
confirm-without-stream guard now requires a stream when render_html is
selectable.
Adds regression rows and benign controls for each, and updates the render_html
provisional-card and confirm-gate tests to the new behavior.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: extend auto-mode gates for indirect file lists, dynamic lookups, HTML network loads, and Anthropic render_html
Follow-ups on the previous classifier round:
- terminal: wc/du/find --files0-from (and find's -files0-from primary) read a
NUL-separated list of input paths from a file, the same indirect mechanism as
sort --files0-from, so a crafted list reads arbitrary host files past the
literal path/root checks. Gate them like sort.
- python: a namespace lookup through a dict-style call (f =
__builtins__.__dict__.get('open'), globals().get('open'), vars(x).get(...))
can return open/eval/a mutator, so poison the bound name like getattr/subscript
lookups already are. An ordinary dict .get or os.environ.get stays safe.
- render_html: broaden the network detector so a canvas that loads a resource
via CSS url()/@import, srcset, or a root-relative (/path) or protocol-relative
(//host) src/href is treated as networked, not just fetch/WebSocket/remote
script. Relative ./x and url(#id)/data: refs stay static/safe.
- Anthropic /v1/messages: drop render_html from the unprompted-safe server-tool
set. Since it can prompt (networked canvas) and this channel invokes the loop
without confirm, selecting it under ask/auto/omitted now rejects like
terminal/python; off/full (or an explicit confirm opt-out) run it.
Adds regression rows and benign controls for each, plus an Anthropic route test.
* Studio: close six more auto-mode classifier gaps
- terminal: a glob that expands to a project .env (cat .e?v) now asks; .env
joins the sensitive glob-basename set, matching the literal-path gate.
- python: an open bound onto an attribute (box.f = open; box.f('out','w'))
is tracked by attribute name, and open invoked via .__call__
(open.__call__('out','w'), unwrapped to the underlying callable) is gated,
so neither slips past the name-based open-alias checks. Benign attribute
callables and .__call__ on non-writers stay safe.
- python: a namespace lookup via .get/.pop/.setdefault already covered the
builtins case; unchanged here.
- MCP: a mutating HTTP verb in a method/verb argument (get_url
{"method": "DELETE"|"POST"|"PUT"|"PATCH"}) now asks, so a generic HTTP
tool cannot mutate an external service unprompted; GET/HEAD stay safe.
- MCP: a credential/secret environment-variable value (get_env
{"name": "OPENAI_API_KEY"}) is treated as a sensitive read via the same
credential-noun match used for tool names; PATH/HOME stay safe.
- render_html: self-navigation sinks (location.assign/replace, window.open,
assigning a URL to (window.)location(.href)) join the network detector, so a
canvas that navigates itself to an external URL asks; location.reload() /
history.back() stay static.
Adds regression rows and benign controls for each.
* Studio: gate obfuscated canvas egress, sensitive-dir iteration, and MCP metadata-host reads
- render_html: strip block comments before the network scan so fetch/*x*/(...)
cannot hide egress, and match bracket-access forms (window['fetch'](...),
self['open'](...)). Line // comments are left alone so the // in an https URL
is not eaten. A comment-only canvas stays static.
- python: enumerating a directory outside the sandbox (Path('/etc').iterdir(),
os.scandir('/etc'), os.listdir('/home'), os.walk('/')) reads host filenames
the direct /etc/passwd checks would prompt for, so gate it when the target dir
folds to an absolute/tilde/sensitive path; a relative dir stays safe and an
unresolved dynamic dir is left to other checks.
- MCP: a read-named HTTP tool pointed at a cloud-metadata / link-local host
(fetch_url {"url": "http://169.254.169.254/..."}, metadata.google.internal)
reads instance credentials, so classify those URL arguments as sensitive,
mirroring the sandbox SSRF blocklist; ordinary and localhost URLs stay safe.
Adds regression rows and benign controls for each.
* Studio: gate meta-refresh navigation, pandas HTML/markdown exporters, absolute glob roots, and checksum verify mode
* Studio: gate starred open writes, builtins.__import__, computed render_html sinks, and procfs fd reads in auto mode
* Studio: gate remote worker canvases, huggingface_hub downloads, and write callables passed to user helpers in auto mode
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: offer the latest transformers release for brand-new architectures
When a model's config.json model_type is absent from every installed
transformers overlay (base 4.57.x and the .venv_t5_530/550/510 sidecars),
Studio now checks, unauthenticated and cached, whether the newest
transformers ships it:
- utils/transformers_latest.py fetches the latest release version from
https://pypi.org/pypi/transformers/json and the CONFIG_MAPPING_NAMES
sources for that tag and for main from raw.githubusercontent.com
(never api.github.com), parsing them with the same AST extractor the
static router uses (no code execution, no trust_remote_code). Results
are cached in memory and in a JSON snapshot under studio_root()/cache
with a one day ttl; fetches are bounded to 5s with one retry and a
failure backoff, and offline mode or the new kill switch
UNSLOTH_STUDIO_NO_LATEST_TRANSFORMERS=1 short-circuits to None.
- POST /api/inference/validate gains requires_transformers_upgrade plus
a transformers_upgrade payload (model_type, pypi_version,
supported_in_pypi, supported_in_main) so the frontend can raise the
install consent dialog before /load, mirroring the existing
remote-code consent flow. The check fires only when the model_type is
unknown to all installed overlays and the hardcoded tier tables.
- POST /api/inference/install-latest-transformers provisions a new
persistent .venv_t5_latest sidecar after user consent, pinned to the
exact PyPI version (re-verified server-side) with the same
--target/--no-deps recipe as the fixed sidecars. A JSON pin marker
inside the dir records the installed package set, so restarts
revalidate it and routing resolves the new highest-ranked tier
automatically. A dependency preflight (compat_plan) compares the
release's requires_dist against the running env: unsatisfied
tokenizers/safetensors floors are shadow-installed as exact pins into
the sidecar, anything else unsatisfied blocks the install with a
clear message.
Routing for every already-supported model_type is unchanged: the
hardcoded lists and the 530/550/510 static resolver run first, the new
tier only participates once its venv exists, and the probe order gains
the latest sidecar only when provisioned. Verified against live PyPI
and GitHub (transformers 5.13.0: 674 model_types, 26 absent from all
installed overlays, e.g. cosmos3_omni; 4 dev-only on main) and with a
real sidecar install plus restart persistence. 64 new tests; the
existing 200-test transformers_version suite passes unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Latest-transformers check: fetch outside the lock, serialize installs
Release the module lock during the network refresh so a slow fetch cannot
stall other threads in the ASGI pool; concurrent callers during a fetch get
None (the graceful fallthrough) via an in-flight flag instead of stacking
fetches. Serialize install_latest_transformers with an in-progress flag so
concurrent consents cannot race the sidecar delete and recreate; the loser
gets a structured already-in-progress refusal.
* Latest-transformers check: LoRA bases, pin-gated mapping, live reverify
Run the upgrade check over the [adapter, base] target set so a LoRA whose
base model is a brand-new architecture surfaces the prompt (the worker
activates transformers for the base, not the adapter).
Gate the latest overlay's mapping lookup on a valid pin marker, matching
activation and the probe order, so a partial or manual .venv_t5_latest dir
cannot be routed to and then refused at activation.
Re-verify the requested version against a live PyPI snapshot at install
time, falling back to the cached one on fetch failure, so a release
published inside the cache TTL is not silently missed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Latest-transformers check: nested config types and latest-tier vision probe
Collect every model_type in the config (top level plus each nested
sub-config) and signal on the first one missing from all installed
overlays, so a supported wrapper carrying a brand-new backbone still
surfaces the upgrade prompt; wrappers instantiate sub-configs through
CONFIG_MAPPING and would fail on the nested type.
Route the vision capability subprocess through the pinned latest sidecar
when the model resolves to the latest tier, so latest-only VLMs are not
misclassified as text-only; every other tier keeps the 5.5 sidecar used
today.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Latest tier: nested routing, vision probe after raw miss, safe upgrades
Route by every model_type in the config: a nested sub-config type can raise
the tier (wrappers instantiate sub-configs through CONFIG_MAPPING), so a
supported wrapper with a latest-only backbone routes to latest once
installed instead of staying on default. An unknown nested type never
vetoes; the primary type keeps its previous semantics. The collector is
shared with the upgrade checker.
Vision detection: when the raw heuristics say False for a model that routes
to the latest tier, run the AutoConfig subprocess under the pinned latest
sidecar instead of trusting heuristics built from older transformers.
Provisioning: stage-and-swap. Build the new sidecar in .venv_t5_latest.staging
and swap it in only when the install and pin marker are complete, so a failed
upgrade never destroys a previously working sidecar; restore the old dir if
the final swap fails.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Latest-transformers checker, vision subprocess, and cache fixes
Require the latest release to support every missing model_type (the
primary included) before prompting; a nested-only match cannot make the
model loadable, so no install is offered for it.
The vision-check subprocess now unions the active sidecar's own
registry mappings into the inlined parent-process detection sets, so
architectures only the sidecar knows classify correctly.
A successful sidecar install clears the tier probe cache, the latest
tier's model_type mapping, and the vision-detection cache so the new
venv takes effect without a restart. Tests for all three.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Aggregate upgrade support flags and keep install off /v1
The upgrade signal now reports supported_in_pypi only when the latest
release covers every missing model_type; a mix with a main-only nested
type surfaces as dev-only so no PyPI install is offered that would
still fail at load. The consented install endpoint moves to
studio_router so it is not reachable through the OpenAI-compatible /v1
mount. Tests for both.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor the latest-transformers kill switch in routing
With UNSLOTH_STUDIO_NO_LATEST_TRANSFORMERS set after the sidecar was
provisioned, the latest tier still joined mapping and probe routing
because only the pin was checked. Both admission points now also check
the kill switch, so operators can roll back a problematic sidecar
without deleting files.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Repair the latest sidecar through stage-and-swap
The lazy repair path installed into the live .venv_t5_latest, which
_ensure_venv_dir wipes first, so a failed repair deleted the pinned
sidecar and its marker. Both the consented install and the repair now
share one stage-and-swap helper: the incomplete-but-pinned dir survives
any failure and a later attempt can still repair it.
* Tighten comments
* Remove the staging dir when a latest-sidecar install fails
A pip failure inside _ensure_venv_dir returns False without raising, so
the except cleanup never ran and the partial .venv_t5_latest.staging
leaked until a later attempt. Also note on the validate response fields
that frontend consumption ships in the follow-up PR.
* Add the transformers-upgrade consent dialog to the frontend
When /validate reports requires_transformers_upgrade, every explicit load
path (chat runtime and the compare composer) now pauses on a consent
dialog modeled on the remote-code one: it names the model_type and the
latest PyPI transformers version, and on Accept calls
/api/inference/install-latest-transformers itself, shows an installing
state, and resumes the original load automatically on success. Errors
surface in the dialog with a retry; Cancel aborts the load like the
trust dialog's deny path. Architectures shipped only on transformers
main get a dev-only notice with no install button. Background auto-load
skips upgrade-requiring candidates instead of prompting, mirroring the
trust_remote_code rule. The dialog mounts once in the root layout and
runs before the security dialogs, since no load can proceed without the
runtime.
* Route a non-installable new architecture to the custom-code consent as a last resort
When the upgrade dialog has no installable PyPI release (the architecture
is only on transformers main, which Studio never installs), the dialog now
says so explicitly, and when the model also declares custom (auto_map)
code it offers Continue with custom code: resolving the paused load into
the existing trust_remote_code consent gate instead of hard-aborting.
Models with no custom code keep the Cancel-only notice. The backend
returns no upgrade signal at all for architectures unknown to both PyPI
and main, so those still route straight to the unchanged security gate.
* Force a 16-bit load for models on the latest-transformers sidecar
Live validation with Zyphra/ZAYA1-8B (model_type zaya, shipped by
transformers 5.13.1 but unknown to every installed tier) surfaced a
generation crash when the consented sidecar load kept the default bnb
4-bit quantization: transformers' grouped-MoE kernels feed the packed
uint8 expert weights straight into torch._grouped_mm, and generation
dies (plain 16-bit works). New latest_tier_active_for() mirrors the
sidecar activation's tier resolution and never raises; the inference
worker flips load_in_4bit off when it reports true, and the load route
applies the same flip so the pre-load VRAM guard and the worker command
agree. Fixed tiers are untouched. With the guard, ZAYA1-8B loads and
generates correctly in Studio chat.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Offer the custom-code fallback when a latest-sidecar install fails
* Fail remote mapping fetches wholesale and mirror the 16-bit flip in validate
A transient fetch or parse failure of one auto-mapping file no longer caches
a partial latest-release map for the TTL (a real 404 on pre-5.10 tags is
still tolerated), and validate_model now applies the same latest-sidecar
16-bit sizing flip as /load before the training guard so the two agree.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments in the latest-transformers changes
* Resolve remote LoRA bases, fold nested tiers, and guard the sidecar swap
latest_tier_active_for now resolves a remote adapter's base model the same
way worker pre-activation does (and returns early without a sidecar pin), a
hardcoded fast-path tier is raised when a nested sub-config's model_type
needs a higher sidecar, and the install route refuses to swap .venv_t5_latest
while training runs on it and unloads a latest-tier chat model first.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate the sidecar install on worker liveness and size installable upgrades 16-bit
The install route now refuses while any training or export runs (tier
re-resolution without the load token is unreliable for gated repos), holds
the inference lifecycle gate across the unload and the swap so no load can
interleave, and passes the model name to unload_model. validate_model runs
the upgrade check before the training guard and sizes an installable
upgrade as 16-bit, matching what /load and the worker will force after the
consented install.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close the sidecar install races and honor the kill switch over cached mappings
Training starts and mutating export routes now refuse while a transformers
install is in progress (shared is_install_in_progress flag), the chat unload
and idle export-worker teardown moved into a before_swap hook that runs only
once the staged install succeeded, and _config_model_types checks the kill
switch before returning a cached latest mapping.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reserve the sidecar swap before the gate wait and abort it on failed teardown
The install-in-progress flag moved into a shared sidecar swap reservation in
transformers_version, taken by the install route before awaiting the
inference lifecycle gate (so training and export starts see it for the whole
window) and by the lazy .venv_t5_latest repair path. The before_swap hook
now raises when the chat unload or export teardown reports failure, leaving
the previous sidecar untouched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Back the sidecar swap reservation with a cross-process lock file
The lazy repair runs inside worker subprocesses, where a module-level flag
is invisible to the parent's route checks. The reservation now also creates
a lock file next to .venv_t5_latest (O_EXCL, owner-only removal, stale after
two hours for crashed owners), so is_install_in_progress sees a repair from
any Studio process.
* Hand the swap reservation to the installer thread and harden pre-swap teardown
A cancelled install request no longer releases the reservation while the
installer thread is still staging (the thread owns and releases it, shielded
from cancellation). The route refuses while another inference request is
generating, export teardown runs before the chat unload and is judged by
worker liveness rather than the cleanup return value, and a live inference
worker with no active model (failed load residue) is shut down before the
swap.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the lifecycle gate with the installer and recheck the swap at spawn time
The gate moved into the shielded install task so a cancelled POST cannot
release the guard /load honors while the installer still runs, cached latest
probe results are ignored while the kill switch is set, and the training and
export subprocess spawns recheck the sidecar swap reservation right before
spawning (the route-level guards are one-shot and validation can outlast an
install's start).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close the spawn-registration windows against the sidecar install
Training marks the spawn in progress before its reservation recheck and
is_training_active honors the flag, so the install route sees a start that
has passed proc.start() but not yet recorded _proc. Export load-checkpoint
rechecks the reservation after setting _export_active and before tearing
down the old worker, so losing the race keeps the loaded checkpoint instead
of surfacing a 500.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Refine the install-window interleavings around worker teardown
The inference busy count is rechecked under the lifecycle gate (streams
start by taking that gate, so nothing slips past a held gate), the training
handshake moved ahead of the VRAM-freeing before_spawn hook so a lost race
leaves chat/export intact, the export spawn-time check is op-aware (inside
an active op the install is the side that aborts), and the Xet-stall respawn
waits out a transient reservation instead of stranding the run.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Track the install's server-side unload and guard export ops against the swap
The upgrade dialog store records when its install actually ran (the server
unloads the active chat model before swapping), and the load flow then marks
the previous model as unloaded so a later cancelled gate still triggers
rollback; the custom-code fallback leaves the flag unset. _run_export gained
the same reservation handshake as load_checkpoint so an install cannot block
behind an hours-long export op instead of returning 409.
* Tighten comments in the install-guard and upgrade-consent changes
* Surface install-race refusals cleanly and roll back after a failed swap unload
/load refuses while the sidecar swap is reserved so a load cannot succeed
and immediately be unloaded by the pre-swap teardown, worker starts that
lose the install race raise a typed SidecarSwapInProgress mapped to 409
instead of a 500, the install response reports model_unloaded even on a
structured failure so the client can restore its state, and the compare
flow tracks the server-side unload like the primary load path and clears a
stale checkpoint on abort.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Type the export install races, scope the lock release, and keep the unload signal
Export load-checkpoint and export ops raise SidecarSwapInProgress (mapped to
409 in every export route) instead of a 400-shaped failure, the export spawn
check distinguishes repair reservations (always refused) from install ones
(op-aware), the swap lock release only unlinks a lock this process wrote so
a stale-superseded owner cannot drop the new owner's live lock, and the
frontend unload signal survives a superseding consent via read-and-clear
consumption instead of a reset.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Finalize a stalled run when the respawn loses the install race and latch the unload signal
The Xet-stall respawn timeout now finalizes the run as a failure instead of
raising into the pump's broad finalization catch (which stranded it in a
training state with no worker), and a successful install retry ORs the
model_unloaded signal with the latched value so a failed-after-unload first
attempt still triggers rollback.
* Recheck the swap under the load gate and latch the unload before resolver checks
/load rechecks the sidecar reservation after acquiring the lifecycle gate
(an install can reserve while the load queues on it), and the dialog store
latches model_unloaded as soon as the install response arrives, before any
resolver-identity guard, so a superseded consent's unload still reaches
whichever load consumes the signal next.
* Report cleared-state unload failures, guard queued installs, and fold name tiers
A failed chat unload that still cleared the orchestrator's model state now
reports model_unloaded so the client rolls back, the installer aborts with
a 409 when a model load completed while it waited on the lifecycle gate,
and the fixed-tier name fast path consults the config mapping when a latest
sidecar is pinned so an accepted upgrade routes to the sidecar it installed
(no I/O added to the unpinned path).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Report cleared-state unload failures and harden the spawn handshake flag
The failed-unload branch in before_swap now detects that the orchestrator
cleared its model state and reports model_unloaded before aborting (the
earlier commit claimed this fix but a scripting error dropped the edit),
the installer's queued-load check compares a load generation counter so a
same-model reload is caught, and both training spawn sites wrap everything
after the handshake in a guard that resets _spawn_in_progress on any
exception so a failed start cannot wedge is_training_active.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Bump the load generation when the load is published, not at load start
A start-time bump is already visible when the installer snapshots mid-load,
so a same-model reload completing after the snapshot looked unchanged and
could be unloaded by the swap. The counter now increments alongside the
active_model_name publish.
* Self-heal a broken pinned sidecar, guard lazy repairs, and refresh stale retries
A valid pin whose transformers source dir vanished now triggers the repair
from the routing path (with a five minute backoff after failures) instead of
silently routing latest-only models to older tiers, the lazy repair refuses
while parent-visible chat/training/export workers are active since it has no
teardown of its own, and a version-mismatch install failure carries the
superseding release so the dialog's Retry re-requests a version that can
succeed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Flip latest-tier loads to 16-bit outside chat and protect export state
Training and export workers now apply the same latest-sidecar 16-bit flip
as the chat worker so a brand-new grouped-MoE architecture cannot reach bnb
4-bit through those paths, the latest-tier vision override returns None on
an inconclusive probe so a transient failure is not cached as not-vision,
and the install route refuses while an idle export checkpoint is loaded
rather than discard it with no rollback signal on a failed swap.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address parallel-review findings on the sidecar guards and install checks
The training route sizes latest-tier jobs 16-bit before GPU selection, the
inference subprocess spawn rechecks the swap reservation like training and
export (covering the OpenAI auto-switch path) with the typed error mapped
to a retryable 409, compat_plan blocks the install when dependency metadata
cannot be fetched instead of proceeding unverified, snapshot model-type
lists must contain only strings, and pin-marker package specs are validated
against the sidecar's own package set before ever reaching pip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Parent-only repairs, live-owner locks, remote-base activation, pre-teardown recheck
Lazy sidecar repairs now refuse inside worker children (whose empty backend
singletons cannot see live siblings) and run only in the parent where the
active-worker guard is real, swap-lock staleness requires the owner pid to
be dead so a slow live install is never superseded, both activation entry
points resolve a remote adapter's base model like the inference worker and
latest_tier_active_for already do, and load_model rechecks the reservation
before tearing down the old worker so losing the race keeps the current
model loaded.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Check workers under the repair reservation and keep state on refused swaps
The lazy repair now reserves first and checks workers under the reservation
(worker starts set their active markers before rechecking, so every
interleaving aborts one side), with export ops and in-flight inference loads
counted as active. The inference pre-teardown and spawn guards refuse only
repair reservations since an install shares the load's lifecycle gate and
aborts via its queued-load snapshot, a SidecarSwapInProgress raised before
teardown no longer clears the live model mirrors, and an export spawn abort
after teardown clears current_checkpoint so the page cannot claim a loaded
checkpoint with no worker.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Repair a present-but-incomplete latest sidecar from routing
The routing self-heal only fired when the pinned sidecar's transformers/
dir was missing. A sidecar that kept transformers/ but lost another pinned
package still routed models to the latest tier, and workers refuse
parent-only repairs, so every load failed until a manual reinstall. Routing
now validates the full pin (via _venv_dir_is_valid) and repairs any
incomplete sidecar under the same swap reservation and 5-minute backoff.
* Treat an unrepaired latest sidecar as unavailable in routing
When the pinned sidecar is incomplete and the lazy repair fails (offline,
pip failure, workers active) or is inside the backoff window, routing
returned the source dir anyway, sending models to a tier whose worker
activation is known to fail. Return None instead so models an older tier
supports keep loading there until a repair succeeds, matching the behavior
when the sidecar dir is missing entirely.
* Harden sidecar swap and repair against crash, survivor, and 16-bit paths
Reclaim a swap lock as soon as its recorded owner PID is dead instead of
waiting out the two-hour cutoff, so a crash mid-install no longer wedges
/load, training, export, and repair for hours. A lock whose PID cannot be
read yet still uses the long cutoff so the create-before-write window is
never mistaken for dead.
Probe process liveness with OpenProcess on Windows: os.kill(pid, 0) there
is CTRL_C_EVENT (a real Ctrl+C via GenerateConsoleCtrlEvent), not a
harmless check, and psutil is not always present.
Return whether _shutdown_subprocess actually killed the worker and keep the
live handle when it survives terminate/kill (an uninterruptible CUDA
syscall can outlive SIGKILL). The pre-swap liveness guard now trusts that
result, so the destructive .venv_t5_latest rename cannot proceed while a
live worker still holds sidecar modules.
Recover a sidecar stranded at .old when a swap's activation rename and its
rollback both fail: reading the pin restores it when no swap holds the
reservation, so latest-tier models are not permanently broken.
Resolve the latest tier in the parent for export loads and for explicitly
16-bit training runs, not only 4-bit ones: tier resolution self-heals an
incomplete sidecar, and repairs are parent-only, so those paths could not
recover before. Sidecar integrity and quantization are independent.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Revert the parent-side latest-tier repair probe on training and export loads
The probe ran before the route freed VRAM, so a resident chat or export worker
made _workers_active_for_repair() refuse the parent-only repair; the route then
tore that worker down and spawned a child that also cannot repair, so an
incomplete sidecar still failed to load. Repairing correctly requires running the
repair between the worker teardown and the child spawn, decoupled from VRAM
sizing, which is a larger change tracked separately. Restore the prior behavior
so these paths match the reviewed form and do not partially attempt a repair that
cannot complete while workers are resident.
* Honor failed worker shutdowns on load and revalidate the cached latest mapping
The fresh-load paths spawned a new worker straight after _shutdown_subprocess
without checking its result, so a worker that outlived terminate/kill (a wedged
CUDA syscall) had its handle overwritten by the replacement while it still held
GPU memory, and is_worker_alive/the pre-swap guard could no longer see it. Both
the inference load and the export checkpoint load now abort when the old worker
did not exit, so the load can be retried once it does.
_config_model_types returned a cached latest mapping without re-checking the
sidecar, so a sidecar deleted or broken in-process after its first parse was
never re-validated: routing kept sending latest-only models to the stale latest
tier while activation failed. The cached latest mapping is now dropped and
re-resolved (self-healing) when the sidecar is no longer intact.
* Drop cached latest mapping when the pin is gone; keep 4-bit for custom-code fallback
_latest_sidecar_intact now returns False when the pin marker itself is gone, not
just when a pinned package is missing. Otherwise a cached latest mapping outlived
a deleted pin: _config_model_types kept returning it, so routing sent latest-only
models to a tier whose worker activation then failed (no pinned version) until
restart. It now drops the cache and re-resolves to no latest tier. The
_overlay_transformers_dir caller already gates on a present pin, so it is
unaffected.
validate_model forced 16-bit sizing whenever a PyPI upgrade was merely offered,
even for a model that can fall back to its own auto_map code. /load loads such a
model 4-bit without the install, and the install route refuses while training is
active, so 16-bit sizing here returned a VRAM 409 for the only viable 4-bit path.
The offered-upgrade flip is now gated on the absence of a custom-code fallback;
an already-active latest sidecar still always sizes 16-bit.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: expose Windows drive roots in the folder browser
The model-selection folder browser bounds navigation to the roots returned
by _build_browse_allowlist(), which exposed Linux removable-media mounts via
linux_run_media_mount_roots() but had no Windows analog. As a result a user
on C: could not browse to D:/E: to pick a model directory.
Add windows_drive_roots(), a Windows-only companion to
linux_run_media_mount_roots() that lists readable logical drive roots, and
wire it into both browse-allowlist builders and their suggestion chips so
other drives are both navigable and offered as quick-picks. The helper is a
no-op on Linux/macOS, so existing platforms are unaffected.
Closes#6368
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: cover the Windows drive-root browse wiring with an integration test
Add an allowlist integration test mirroring the Linux side's
test_legacy_browse_allowlist_includes_linux_run_media_mounts: it extracts
_build_browse_allowlist from routes/models.py, stubs external_media so
windows_drive_roots() yields a fake drive root, and asserts that root becomes
browsable through the built allowlist. Proves the wiring, not just the helper.
* Studio: skip inactive drives via GetLogicalDrives before probing
Resolve active logical drives from GetLogicalDrives() before probing each
letter with os.path.isdir. Probing a drive letter mapped to a disconnected
network share can otherwise block the async backend for tens of seconds per
letter. The call degrades gracefully (falls back to probing all letters) when
ctypes/windll is unavailable, so behavior is unchanged on Linux/macOS. Tests
override the bitmask source to stay deterministic on real Windows hosts.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: allow browsing descendants of a drive-root allowlist entry
routes/models.py _is_path_inside_allowlist() checked descendants with
startswith(root_real + os.sep). A drive root ("D:\") already ends in a
separator, so the prefix became "D:\\" and a child like "D:\models" was
rejected with 403 after the browser opened the drive root. Only append a
separator when the root does not already end in one. folder_browser.py already
uses commonpath and was unaffected. Adds a regression test covering the
separator-terminated-root descendant case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: enforce the system-directory denylist during folder browsing
Exposing whole Windows drive roots (and any legacy-registered filesystem root)
widened the browse allowlist above system directories, but the browse
resolvers only re-applied the credential/config denylist, not the
_denied_path_prefixes() system-dir denylist that scan-folder registration
enforces. That let browse-folders enumerate C:\Windows, C:\Program Files,
/etc and /proc.
- Add is_denied_system_path() to both storage modules and enforce it in both
browse resolvers (legacy routes/models.py and hub folder_browser.py), on each
resolved child and on the final target, keeping the /run/media carve-out.
- Rework the legacy _is_path_inside_allowlist to use splitdrive + commonpath so
a Windows drive root authorizes its descendants while a bare POSIX / does not,
and to compare case-insensitively like the hub browser.
- Reject the filesystem root in the legacy add_scan_folder, matching the hub.
- Hide denied system dirs from browse listings and suggestion chips.
- Add tests/test_browse_denylist.py and update the external-media path tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make browse-denylist tests OS-portable
The browse-time denylist tests used real /etc and tmp_path locations; on macOS
tmp lives under the (legitimately denied) /private/var and /etc resolves to
/private/etc, so three tests failed there. Pin the platform / use a tmp-based
denied prefix so they assert the same behavior on Linux, macOS and Windows.
* Studio: apply the bare POSIX-root guard to the hub folder browser too
The _is_path_inside_allowlist guard that stops a legacy-registered '/' scan
folder from authorizing every absolute path lived only in the legacy browser.
The hub browser used commonpath without it, so a stale '/' row let it descend
into /var, /root, /home -- which the system-directory denylist (/proc /sys /dev
/etc /boot /run) does not cover, while the legacy browser blocked them. Mirror
the legacy guard so both browsers treat '/' identically.
Also resolve each directory entry before the denylist check in both listing
loops, so a symlink or junction pointing into a denied dir is hidden instead of
rendered as a row that 403s on descent. Adds legacy-vs-hub parity tests.
* Studio: bound Windows drive probing so a disconnected mapping can't stall the browser
GetLogicalDrives includes mapped network drives, so a disconnected but still
mapped drive (e.g. Z: -> \\nas\share) stays set in the bitmask and reaches
os.path.isdir, which can block for tens of seconds while Windows tries to
reconnect. Because windows_drive_roots() runs synchronously while building both
folder-browser responses, one stale mapping stalled every browse request.
Probe each surviving drive in a daemon thread bounded by a short timeout and
skip it if it does not answer in time, so a hung mapping is dropped instead of
blocking the caller. Connected drives (local or network) still respond well
within the timeout, so drive discovery is unchanged. Corrects the
GetLogicalDrives docstring, which claimed the bitmask alone prevented the stall.
* Studio: probe drive/media roots once per browse request, not twice
Both folder browsers called windows_drive_roots() (and
linux_run_media_mount_roots()) twice per browse request: once to seed the
allowlist in _build_browse_allowlist() and again to build the suggestion chips.
With the bounded drive probe, a disconnected mapped network drive then paid the
timeout twice per folder click. Probe both once in the request handler and pass
the results into _build_browse_allowlist(), reusing them for the chips, in both
the legacy and hub browsers. Adds a test asserting the roots are reused, not
re-probed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: run the legacy browse endpoint in the threadpool, fix its stale test
Two follow-ups from review of the drive-probe changes:
- browse_folders was 'async def' but does only blocking filesystem I/O (the
timeout-bounded drive probe, iterdir, realpath). On the event loop a
disconnected mapped drive waiting out its probe timeout stalled every other
request. Declare it sync 'def' so FastAPI runs it in the threadpool, matching
the hub browse endpoint. No await was used in the body.
- test_browse_folders_hides_sensitive_dirs monkeypatched _build_browse_allowlist
with a zero-arg lambda; the once-per-request refactor now calls it with
(media_roots, drive_roots), so the lambda raised TypeError. Accept and ignore
the args.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: probe Windows drive roots concurrently so multiple dead mappings don't stack timeouts
windows_drive_roots() probed each candidate serially, so N disconnected-but-mapped network drives each paid the full per-drive timeout in turn (e.g. four stale mappings added ~8s to every folder-browser request). Collect the candidate roots first, then probe them all at once under a single overall deadline, so the added delay stays at ~one timeout regardless of how many drives are disconnected. _readable_dir_within stays as a thin single-path wrapper for its existing callers/tests.
* Studio: tighten comments in the folder-browser drive-root changes
Condense the comments and docstrings added by the Windows drive-root and
system-directory denylist work to be shorter and clearer while keeping the
security and correctness rationale intact. Comment and docstring text only;
no code changes.
* Studio: iterate the input, not the results dict, when collecting readable drive probes
_readable_dirs_within returned {path for path, ok in results.items()...}, but a probe thread that exceeded the join deadline is still alive and can insert its key into results during that iteration, raising 'dictionary changed size during iteration' -- reachable exactly in the disconnected-mapped-drive case the probe exists for. Iterate the fixed input list and read results.get(path) (an atomic read) instead.
* Studio: keep the browse-route containment tests denylist-inert so they pass on macOS
test_browse_folders_route.py exercises allowlist containment and the file-vs-directory guard, not the system-directory denylist. On macOS pytest tmp_path resolves under /private/var, a denied prefix, so _resolve_browse_target 403s the fixture dirs before the containment logic runs (4 failures). Add an autouse fixture that makes is_denied_system_path inert in this file; the denylist keeps its own coverage in test_browse_denylist.py.
* Studio: keep the hub browse tests denylist-inert so they pass on macOS
* Studio: register a UNC share root; only reject local filesystem roots
* Studio: reject device drive roots and browse a registered UNC share root
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: treat device-namespace volume GUID roots as local filesystem roots
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Unsloth: appearance palettes, customization options, and control restyle
Adds Standard, Classic, and Minimal color palettes to Appearance settings,
each adapting to light and dark mode. Classic is a neutral enterprise look
that reserves its blue accent for toggles, badges, and focus rings; Minimal
is strictly black, grey, and white.
Adds customization options scoped to the active mode: accent, background,
and foreground colors with an in-app color picker, UI and code fonts with a
searchable dropdown covering bundled, device, and imported fonts, font file
import, UI and code font sizes, contrast, pointer cursors, reduce motion,
font smoothing, and translucent sidebar. Settings persist through the
personalization API with backend validation and sync across devices.
Restyles core controls for a cleaner, flatter look in both modes: bordered
white input fields, fully rounded pills for single-row controls, no drop
shadows, simple straight-line chevrons replacing all rounded arrow icons,
and consistent hover tones in dropdown menus. Popovers now portal into the
open dialog so their lists scroll correctly inside modal dialogs.
Moves Language into General settings and Chat defaults into the Chat tab
above the Canvas section.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Unsloth: appearance follow-ups, font options, and settings search
Neutralizes focus and selection rings across all palettes so highlighted
elements, including typing boxes and the selected palette card, never take
the accent color. The custom accent no longer recolors rings.
Restyles the color controls as filled pills showing the hex value inside,
with text and border contrast picked from the color's luminance. Menus in
popovers now match the app's dropdown menus: rounded-lg corners, tighter
padding, accent hover rows, and a bordered search field. Popovers inside
modal dialogs are modal so their lists scroll with the wheel. Outline
buttons share the same dark fills as dropdown triggers.
Adds heading and chat font options next to the UI and code fonts, each
using the searchable font dropdown and persisting through the
personalization API. Removes the translucent sidebar option end to end.
Adds settings search: a search field at the top of the settings sidebar
that filters setting names across every tab, grouped by tab with icons,
and jumps to the tab on click.
* Unsloth: use the shared accent token for dark hover fills
The settings dialog nav, its close button, the model selector, and the
project switcher hovered with hardcoded blue tinted greys (#3a3d43,
#2d2e32) in dark mode while every menu and sidebar uses --accent. All
hover and active pill fills now use the accent token so dark hovers are
the same everywhere and adapt to the active palette.
* Unsloth: settings search polish and jump to matched setting
Widens the settings dialog to 880px and the sidebar column to 248px so
the search field has more room. The search pill aligns with the left
start of the Settings title, gets more spacing above and below, and its
icon and placeholder sit slightly further left.
Search results now jump to the exact setting: rows and sections expose
their label as a data attribute, and picking a result opens the tab,
scrolls the matched row into view, and flashes it briefly.
* Unsloth: settings search bar spans the full nav pill width
The search field now starts and ends at the same edges as the nav hover
pills instead of being inset to the title text.
* Unsloth: address review findings on motion, sync, and font limits
Reduce motion Off now opts back out of the OS reduced-motion preference
for CSS animations via a force-motion class that the media rules skip,
and forcing reduce motion On keeps the loader exceptions (spinners,
loading dots, progress bars) animating.
When the color scheme follows the system, the resolved mode is now part
of the theme store snapshot, so an OS scheme flip re-renders consumers
and reapplies per-mode custom colors instead of leaving stale inline
variables from the previous mode.
Imported fonts get an aggregate size cap (4.4M characters) on both the
frontend sanitizer and the backend model so the persisted store always
fits browser localStorage quotas, with a clear error toast when an
import would exceed it. Backend validation also tightens imported font
names (rejects CSS delimiter characters) and requires strict base64
font data URLs, matching the frontend patterns.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Unsloth: profile toggle to hide the sloth in the chat greeting
Adds a Show greeting sloth switch to Settings > Profile. The chat welcome
hides the mascot when it is off. The preference persists locally and
through the personalization API, with backend validation and tests, and
the row is reachable from settings search in all four locales.
* Unsloth: control restyle, dropdown scrolling, and palette consistency
Settings sidebar puts search on top with the tab list under a small
Settings label. Combobox popups scroll with the wheel inside dialogs by
falling back to manual list scrolling while a dialog scroll lock is
active, and the local model selector popover became modal for the same
reason. Number inputs swap native spinners for a shared grey stepper
that clamps to min, max, and step. Run settings fields in light mode use
the same white fill and border as the settings dialog. Selection and
focus rings derive from each palette's border color instead of near
black, hover borders soften the same way, the Classic sidebar stays
white like Standard, decorative greens follow the palette accent, and
meaning-carrying marks like the hub verified badge keep the brand green
in every palette.
* Unsloth: palette card selection keyed off the palette attribute
Switching palettes restyles the whole page the moment data-palette lands
on the html element, but the React re-render that moves the selection
classes arrives later, so the ring and check briefly stayed on the
previous card with the new palette's colors. The active ring and check
now key off html[data-palette] in CSS, so they swap in the same style
pass that swaps the tokens. Also adds breathing room around the settings
search bar and under the Settings label, shortens the greeting sloth
description, and renames the avatar section to Or pick a sloth profile
picture in all locales.
* Unsloth: restore neutral rings, drop the palette check, sidebar spacing
Puts the ring tokens back to their fixed per palette values and removes
the hover border darkening, undoing the derived border experiment. The
selected palette card no longer shows a check since the ring already
marks it. The settings sidebar search bar, nav pills, and search results
get a little side padding, and the Settings label lines up with the pill
text.
* Unsloth: indicator restyle, sidebar menu customization, edge fade toggle
- Derive focus and selection rings from the border color so indicators
stay 1px and adapt to every theme and palette
- Suppress mouse focus rings except on pressed controls to remove the
selection flash on the avatar and palette pickers
- Defer settings panel rendering so the active nav pill updates instantly
- Customizable sidebar user menu with drag to reorder and shortcuts to
the settings tabs
- Grey hover for the standard light palette instead of green
- Borderless controls in dark mode with fill based focus states
- Profile picture: no picture option, pencil edit icon, atomic selection
- Font dropdowns: narrower triggers and the resolved default shown as
Inter Variable (Default)
- System prompt border darkens on focus
- New appearance setting to swap edge fades for thin divider lines
- Move the theme bootstrap to an external script to satisfy CSP
* Unsloth: harden theme boot and Firefox scroll container focus
- Guard the theme and palette storage reads separately so a blocked
localStorage (private browsing) still resolves a mode from the OS
preference instead of skipping the boot entirely
- Firefox makes scrollable containers keyboard focusable and drew its
3px UA outline on them; swap it for the app's soft 1px indicator
* Unsloth: make the UI and code font settings reach the font utilities
The theme block declared the sans and mono stacks as literals, so
Tailwind inlined them into every font-sans and font-mono utility at
build time and the runtime overrides from Settings > Appearance never
applied. Reference the :root tokens instead, matching how the color
tokens already work.
* Unsloth: in-dropdown font upload, accent meters and avatar, naming cleanup
- Move font importing into each font dropdown: Upload and Select folder
sit side by side under the list, imported fonts get an inline remove,
and the standalone Import font row is gone
- Uploads reuse fonts the user already has (bundled, imported, or
installed, matched by file name with style suffixes stripped) instead
of embedding a duplicate copy; only new fonts are embedded
- Folder scan lists font files from a picked folder in every dropdown
for the session; picking one imports it through the same path
- Fallback avatar uses the control accent with a readable foreground
instead of the neutral primary that rendered black outside standard
- Monitor bars, progress defaults, sliders, and usage meters use the
control accent; warning and danger tiers stay amber and red
- User facing strings that called the app just Studio now say Unsloth
in all four locales, keeping Unsloth Studio and LM Studio intact
* Unsloth: left align the font upload actions and divide them
Upload and Select folder now read from the left like the list items,
with a short vertical rule between the two.
* Unsloth: keep sliders neutral and the chat greeting on Hellix
- Sliders are controls, not meters, so their fill goes back to the
neutral primary instead of the palette accent
- The base h1 rule reads --font-heading with !important and the chat
thread root resets that variable to the sans stack, which pulled the
greeting off Hellix; restore the stack on the greeting element
* Unsloth: move the None avatar cell last and keep footer actions on one line
- None sits after the sloth pictures instead of leading the grid
- Upload shrinks to its label so Select folder no longer wraps
* Unsloth: size the folder action to its label
Both footer actions now hug their content so the hover pill does not
stretch across the leftover row width.
* Unsloth: separators only between unrelated settings clusters
Rows inside a titled section are related, so the per row divide-y is
gone from SettingsSection. A SettingsGroupDivider marks the two real
boundaries in the theme section (colors to fonts, fonts to contrast)
and the Clear all chats row gets its destructive border back now that
divide-y no longer draws one for it.
* Unsloth: balance the two font upload actions
Both actions share the footer row evenly again; nowrap keeps Select
folder on one line at the narrower width.
* Unsloth: drop the theme section dividers and split the chat menu groups
The colors, fonts, and contrast rows read fine without rules, and the
chat menu gains its one real boundary between the pin toggles and the
disclaimer rows.
* Unsloth: normalize oversized sidebar menus and reject newline font data URLs
Two backend validation fixes in PersonalizationCustomization:
- sidebarMenu refused any list longer than the number of distinct ids
because Field(max_length) is enforced before the dedupe validator runs.
A stale or duplicated payload that would normalize to one entry per id
was rejected outright, defeating the normalizer that exists for exactly
that case. Cap the incoming list at a generous multiple so it reaches
the validator; a pathologically long list is still refused.
- The imported font dataUrl validator used re.match on a pattern ending
in $, which also matches just before a trailing newline, so
"data:font/woff2;base64,AAAA\n" passed even though the frontend JS
pattern rejects it. Use re.fullmatch for parity.
Adds covering tests for both.
* Unsloth: preview fonts in their own typeface and slim the color pills
- Every font dropdown entry, the default item, and the closed trigger
render in the font they name, falling back to the UI stack for
families the browser cannot resolve
- Color swatch pills drop from 36px to 28px so they sit closer to the
row label height
* Unsloth: drop the font row and theme section descriptions
The labels carry the meaning on their own; the mode switching note in
particular read long and confusing.
* Unsloth: let the chat greeting follow the heading font setting
The greeting stays on Hellix by default but adopts a chosen heading
font through a --custom-heading-font variable the applier sets only
while an override exists, so the thread root's sans reset for chat
prose no longer hides the user's pick from the greeting.
* Unsloth: divide the theme section clusters and align the color pill height
Separators return between colors and fonts and between fonts and
contrast, and the color pills share the 32px height of the font
dropdown triggers.
* Unsloth: color pills at half the dropdown width
Fixed w-24 against the w-48 font triggers, with tighter padding so the
hex value still fits.
* Studio: update dep-removal test after next-themes was replaced
The frontend no longer declares next-themes or imports it in src (it was
replaced by the custom theme store and boot script), so the checker now
reports its removal as a safe no-op. The C1 and C8 fixtures in
test_frontend_dep_removal.py still asserted next-themes was a used
dependency, which fails the studio frontend CI dependency-removal safety
check. Update C1 to expect a no-op PASS and drop next-themes from the C8
expected failures so the suite matches the checker's correct output.
* Studio: remove unused ageLabel and exportCollectionJsonl helpers
* Studio: fix blocked-storage theme desync, search jump race, font validation
- theme-store.ts: keep an in-memory currentTheme/currentPalette so a selected
value survives when localStorage is blocked (private browsing). The snapshots
previously re-read empty storage and reverted React state to the default while
the DOM already changed. The matchMedia handler no longer re-reads storage, so
it cannot clobber the in-memory choice; cross-tab storage events still adopt.
- settings-dialog.tsx: the search jump waited a single fixed 60ms for the
deferred tab panel to render, then silently missed under render lag. Retry
across animation frames until the target row exists, then scroll and flash.
- settings.py: apply the font-name character check to the four selected-font
fields (uiFont/headingFont/chatFont/codeFont), and forbid backslash, comma,
slash and control characters so a name cannot escape the quoted CSS
font-family or smuggle extra fallbacks. Adds covering tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix appearance customization edge cases for PR #7077
- Reset all local preferences now also clears palette and appearance customization
- Number input wrapper keeps full width so fields fill their flex/grid cell, and the stepper stays pinned to the field edge
- Number stepper snaps to the min anchored step grid like the native spinner instead of leaving a step-invalid value
- Code font now applies to chat code fences and inline code via a dedicated token
- Reduce motion (on/off) is honored by onboarding/tour confetti and the theme toggle view transition
- Re-importing a font under the same name with new bytes now swaps the FontFace
- Keep local customization when a synced record predates the customization field, and re-push it
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Align client font name sanitization with server validation for PR #7077
sanitizeFont now strips the same characters the backend _FONT_NAME_FORBIDDEN
rejects (backslash, slash, comma, backtick) plus control chars, so a locally
chosen font name can no longer pass the client but fail the personalization PUT
and silently stall appearance sync.
* Address follow-up review items for PR #7077
- Number input wrapper carries React Flow interaction classes (nodrag/nopan/nowheel) so clicking the stepper arrows increments instead of dragging the node
- Preserve local palette and greeting-sloth toggle when the synced record predates those fields, and re-push them, mirroring the customization handling (new paletteSaved and greetingSlothSaved response flags)
- Add settings-search scroll targets (data-settings-label) for the Profile title, description, display name, nickname, and avatar shape rows
* Preserve absent personalization fields on PUT for PR #7077
A stale client that omits palette or customization previously had those
defaults materialized by model_dump() and persisted, which flipped
paletteSaved/customizationSaved to true and defeated the legacy detection.
The PUT now dumps only the request's set fields and merges them onto the
stored record, so omitted fields keep whatever was already stored.
* Persist theme and palette via a fixed allow-list for PR #7077
The theme/palette values reach setTheme/setPalette from the authenticated
personalization sync, which made the CodeQL clear-text-storage query treat
writing them to localStorage as storing sensitive data. Store a re-derived
literal from a constant map instead, so a plain UI preference is not tracked
as sensitive; behavior is unchanged.
* Harden imported-font handling for PR #7077
- syncImportedFonts: a rejected FontFace.load() only clears the registry entry
if it still points at that face, so a same-name re-import while the old load
was pending is no longer untracked/leaked.
- Cap imported-font names to the backend length (100) so an over-long name can
no longer pass the client but fail the personalization PUT and stall sync.
- Add a backend test that a stale PUT preserves an existing stored palette and
customization (not just that absent fields stay absent).
* Return the merged personalization record from PUT
The PUT /personalization handler returned the request payload, which
Pydantic had already filled with defaults for any field the client
omitted. A partial or stale write (for example a client sending only
theme) therefore got back a response that contradicted both storage and
the next GET: preserved fields like palette and the custom font showed
their defaults instead of the stored values.
Return model_validate(merged) so the response mirrors what was stored.
The stored record is still the full merged dict, so legacy fields the
model does not know about are preserved as before.
* Fix small UI and keyboard-focus defects in appearance settings
- Settings search now scrolls to the result within its destination tab
instead of a same-named row in the previously rendered deferred tab
(for example "Storage" and "Models folder" appear in both General and
Resources).
- The reduce-motion segmented control honors its own Off/On/System choice
by reading useReducedMotionConfig instead of the OS-only useReducedMotion.
- The color picker saturation/value area is operable by keyboard, so the
role="slider" surface responds to the arrow keys it advertises.
- Profile avatars and palette cards show a visible keyboard focus ring
again.
- Guard the persisted appearance-customization write so a blocked or full
localStorage does not throw out of a store action, matching the theme
store.
- Import the appearance store symbols from the settings feature barrel.
* Tighten appearance fix comments
---------
Co-authored-by: shimmyshimmer <shimmyshimmer@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: persistent stdio MCP sessions so server state survives across tool calls
call_tool_sync spawned a fresh stdio subprocess per tool call
(keep_alive=False) and tore it down when the call returned, so any stateful
MCP server lost its state between calls: with @playwright/mcp,
browser_navigate opened the page in one subprocess and
browser_take_screenshot ran in a brand-new one, screenshotting about:blank.
Keep one connected client per (command, env) on a dedicated event-loop
thread and reuse it across calls:
- idle sessions are reaped after 5 minutes (in-flight calls excluded) and
everything closes at exit, preserving the old design's no-orphans property
- a dead subprocess is detected via is_connected() and retried once on a
fresh session; tool-level errors leave the session alone
- cancel and timeout semantics are unchanged, and a timed-out call does not
tear the session down
- updating a server's endpoint/env/enabled state or deleting it closes its
live session
- HTTP/SSE servers stay one-shot per call
* address review feedback
* fix stdio session cleanup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* address review: per-thread MCP scope, close-during-connect and abort races
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* address review: unblock no-limit calls on close, drain borrowers before close, scope closes to url+env
* don't retry sessions closed by config changes, re-verify server row before caching, keep env secrets out of generation keys
* fail fast on connect errors and make the stdio key-lock wait cancellable
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* quote MCP scope parts so IDs with colons can't collide
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* serialize per-session stdio calls, span one timeout budget across connect and call, hash urls in generation keys
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden persistent stdio MCP sessions: crash recovery, concurrency, scoping
- Evict a stdio session on any transport-level (non-ToolError) call failure and
do not replay it, so a mid-call subprocess crash can no longer poison the scope.
Never gate liveness on Client.is_connected() (it only reports that a session
object exists, not that the subprocess is alive); add a version-adaptive
dead-transport probe that works on fastmcp 3.0.2 and newer.
- Re-check closed/defunct/config and transport liveness after acquiring the call
lock, and retire a session before releasing the lock, so a queued same-scope
caller never reuses a session that another caller's timeout already retired.
- Force a ProactorEventLoop on Windows so the stdio transport can always spawn
subprocesses regardless of the active event-loop policy.
- Scope stdio sessions per conversation: require thread_id to persist, and tag
the fields so a session_id and a thread_id with the same value cannot collide.
A session_id alone is project-wide, so it now falls back to a safe one-shot
session instead of sharing browser/DB/REPL state across conversations.
- Forward thread_id on the Anthropic Messages path.
- Treat timeout=None as unlimited on connect and the key lock (was capped at 60s).
- Bound the session cache (default 32, override via
UNSLOTH_STUDIO_MAX_STDIO_MCP_SESSIONS) with LRU eviction of idle sessions.
- Run config_check on cache hits, and log a redacted exe#digest label instead of
the raw command so credentials in argv never reach the logs.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim the stdio MCP session cache on release and skip close-generation for HTTP servers
Two fixes from review of the persistent stdio session lifecycle:
- Re-enforce the session cap when a session goes idle. A concurrent burst of
distinct-scope calls can overshoot the cap while every cached session is busy
(insert-time eviction only reclaims idle sessions), and the overshoot used to
persist until the 5-minute idle reaper. _release_stdio_session now trims the
idle overshoot back within the cap, without ever evicting an in-flight call.
- close_stdio_sessions() now no-ops for a specific non-stdio (HTTP/SSE) url.
Those transports are never cached as stdio sessions, so calling it on every
HTTP server update or delete used to accrue an unbounded close-generation entry.
Both are covered by regression tests that fail before the change and pass after.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the live stdio MCP session across a display-name rename
The edit dialog resends url, headers, and use_oauth unchanged whenever a
server is saved, so gating the tool-cache invalidation and stdio session
close on field presence dropped the persistent process on a plain rename
or any no-op edit. Gate on a real value change against the stored row so
only a genuine endpoint, auth, or enable change closes the session.
Regression tests: a rename that resends unchanged url/headers/oauth keeps
the session; a real command change still closes it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments in the stdio MCP session lifecycle
Collapse a few verbose comments to fewer lines with the wording preserved,
and drop one that restated the clear_oauth_tokens_async docstring. Comments
only; no code change.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Studio: resolve the repo-root MTP drafter after the MTP/ GGUF rename
The Gemma 4 QAT GGUF repos renamed the higher-precision MTP/ subdir
copies from gemma-4-...-<quant>-MTP.gguf to mtp-gemma-4-...-<quant>.gguf,
so their basenames now start with the same mtp- prefix as the small
repo-root drafter (mtp-gemma-4-E4B-it.gguf).
The drafter selectors filtered candidates by a mtp- basename prefix and
took the first in sort order. With the new names the MTP/ copies also
match, and because MTP/ (uppercase) sorts before the lowercase root file,
selection flipped to the large BF16 copy under MTP/ instead of the root
drafter both functions document they should pick.
Restrict both selectors, and the companion byte estimate, to root-level
mtp-*.gguf so the MTP/ copies stay explicit-selection only:
- core/inference/llama_cpp.py _pick_mtp (loader auto-download)
- hub/utils/gguf_plan.py preferred_mtp_sibling (Hub variant plans)
- routes/inference.py _remote_gguf_companion_bytes (VRAM headroom)
Also reuse a drafter already in the local cache before downloading, so a
device that already holds a copy on disk does not re-fetch it.
Old-scheme names keep working (they have no root-level mtp- sibling to
mis-select). Adds regression tests for the new naming, both selection
paths, and the on-disk reuse.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate MTP drafter cache reuse to offline mode
Reuse the cached drafter only when HF is offline. Online, route back
through _download_companion_gguf/hf_hub_download so the current revision
is checked (etag) and a changed drafter is refetched, matching the
offline-only cross-snapshot reuse already used for the main GGUF. This
avoids pairing freshly downloaded weights with a stale cached draft.
Make the reuse tests offline and add an online-skips-reuse test.
* Studio: prefer a root MTP drafter across all cached snapshots
Offline reuse scanned snapshots one at a time and returned the first
snapshot that held any drafter, only preferring root within it. A newer
partial snapshot with just the MTP/ copy could shadow the small root
drafter in an older snapshot. Collect drafters across all snapshots and
prefer any repo-root file before an MTP/ copy.
* Studio: keep newest-first snapshot order when reusing cached drafters
Collecting root candidates and sorting by absolute snapshot path could
pick a drafter from an older snapshot. _iter_hf_cache_snapshots yields
newest first and the main GGUF is resolved in that order, so preserve it
(root still preferred over MTP/ copies) to avoid pairing a fresh main
weight with a stale drafter revision.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio /v1/messages: accept thinking and unknown content blocks
The Anthropic-compatible /v1/messages endpoint modeled a message's content as
Union[str, list[{text|image|tool_use|tool_result}]], so any other block type
made Pydantic reject the whole request with
`messages.N.content.str: Input should be a valid string`. Resuming a Claude
session commonly replays assistant turns that carry `thinking` (extended
thinking) blocks, and sometimes a null content for a tool-only turn, both of
which tripped this and returned a 400.
Accept them:
- Add a permissive AnthropicUnknownBlock fallback (any block whose type is not
one of the four known ones), so thinking/redacted_thinking/provider-specific/
future blocks validate. A validator keeps known types on their typed models,
so a malformed known block (e.g. a tool_use without id) still fails cleanly.
- Coerce a null message (and tool_result) content to "" so the converter's
`for block in content` stays safe.
The converter already drops block types it does not translate, so a thinking
block is not forwarded to the model.
* Studio /v1/messages: keep user content validation strict
Make the thinking/null leniency role-aware so it never silently drops real
user input. Assistant turns (replayed history) still accept unknown/thinking
blocks and coerce a null tool-only turn to empty. User turns keep the strict
boundary: a null user content is rejected, and a content block the converter
cannot translate is rejected instead of being dropped into an empty prompt.
Also remove an empty file committed by accident.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio /v1/messages: coalesce resumed user turns and tighten content checks
- The /v1/messages count and generation paths now coalesce the adjacent user
turns that dropping an empty or null assistant turn can leave behind, so a
strict GGUF chat template no longer 400s on non-alternating roles.
- A user content block with a non-string type (list / dict) is rejected as a
clean 400 instead of raising TypeError and escaping as a 500.
- The assistant null-to-empty coercion only applies to an explicit null; an
assistant turn that omits content entirely still fails required-field
validation instead of being silently coerced to an empty string.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio /v1/messages: tighten comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* unstructured block removal
* Enhance unstructured block handling
* Restrict block cleanup to upload UIDs
* cleanup for seed block uploads
* upload cleanup queue for unstructured blocks in recipe studio
* Fix unstructured upload cleanup edge cases
* Fix unstructured upload import ownership
* Fix-unstructured-import-path-ownership
* Guard failed-delete restore against stale block in unstructured drop zone
* Drain queued upload cleanups when autosave is skipped
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* feat: detect installed coding agent CLIs in Studio settings
The API-keys panel only ever showed the "claude" flavor of the
`unsloth start` command, so anyone using Codex, OpenCode, OpenClaw,
Hermes, or Pi had to manually rewrite the copied command by hand.
Add a backend check that looks for each agent's CLI binary on PATH
(shutil.which, mirroring the pattern already used elsewhere in
studio/backend/utils) and expose it as GET /api/settings/coding-agents.
The API-keys panel now renders a picker for all six supported agents,
marks the ones it finds installed, and defaults to one of those instead
of always falling back to claude.
Includes unit tests for the detection helper.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* address review feedback on coding-agent detection
Three fixes from PR review:
- detect_installed_coding_agents now treats a PATH lookup failure as
"not installed" instead of letting it bubble up and break the
settings endpoint; added a regression test for it.
- CodingAgentsResponse.agents is now typed as an immutable tuple
instead of a list built from one, matching CODING_AGENTS itself.
- Fixed a race in the API-keys panel: picking an agent while the
installed-CLI check is still in flight could get silently overwritten
once that check resolved. A ref now tracks whether the user has made
a manual choice, so the auto-detected default only applies before
that happens.
* Address Codex feedback: GGUF gating and remote-detection scope
- codex refuses to launch against a non-GGUF (transformers-backed) model
(unsloth_cli's _require_gguf_for_codex), so auto-defaulting to it produced
a copy-pasteable command that fails immediately whenever the loaded model
isn't GGUF. Add useActiveModelIsGguf() (looks up the active checkpoint in
the chat runtime store) and a correction effect that steers the auto-pick
away from codex unless the loaded model qualifies, without ever touching a
choice the user made by hand.
- Detection runs via shutil.which on the Studio backend host, which isn't
the same machine as the browser in a tunnel/remote session. Reword the
'installed'/'detected' copy to say so explicitly when the tunnel URL is
in use, instead of implying the check ran on the viewer's own device.
* Rework auto-default per review: loopback gating + inline GGUF check
Replaces the previous approach with the exact shape discussed on the PR:
- Export isLoopbackHost/normalizeHost from agent-command.ts. The detection
endpoint runs shutil.which on the Studio backend, which only describes the
browser's own machine when the base this panel targets resolves to
loopback. For a LAN or tunnel/remote base, gate the whole thing off --
don't mark anything as "detected" and don't let it drive the default --
instead of just relabeling the copy.
- Drop the separate GGUF-correction effect and useActiveModelIsGguf hook.
Read useChatRuntimeStore.getState().activeGgufVariant inline inside the
existing detection effect's .then() (so it doesn't need to sit in the
effect's deps), and pick the first detected agent that isn't codex unless
the loaded model is GGUF, leaving the existing default untouched when no
compatible agent is detected.
Verified both branches (loopback vs LAN/tunnel base, gguf vs non-gguf,
manual pick preserved, no-compatible-agent fallback) with a standalone
port of the .then() logic.
* Address latest Codex findings: stale detection, model swap, cache
- Clear detectedAgents (and skip the network call entirely) when the panel
leaves a loopback base, instead of leaving a previous loopback detection
result marked 'installed' for a command that now targets a LAN/tunnel/
remote host.
- Add a separate, network-free correction effect keyed on the live
activeGgufVariant: if codex was auto-picked while a GGUF model was loaded
and the user then switches to a transformers-backed model while this panel
stays mounted, steer away from codex instead of leaving a command that
unsloth_cli's _require_gguf_for_codex will now reject. Never touches a
manual pick.
- Drop coding-agents.ts's module-lifetime cache. Installed-CLI detection is
environment state, not a persisted setting, so a stale positive/negative
from before the user installed something (or reopened the tab) is worse
than one extra cheap local API call per mount; keep only the in-flight
de-dupe for concurrent callers.
Verified the correction-effect logic (gguf->non-gguf swap with/without a
fallback, still-gguf no-op, manual pick never overridden) with a standalone
port of the effect.
* Make the codex/GGUF auto-pick symmetric in both directions
The correction effect only steered away from codex when the model stopped
being GGUF; it never steered back toward codex if the model became GGUF
*after* a non-GGUF-gated fallback had already picked something else (e.g.
codex is the only detected CLI, a transformers model is loaded so the
selection correctly falls back to the claude default, then the user loads a
GGUF model while the panel stays mounted -- codex never gets reconsidered).
Consolidate into one effect that re-derives the preferred detected agent
from scratch whenever detectedAgents or activeGgufVariant changes, in either
direction, instead of only reacting to the codex-specific downgrade case.
The fetch effect now only populates detectedAgents/availableAgents; this
effect is the single source of truth for what gets auto-picked from that
list. Never overrides a manual choice.
Verified both transition directions plus the manual-pick-survives and
initial-detection cases with a standalone port of the derivation logic.
* Reset the auto-pick to the default when it stops being trustworthy
Two more real gaps from the latest Codex pass on d988f52:
- The unified derivation effect only handled the case where a *different*
detected agent could take over. If codex was the only detected agent and
auto-picked while a GGUF model was loaded, then the model stopped being
GGUF, 'preferred' came back undefined and the effect silently left the
selection on codex -- exactly the command unsloth_cli's
_require_gguf_for_codex now rejects. Fall back to DEFAULT_AGENT in that
case instead of leaving it untouched.
- Leaving a loopback base cleared detectedAgents (so the 'installed' badges
correctly disappear) but left whatever agent had been auto-picked from
that now-stale, server-side-only detection still selected. Reset to
DEFAULT_AGENT there too, unless the user picked by hand.
Introduces a shared DEFAULT_AGENT constant instead of repeating the "claude"
literal at each reset site. Verified all five cases (both new resets, both
manual-pick-survives variants, and the existing multi-detected-agent
fallback still preferring another compatible agent over resetting) with a
standalone port of the effects.
* Derive GGUF-ness from the actual loaded state, not just the variant string
activeGgufVariant only covers an HF-repo GGUF pick (a specific quant
variant string). A direct local .gguf file -- custom folder, LM
Studio, or drag-drop -- is just as much a GGUF the codex preflight
(unsloth_cli's _require_gguf_for_codex) would accept, but it never has
a "variant" to report, so it read as non-GGUF here even though
/api/inference/status correctly reports is_gguf: true for it. That
mismatch could leave a Codex-only install not auto-selected, or reset
an auto-picked Codex, for a model that actually supports it.
Combined activeGgufVariant with activeNativePathToken (covers the
drag-drop/picked-file case) and ggufContextLength (only ever populated
when the backend last reported is_gguf: true for the active model, see
applyActiveModelStatusToStore) so all three paths a model can be GGUF
through are covered, matching the same is_gguf-or-equivalent check
hasGgufSource already applies to a staged pick elsewhere in this
codebase.
* Clear stale native-path token on a non-GGUF status refresh
When a native (drag-dropped or picked) GGUF was loaded and the backend later
switches to a transformers model outside the UI load path, refresh() adopts the
new /api/inference/status via setCheckpoint and applyActiveModelStatusToStore.
Those reset activeGgufVariant and ggufContextLength but never clear
activeNativePathToken, so the isGguf OR stays true after the switch and a
Codex-only detection auto-selects unsloth start codex for a non-GGUF model its
preflight rejects.
Drop activeNativePathToken in applyActiveModelStatusToStore whenever the status
is non-GGUF. A real GGUF load reports is_gguf: true, so its token is preserved
(the load path owns it); only a non-GGUF status clears it.
* Add the AGPL-3.0 header to the new studio contract test
* Fix/adjust agent detection for PR #6909
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
Co-authored-by: wasimysaid <112766706+wasimysaid@users.noreply.github.com>
* feat(cli): detect MLX distributed launch context
* feat(mlx): wire distributed inference backend
* feat(cli): broadcast MLX distributed chat turns
* fix(cli): wait indefinitely for distributed chat turns
* fix(cli): report MLX distributed load errors cleanly
* fix(mlx): route distributed vlm through loader
* fix(cli): detect inline MLX host JSON
* fix(studio): harden distributed object sharing
* fix(studio): select JACCL distributed backend
* fix(cli): abort distributed error paths
* Distinguish real stream errors from model text via GenStreamError in distributed CLI
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fail loud when MLX distributed init returns a singleton group
The worker only reaches this block when distributed was explicitly
requested. A singleton (size 1) group means the launch failed to form a
real group (MLX built without distributed support, or an invalid launch
env/hostfile); silently continuing leaves nonzero ranks looping forever
on share_distributed_object. Raise instead so the surrounding handler
returns a clear load error.
* Tighten MLX distributed inference comments
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: danielhanchen <unslothai@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: heal DiffusionGemma tool calls into structured tool_calls
* Fall back to supports_tools for backends without the passthrough capability
* Route DiffusionGemma client tools through passthrough when enable_tools is on
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop orphaned strip_tool_call_markup import after syncing with main
* Tighten supports_tool_passthrough comment
* Re-run CI on current main
---------
Co-authored-by: danielhanchen <unslothai@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* show chat by by last activity
* Update chat thread updated_at logic and enhance sidebar chat item handling
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Run the malware gate on the RAG embedding model before it loads
Setting the RAG embedding model through PUT /api/settings/embedding-model
persisted an arbitrary repo and later handed it straight to
SentenceTransformer, which deserializes pickle weights. Unlike the normal
model-load paths, this route never ran evaluate_file_security, and force
skipped verification entirely, so a repo Hugging Face flags as unsafe (or
any repo under force) could be downloaded and loaded in the backend
process without a scan.
Run the malware/pickle scan at both ends: the settings endpoint now scans
before persisting and returns 409 on a flagged repo even under force
(force still only skips the is-embedding-model type check for offline or
local repos), and the embedder scans again at the load sink so a name that
arrives via env or default is covered too. Local paths and unreachable
scans fail open inside evaluate_file_security, and the sink never bricks
the embedder on a gate error.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Thread the load token into the embedding scan and hard-fail on a block
The load-sink scan ran without a token, so evaluate_file_security (which
passes token=False when none is given) could not reach a gated or private
repo and failed open for exactly the model SentenceTransformer would still
load. Resolve the loader's own token (HF_TOKEN env or the cached login)
and pass it to the sink scan, and fall back to it in the settings endpoint
when the request omits one.
The sink previously raised a plain RuntimeError, which the llama-server
fallback in encode() and _build_st_backend_or_fallback() swallowed as a
routine ST failure, silently switching backends instead of blocking. Raise
a distinct UnsafeEmbeddingModelError that both fallback paths re-raise, so
a flagged model hard-fails.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scan sentence-transformers module dirs and scope the embedding pickle gate to the ST backend
Extend the RAG embedding malware gate so a poisoned pickle under a SentenceTransformer
module dir (for example 0_Transformer/pytorch_model.bin) blocks. Those dirs are read
from the repo's modules.json and passed as load roots to evaluate_file_security at both
the settings endpoint and the load sink, so such a pickle is treated as root-level there
instead of an unreferenced nested shard that was previously allowed.
Scope the ST pickle scan to the sentence-transformers backend. On the llama-server
backend the embedder loads GGUF files (inert) from the -GGUF companion repo, never the
ST repo's pickle, so a custom ST repo with a flagged pickle and a clean GGUF companion
is no longer rejected. The existing GGUF availability checks already cover that path.
Return 403 for the hard security block instead of 409. The settings UI routes every 409
into the forceable save-anyway flow, but this block cannot be bypassed by force, so it
now uses a distinct status the client treats as non-forceable.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Base the embedding pickle scan on the actual backend, not just the resolver
_llama_backend_active only consulted the auto resolver, so on a GPU box
where auto resolves to sentence-transformers but the process already fell
back to the llama-server backend at runtime (a torch or CUDA load/encode
failure), it returned False and the settings endpoint hard-blocked a save
whose ST pickle is flagged even though the process loads only inert GGUF.
Add active_backend_is_llama, which reflects the actual built backend (True
when the cached backend is a LlamaServerBackend, including a runtime
fallback) and otherwise defers to the resolver as a fresh process would,
and delegate _llama_backend_active to it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Report the cached embedding backend verbatim, not the resolver
active_backend_is_llama() fell through to the config resolver whenever a
backend was already built but was not llama-server, so a live
sentence-transformers backend could report llama=True once the resolver
picked llama (GPU heuristic or a runtime config change) and wrongly skip
its pickle scan. Once a backend exists, return isinstance(backend,
LlamaServerBackend) directly; only defer to the resolver before any
backend is built.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: apply presence_penalty on the safetensors and MLX inference paths
The safetensors and MLX generate paths resolved the inference config and
then dropped presence_penalty before generation, so the same model applied
the configured value under GGUF and 0 under safetensors/MLX. Thread the
already-resolved presence_penalty through the orchestrator command, worker
gen_kwargs, and the safetensors/MLX generate calls, and apply it with a
small logits processor (subtract once per distinct completion token,
prompt excluded, presence not frequency, zero is a no-op, negatives raise).
Backwards compatible: presence_penalty defaults to 0.0 (byte-identical
output when unset) and the GGUF path is unchanged. Also forward min_p on
the legacy /generate/stream route and add the missing min_p field to
GenerateRequest.
* Studio: bound presence_penalty generated ids to valid vocab range on both paths
The presence-penalty logits processors index by generated token ids. The
torch path filtered only the upper bound (seen < vocab_size), so a negative
id would silently wrap to the wrong row; the MLX path had no bound at all,
and MLX out-of-bounds indexing is documented undefined behavior (crash or
memory corruption on Apple Silicon), unlike torch's harmless negative wrap.
Bound generated ids to [0, vocab) consistently on both paths:
- torch: seen[(seen >= 0) & (seen < vocab_size)] (zero-regression safety net;
real completion tokens are always in range).
- MLX: route out-of-range/negative ids to a discarded scratch slot via
mx.where and a (vocab + 1)-wide scatter-assign mask, then subtract. MLX has
no boolean-mask filtering (data-dependent output shape), so this keeps a
fixed shape, stays on-device, and preserves once-per-distinct-token
semantics without any torch/numpy dependency.
Add torch tests for out-of-range and negative ids (only in-range distinct
ids penalized, stray ids ignored, no wrong-index wrap) and a bound-documenting
MLX test that runs on the arm64 macOS CI.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: client-tool passthrough healing for safetensors and MLX
PR 6801 made response-side tool-call healing default-on for the client-tool
passthrough, but only on the GGUF path: the passthrough branch in
/v1/chat/completions is gated on using_gguf, and the safetensors section never
reads payload.tools, so a client-tools request against a safetensors or MLX
model silently dropped the tool schemas and returned prose with no tool_calls.
Add the missing leg. When a non-GGUF model is loaded, the request declares
client tools (or carries tool-role history), server-side tools are off, and the
template supports tools, the route now:
- renders the tools into the chat template for a single turn via the existing
backend.generate_chat_response(..., tools=...) seam (worker templating
already accepts role=tool and assistant.tool_calls messages, normalized with
_openai_messages_for_passthrough);
- non-streaming: promotes text-form calls with heal_openai_message, honors the
opt-in nudge single retry (nudge_should_retry / nudge_messages), caps healed
calls when parallel_tool_calls=false (covers the nudge retry too), and sets
finish_reason=tool_calls with content null on a pure tool-call turn;
- streaming: derives deltas from the worker's cumulative snapshots and feeds
StreamToolCallHealer, emitting healed tool-call deltas and the correct
finish chunk, guarded against repeated or shrinking snapshots.
heal_gate semantics are identical to the GGUF passthrough: default on,
auto_heal_tool_calls=false or UNSLOTH_DISABLE_TOOL_CALL_HEALING=1 relays
verbatim, tool_choice narrows promotion, undeclared names stay text. MLX rides
the same orchestrator seam, so both local backends gain the behavior.
CompletionMessage.content becomes Optional so a promoted pure tool-call turn
matches the OpenAI contract (content null when only tool_calls return).
Adds tests/test_sf_client_tools_passthrough.py (22 cases: healing, gating,
opt-outs, streaming deltas, tool-role history, dict-arguments history, forced
tool_choice, parallel cap, usage, nudge on/off/double-failure, generator error
hygiene, disconnect reset, empty output, MLX path).
* Address review: tool_choice none, developer folding, retry fallback, monitor reply
Four review follow-ups on the safetensors/MLX client-tool passthrough leg:
- tool_choice="none" keeps the tool-history templating but no longer
advertises the tools, so a forced final-answer turn is not prompted into
emitting markup that the (correctly disabled) healer would relay as prose.
Mirrors the GGUF passthrough where llama-server honors tool_choice itself.
- OpenAI "developer" messages fold into a single leading system message via
_set_or_prepend_system_message before templating; local templates reject the
role and the fallback formatter drops it.
- A nudge retry that fails or is cancelled after the original answer exists
falls back to the first response instead of surfacing a 500, matching the
GGUF nudge path.
- The API monitor records the healed tool call summary instead of the raw
markup on a promoted turn.
Adds four regression tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: forced tool_choice templating, content-part flattening, stream monitor parity
- A forced tool_choice function is now the only schema rendered into the
local template, so the advertised tools and the healer allowlist can no
longer disagree (llama-server enforces tool_choice itself on the GGUF path).
- Content-part lists are flattened to their text parts before templating.
Remote image URLs are not decodable locally, so such requests reached this
path with part lists that raise inside apply_chat_template on text-only
templates; the plain non-GGUF path has always flattened them.
- The streaming monitor entry is now fed from the healed events the client
actually receives, recording promoted calls as the [tool_calls] summary
the non-streaming path records.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: gate passthrough on the engaged server path, deserialize templated arguments
- The client-tools gate now keys on _sf_use_tools (whether the server-side
tool path actually claimed the request) instead of the raw mcp_enabled
flag: with an empty MCP registry or a CLI --disable-tools policy, a client
that sets mcp_enabled while declaring its own tools fell through to plain
generation with the tools silently dropped. The GGUF passthrough gate has
no mcp_enabled clause either.
- New _structured_tool_history_for_local_template deserializes assistant
tool_calls[].function.arguments JSON strings into mappings for the
templated copy only: spec-compliant clients send strings, but local chat
templates iterate arguments as a mapping or raise on strings, which
crashed or misrendered multi-turn tool history. The HTTP response and the
GGUF wire shape keep strings.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments and docstrings in the client-tools passthrough
* Report first-attempt usage when a nudge retry is discarded
When nudge_should_retry fires but the retry produces no healable tool call
(or raises), the first response is still delivered to the client. The retry's
generate() had already overwritten stats_holder, so _monitor_usage recorded
the unseen retry's token counts against the request instead of the first
attempt that was actually returned. Capture the first attempt's stats before
the retry and restore them on both the no-heal and exception paths so the
monitor reports the usage of the response the caller received.
* Do not promote buffered tool markup when a stream is cancelled
The streaming client-tool heal path breaks out of the token loop when
cancel_event is set (the registry "Stop" path), but then still fell through to
healer.finalize(), which heals incomplete tool markup at EOF (allow_incomplete)
and emits a tool_calls delta plus finish_reason=tool_calls. Because the Stop
request only sets the event and leaves the SSE socket open, the client received
that promoted call and executed a tool the user had just cancelled. The disconnect
path already returns before finalize; guard finalize and the finish_reason on
cancel_event too, so a cancelled stream ends with finish_reason=stop and no tool
call. Adds a regression test driving a Stop mid-emission with buffered markup.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments in the client-tools passthrough
* Trim client-tools passthrough comments further
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Fix six race conditions when a user switches or cancels a model while a
previous load or generation is still in flight, across the inference
orchestrator and the /load and /unload routes:
- Cancel an in-flight generation on a safetensors/MLX model switch and
serialize unload with load under the inference lifecycle gate.
- Cancel an in-flight load off the lifecycle gate so a Stop-loading
cancel does not wait out the multi-minute load; guard the dispatched
mailbox against a racing unload.
- Recheck the loading marker after spawn and again after the load
response before publishing, so a load cancelled mid-flight is reaped
instead of going live.
- Discard the loading marker before tearing the subprocess down in
cancel_load, closing a spawn-after-cancel window and an orphaned
compare-mode dispatcher during unload.
- Match the unload target before canceling an in-flight GGUF load and
add an off-gate fast path for the still-loading GGUF case.
- Run the Unsloth unload off the event loop so a paused SSE stream
holding _gen_lock cannot block the loop.
Adds studio/backend/tests/test_orchestrator_unload_cancel.py covering
the unload/cancel/switch race paths.
* Studio chat: tool-call nudging on by default (API stays opt-in)
Healing is already default-on everywhere and the nudge retry from the
client-tool passthrough is opt-in on the API. Studio chat had neither
signal: the frontend never sent nudge_tool_calls, and the safetensors
and MLX server-side loop lacked the GGUF loop's plan-without-action
re-prompt entirely.
Backend: the re-prompt helpers move from llama_cpp.py into
tool_call_parser.py (shared, cycle-free; the GGUF loop imports them
under its old names with zero behavior change) and
run_safetensors_tool_loop now re-prompts once at the streaming
no-tool-call exit, gated on Auto-Heal, active tools, nothing executed
yet, and short forward-looking text. Re-prompts do not consume tool
iterations.
Frontend: the chat adapter sends nudge_tool_calls from a new
nudgeToolCalls runtime setting (default true) with the same
persistence, hydration, and settings toggle plumbing as Auto-Heal.
Request-model defaults are untouched, so raw API callers stay opt-in.
* Address review: persist the nudge setting, consume the flag in the loops, skip the re-prompt after RAG autoinject
ChatSettingsPayload uses extra forbid, so a settings patch containing
nudgeToolCalls failed to persist any settings; the field is now typed
and round-trips. nudge_tool_calls now plumbs into both server-side tool
loops and gates the plan-without-action re-prompt with None meaning on,
so API callers keep today's behavior, explicit false disables it, and
Studio's default-on flag actually controls the path Studio chat runs.
The safetensors loop no longer re-prompts after RAG autoinject: the
injected retrieval bypasses the tool controller, so the nothing-executed
gate saw an empty history and re-asked after a successful retrieval.
* Safetensors loop: the plan-without-action retry requires an explicit nudge flag
The retry is new on this loop, so an omitted nudge_tool_calls must not
change existing API behavior; Studio opts in explicitly. The GGUF loop
keeps None as on because its re-prompt predates the flag.
* Suppress the plan-without-action re-prompt after a denied tool confirmation
A denial appends TOOL_REJECTED_MESSAGE but records nothing in the tool
controller history, so the nothing-executed gate re-prompted the model
to call the tool the user had just rejected, producing another
confirmation prompt. A denial now suppresses the re-prompt for the rest
of the request, mirroring the RAG autoinject handling.
* Tighten plan-without-action re-prompt comments
* Tighten plan-without-action re-prompt comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: match unified plan-without-action nudge cap to GGUF default of 3
The shared MAX_ACT_REPROMPTS was set to 1, but GGUF's established default
(llama_cpp.py) has re-prompted a stalling model up to 3 times since #5620.
Restore the GGUF-matched cap so safetensors and MLX inherit the same
behavior, and update the safetensors cap test to assert the cap dynamically.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: parse Mistral [TOOL_CALLS] and rehearsal tool-call shapes
Extends the rescue parsers in core/tool_healing.py and
core/inference/tool_call_parser.py to recognise two extra serialisations
local models commonly emit when bypassing native function calling:
* [TOOL_CALLS]name{json_args} (Devstral-Small-2, Mistral-Small-3.x).
* name[ARGS]{json_args} (reasoning-model rehearsal).
Both extractors use a brace-balance scan that honours escapes and
quoted strings so nested JSON args stay intact.
Also pre-strips <think>...</think> and [THINK]...[/THINK] blocks before
matching so calls emitted after a reasoning preamble are recognised
regardless of position.
Streaming gates (TOOL_XML_SIGNALS, llama_cpp.py _TOOL_XML_SIGNALS) and
the SSE strip regex (routes/inference.py _TOOL_XML_RE) gain the new
sentinels so the parser is actually invoked and the raw markup never
leaks to the UI.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Strip unclosed think blocks and catch rehearsal [ARGS] mid-buffer
The pre-existing ``_THINK_TAG_RE`` only matched closed thinking
blocks (``<think>...</think>`` or ``[THINK]...[/THINK]``). During
streaming the model is still inside the open block when the parser
runs, so any tool-shaped markup the model is REHEARSING inside that
block survived the strip and could be executed as a real call.
Switch both copies of the regex (parser + healing) to accept the
trailing block being terminated by end-of-string in addition to
the explicit closer.
The ``_TOOL_XML_SIGNALS`` list on the llama_cpp streaming buffer
included ``[ARGS]`` to catch rehearsal syntax, but the gate used a
``startswith`` check against the buffer head -- rehearsal is shaped
``name[ARGS]{json}``, so the buffer never STARTS with ``[ARGS]``
and the signal had no effect. Add a substring fallback for the
bracket-style signals so the BUFFERING window can still divert the
stream into DRAINING when rehearsal markup arrives mid-buffer.
Adds three regression tests covering rehearsal inside unclosed
``<think>`` / ``[THINK]`` blocks (must yield no calls) and the
positive case after a closed think block (still parsed).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden bracket-tag tool-call parsing and streaming strip
Address review findings on the Mistral [TOOL_CALLS] / rehearsal [ARGS] paths:
- Accept hyphenated tool names in the bracket parsers and strip patterns.
_MISTRAL_BRACKET_RE and _REHEARSAL_RE used \w+, which dropped or truncated
MCP function names containing dashes (mcp__srv__list-issues). Use [\w-]+ to
match the XML and Gemma parsers.
- Strip a partial bracket marker streamed before its opening brace. The
trailing-unclosed patterns required the {, so a [TOOL_CALLS]web_search or
python[ARGS] split across deltas leaked the raw marker to the UI. Match the
bare marker to end-of-text, mirroring how the bare open tags are stripped.
Closed pairs are unchanged so in-progress markup stays buffered until parsed.
- Strip a truncated bracket tail in the route-level display regex. _TOOL_XML_RE
required a balanced JSON object; a tool call truncated by EOS now strips up
to \Z, like the orphan-opening XML shapes. Complete calls still strip only
their balanced JSON so following prose survives.
Add regression tests for hyphenated names, the streaming partial-marker strip,
and the unclosed-tail route strip.
* Studio: preserve XML parameter indentation in tool_healing
The chat template emits <parameter=k>\nVALUE\n</parameter>; the parameter-start
regex consumed the wrapping newline AND the value's first-line indentation via a
trailing \s*, then str.strip() removed the rest, corrupting code/diff arguments.
Narrow the trailing class to horizontal whitespace and trim exactly one wrapping
newline (_trim_param_value), preserving indentation. Matches SGLang's qwen3_coder
detector and the same fix on the multi-format parser. Add a regression test.
* Studio: tighten Mistral/rehearsal tool-call comments
Compress the comments in the Mistral [TOOL_CALLS] / rehearsal [ARGS] healing shim
and its callers to one or two lines, keeping the bracket-tag stripping rationale,
the thinking-block handling note, and the forge attribution intact.
Comment-only: no code or behavior change (verified with comment_tools.py check
--strip-docstrings; tests green).
* Studio: fix think-strip arg corruption and nested bracket-JSON strip
Review follow-up for the Mistral/rehearsal healing shim:
- The <think>/[THINK] strip ran unconditionally over the whole content before
parsing, so a real tool argument that legitimately contained a <think> /
[THINK] literal was silently corrupted. Don't delete the blocks: compute the
reasoning-block spans and skip any tool-call candidate that STARTS inside one,
across all parse paths (JSON, Gemma, XML, bracket, rehearsal). A rehearsed call
inside reasoning is still ignored; a real call after </think> still parses.
- The bracket-tag display strip used a fixed one-level-nesting regex, so a call
with two-level-nested JSON args either leaked raw markup or, in final mode, let
the catch-all eat the trailing prose. Add a balanced-brace
_strip_bracket_tag_calls pass (any nesting depth) used by strip_tool_call_markup
and the route display strip.
Add regressions: <think>/[THINK] literal inside a real argument, rehearsal-inside-
think with a real call after, and two-level-nested bracket/rehearsal strip keeping
trailing prose.
* Studio: correct think-block comments to match span-skip behavior
The think-strip fix replaced the unconditional think-block strip with a
span-skip (the block is kept and any tool-call candidate starting inside it is
ignored), but two comments still described the old strip-first behavior. Update
the _THINK_TAG_RE comment and the parse_tool_calls_from_text docstring.
* Studio: parse Mistral arrays and call-ids, unify bracket parse/strip, keep it linear
- Parse the canonical Mistral array form (TOOL_CALLS followed by a JSON list of
calls) and emit every call; parse the v11 shape that carries an opaque CALL_ID
token between the name and ARGS (the function name is the token after
TOOL_CALLS, never the call-id); and parse a Mistral call plus a rehearsal call
in one message (the second was dropped yet still stripped from display).
- One shared balanced forward scan (_iter_bracket_spans) backs both the parser
and the strip path, so they no longer diverge. It is linear: each regex is
re-searched only once its cached match falls behind the cursor, replacing the
per-match full-tail re-scan that was O(n^2) (O(n^3) over a stream). A length cap
before the scan is a backstop.
- strip_tool_call_markup preserves think/reasoning blocks verbatim (the parser
skips tool markup inside them), stripping only the visible text around them.
- _in_think uses bisect over the sorted think spans (was a linear scan per
candidate).
- GGUF streaming strip runs the balanced bracket pre-pass before the regex
patterns so nested-arg calls do not leak or eat trailing prose, and the
BUFFERING ARGS detector requires the rehearsal name-ARGS shape.
- Tests: canonical array, array string-args, array strip keeps prose, Mistral
plus rehearsal multi-call, v11 call-id name, think-rehearsal strip
preservation, and bracket-strip linearity.
* Studio: preserve reasoning blocks in the route and streaming strip paths too
Addresses Gemini/Codex review: making strip_tool_call_markup preserve think
blocks left the route display strip and the GGUF streaming strip inconsistent,
so a rehearsed call inside a reasoning block was still deleted from the visible
text on those paths.
- Extract the think-block segmentation into one shared helper (strip_outside_think)
and route all three strip paths through it: strip_tool_call_markup,
_strip_tool_xml_for_display, and the GGUF _strip_tool_markup_streaming closure.
- Add a route-strip regression test that a rehearsal inside a reasoning block is
preserved while a real call outside it is still stripped.
* Studio: fix bracket-tag strip/buffer review findings
Address the live code-review findings on the Mistral bracket-tag / rehearsal
tool-call rescue path:
- tool_healing: a literal think block inside a tool-call argument is no longer
treated as a reasoning block. strip_outside_think now excludes think spans
that sit inside a complete tool-call span, so the call is stripped whole
instead of the split hiding its open/close pair and leaking the raw call.
- tool_healing: the rehearsal trailing-strip pattern requires a following brace
or end-of-text, so prose that merely mentions name[ARGS] is not truncated as
a phantom call. The bracket strip patterns are aligned with the parser
regexes (whitespace, v11 [CALL_ID]/[ARGS] metadata, and the [CALL_ID]
lookbehind).
- routes: strip a truncated canonical Mistral array ([TOOL_CALLS] [{... with no
closing bracket) that the balanced scan cannot remove, align the display
regex with the parser regexes, and apply the same rehearsal-prose guard.
- safetensors loop: mirror the GGUF [ARGS] rehearsal-substring check during
BUFFERING so a rehearsal name does not stream before its [ARGS] arrives.
Adds regression tests for each; existing parser suite stays green.
* Studio: hold split rehearsal tool-name prefix in both streaming loops
A reasoning-model rehearsal call can stream the tool name and its [ARGS] arm in
separate chunks (web_search then [ARGS]{...}). The buffering detector only
recognised the rehearsal once [ARGS] was present, so the bare tool name was
emitted as visible content before the call drained and executed.
Add _is_rehearsal_prefix (mirrored in the safetensors loop and the GGUF loop):
when a no-signal buffer is a bare active-tool name -- or a partial prefix of
NAME[ARGS] -- hold it as a prefix instead of streaming it, so the next chunk's
[ARGS] flips it to a drain. A whitespace in the buffer means prose, not a split
call, so ordinary text still streams.
Adds regression tests for the split rehearsal in both loops and a guard that a
plain non-tool word still streams.
* Studio: route Anthropic tool-call cleanup through the protected display strip
The Anthropic stream, non-stream, and passthrough paths cleaned content with raw
_TOOL_XML_RE.sub instead of _strip_tool_xml_for_display, so a rehearsal call
inside <think> was deleted from the reasoning and a nested [TOOL_CALLS] call
dropped its trailing prose (the OpenAI-compatible paths already use the helper).
Route all four sites (prior-assistant cleanup, streaming content events,
non-stream aggregation, passthrough conversion) through the protected helper, and
add a source-level guard test so raw _TOOL_XML_RE.sub stays confined to the
helper itself.
* Studio: stop split rehearsal tool names leaking once streaming, uncapped, or unrestricted
The split-rehearsal guard (NAME in one chunk, [ARGS]{...} in the next) only held
the name in the initial BUFFERING state. Three gaps remained where the bare tool
name still streamed as visible content before the call drained:
- STREAMING: after prose had already streamed, both loops emitted a trailing
active-tool-name token (and the GGUF/safetensors [ARGS] boundary was not pulled
back over the name). Hold the trailing rehearsal token and release it on the
next chunk, with an end-of-stream flush so a plain answer that merely ends on a
tool-name word is never dropped.
- Buffer cap: a realistic MCP name longer than the 32-char _MAX_BUFFER_CHARS cap
defeated the BUFFERING hold. A rehearsal prefix is self-bounding (it stops
matching once it grows past NAME[ARGS]), so the generic cap no longer applies to
it.
- Unrestricted mode (tools=[]): with no declared tool list, any bare identifier
may be a NAME[ARGS] rehearsal, so the prefix check now recognises one instead of
leaking the name and mis-parsing the call.
Regression tests cover the streaming, long-name, and unrestricted cases plus the
plain-prose paths that must not be held or corrupted.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio tools: protect think blocks in safetensors streaming, hold split rehearsal on initial flush, advertise Mistral tools
Pass-3 review follow-ups on the Mistral [TOOL_CALLS] / rehearsal [ARGS] work:
- Safetensors streaming display strip now preserves think / [THINK] reasoning
verbatim (routes through strip_outside_think like the GGUF path). A call
rehearsed inside a reasoning block was stripped mid-stream and then restored by
the final strip, a non-monotonic shrink/grow that corrupted append-by-length
stream consumers and the visible reasoning.
- The first flush out of BUFFERING (safetensors and GGUF) now applies the same
trailing-name hold the STREAMING branch uses, so a split rehearsal (prose plus a
trailing active tool name in one chunk, [ARGS]{...} in the next) no longer leaks
the bare name before the call drains.
- Safetensors capability gate no longer suppresses tools for Mistral [TOOL_CALLS]
templates, which the shared bracket-tag parser now handles end to end. Llama
python_tag stays suppressed (still unparseable).
- Route display strip applies the open-ended / bare-marker tail arms only on the
segment after the last reasoning block (closed-only regex before it), matching
strip_tool_call_markup, so a bare foo[ARGS] before a reasoning block is preserved
while complete calls are still removed in every segment.
Adds regression tests for each and updates the now-stale Mistral capability test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix tool-call think-marker and bracket-wrapper edge cases
Round-1 review follow-ups on the Mistral/rehearsal tool-call healing:
- tool_healing: a reasoning marker that opens INSIDE a tool call's
arguments is argument data, not a reasoning block. Add
_think_spans_outside_tool_markup (start-inside test) and use it in
both parse_tool_calls_from_text and strip_outside_think so a literal
marker in one call's args no longer hides a later call (parse) or
leaks the raw markup (strip) when the greedy match runs past the
call's closer.
- tool_healing: strip the orphan Mistral v11 [/TOOL_CALLS] closer left
behind after the balanced scan removes the call body. Add a route arm
for the same closer in _TOOL_XML_RE / _TOOL_XML_CLOSED_RE.
- safetensors + llama_cpp streaming strip: run the open-ended (EOS
anchored) tail patterns only on the last segment; segments before a
reasoning block use the closed-only patterns, matching the final
strip and the route strip. A bare foo[ARGS] before a reasoning block
is prose, not a truncated call.
- safetensors streaming detector: validate each [ARGS] hit before
draining. A bare foo[ARGS] in prose (no active tool name in front)
no longer drains the rest of the turn; a later real NAME[ARGS] call
is still found and the prose in between is preserved.
Regression tests added for each case across the parser, strip helpers,
and both streaming loops.
* Strip incomplete-XML tool markup with literal think tags; widen render-html detector
Round-2 review follow-ups.
- tool_healing: an UNCLOSED <tool_call> / <function= call that the parser still
executes via allow_incomplete leaked its markup when an argument contained a
literal think marker. _tool_call_markup_spans only covered closed calls, so the
literal was treated as a reasoning block to preserve. Extend it to the
open-ended XML tail forms (shared as _TOOL_OPEN_XML_TAIL_PATS) so a think marker
inside an unclosed call is argument data and the call's markup is stripped. A
complete call's opener stays bounded to its closed span, and a real reasoning
block with no tool call is still preserved.
- safetensors render-html provisional card: _detect_render_html_tool_start was
XML-only, so a Mistral [TOOL_CALLS]render_html or rehearsal render_html[ARGS]
call executed but skipped the early card. Detect the earliest tool-call marker
across every serialization the loop executes and fire when it is render_html.
Regression tests added for both.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio tools: gate [ARGS] on active tools and skip think-block render_html rehearsal
Round 3 review fixes for the Mistral / rehearsal tool-call parsing path. Both are
asymmetric-fix bugs where one code path applied a guard the analogous paths did not.
- [ARGS] active-tool gating: the streaming state already validates a rehearsal
NAME[ARGS] against the active tool list before draining, but the BUFFERING
detection and the end-of-stream safety-net checks (safetensors and GGUF) treated
any word[ARGS] substring as a tool boundary. An answer containing a literal
foo[ARGS]{...} in prose, where foo is not an enabled tool, was drained, parsed into
a disabled foo no-op, and forced an extra generation turn. Gate those checks on the
active tool name too (unrestricted mode still accepts any name), so inactive-name
prose is neither drained nor parsed. Adds a shared _has_genuine_tool_signal helper
(safetensors) and _gguf_rehearsal_signal_pos / _gguf_has_genuine_tool_signal (GGUF).
- render_html provisional card vs think blocks: the parser skips tool candidates that
start inside a <think>/[THINK] reasoning block, but the provisional render_html
detector scanned raw content. A render_html rehearsed inside <think> followed by a
real non-render_html call emitted a provisional render_html tool_start (reusing the
later call's id) that the loop never executed. Drop candidates that start inside a
think span and use the first marker of each shape outside the blocks. Also resolve
the [TOOL_CALLS] [{...}] array shape through the parser so a nested "name" argument
key no longer fires a false provisional card ahead of the real top-level tool name.
Adds regression tests for both loops: inactive-name foo[ARGS]{...} is not drained into
a disabled no-op or a retry turn, a think-block render_html rehearsal emits no
provisional card, and the array top-level name is read correctly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate ambiguous bare-rehearsal parse and strip on the active tool list
A bare NAME[ARGS]{json} is a genuine rehearsal call only when NAME is an
active tool; otherwise it is prose. The earlier round gated only detection
(so an inactive foo[ARGS] no longer drained the buffer or forced a retry
turn), but the parse and strip stayed unrestricted, which produced two
regressions:
1. An inactive foo[ARGS]{...} placed immediately before a real
web_search[ARGS]{...} in the same content span made the real call fail
to execute (parse consumed the phantom foo call).
2. An inactive foo[ARGS]{...} in a prose answer had its markup stripped
from the visible text, corrupting the sentence to " is just syntax."
Thread enabled_tool_names through the shared parser/strip so parse and
strip apply the SAME active-tool gate as detection:
- core/tool_healing.py: _iter_bracket_spans skips an inactive rehearsal
span; parse_tool_calls_from_text, _strip_bracket_tag_calls,
_strip_markup_segment and strip_tool_call_markup accept and thread the
gate; apply_tool_strip_patterns keeps an inactive rehearsal match.
- core/inference/tool_call_parser.py: wrappers forward the gate.
- core/inference/safetensors_agentic.py and core/inference/llama_cpp.py:
compute the gate from the active tool list (None when unrestricted, to
keep the legacy strip-all behavior) and thread it into every parse and
streaming/final strip site.
- routes/inference.py: _strip_tool_xml_for_display accepts the gate and
keeps an inactive rehearsal via a capture group on its rehearsal arm, so
the display cleanup does not re-strip the already-correct loop output.
The [TOOL_CALLS] control-token arms still strip unconditionally. Wire
the current turn's active tool names into the GGUF and safetensors
content-display sites.
Tests: parse and strip gate coverage in test_tool_call_parser_strict.py,
test_tool_xml_strip.py and test_safetensors_tool_loop.py; end-to-end GGUF
coverage for the real-call-after-inactive-rehearsal case and a
strengthened assertion that the inactive rehearsal prose survives intact.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: render the reasoning block for safetensors and MLX like GGUF
enable_thinking chat templates (Qwen3/Qwen3.5/GLM) prefill an unclosed <think>
into the generation prompt, so the model emits only the closing </think> then
the answer. The safetensors/MLX chat stream emitted that as plain content, so
the reasoning showed inline with no collapsible thinking block, while GGUF
(which surfaces reasoning via reasoning_content) rendered one. This brings
safetensors and MLX to parity.
- _ResponsesReasoningExtractor gains a reasoning_prefilled mode that starts
inside the reasoning block and splits on the first </think>; default False
keeps GGUF and every existing caller byte-identical. It suppresses a stray
re-emitted <think> and holds partial markers back across chunk boundaries.
- _sf_reasoning_prefill_mode gates the mode on reasoning being enabled for the
request, an enable_thinking or enable_thinking_effort style, and the template
actually using the standard <think>/</think> markers. Models with a bespoke
reasoning channel (e.g. gemma's <|think|>/<|channel>) are excluded so their
answer is never swallowed; gpt-oss (Harmony) and thinking-off requests are
excluded too.
- sf_tool_stream and stream_chunks (the latter also serves MLX) feed text
through the extractor, emitting reasoning_content then content deltas, with a
per-turn reset in the tool loop and a flush before each tool_start; only the
visible delta reaches the monitor reply. The two non-streaming drains split
reasoning_content the same way.
- Tests: extractor prefilled mode (streaming and edge cases), the gate matrix
including the gemma-style exclusion, and a route-replay of the tool-loop
reasoning stream.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: skip tool calls rehearsed in prefilled reasoning
Reasoning models (Qwen3.5 enable_thinking) open <think> in the prompt, so the
generated text starts inside the thought and emits only a closing </think> with
no opener. _think_spans_outside_tool_markup only found spans with an explicit
opener, so a NAME[ARGS]{...} or [TOOL_CALLS] call rehearsed in that leading
thought was parsed and executed as a real call.
Add a leading think span (offset 0 through the first close marker) when the
content opens with a bare close, so the rehearsed call is skipped and the
reasoning is preserved by strip_outside_think. Guarded by the existing call-span
check: a literal </think> inside a real call's arguments does not trigger the
span, so a genuine leading call still fires. Tests for both cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: do not start prefilled reasoning mode when reasoning_effort is none
enable_thinking_effort models (e.g. GLM-5.2) express thinking-off via
reasoning_effort="none" rather than enable_thinking=False, but
_sf_reasoning_prefill_mode only looked at enable_thinking, so such a request
started the extractor in prefilled mode. With thinking off the model never emits
</think>, so the whole answer was captured as reasoning_content and the visible
content/stream came back empty. Thread reasoning_effort through and return False
when it is "none". Tests for none vs a real effort level.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: only treat a leading bare </think> as prefilled reasoning when a real call follows
The prefilled-reasoning virtual span fired on any unmatched leading close marker,
so a non-prefilled turn that emits a real call before a stray </think> (for
example "Now web_search[ARGS]{...}</think> answer") had the call swallowed by the
span and dropped. Require that a real tool call also appear after the close (the
actual turn that follows the thought) before adding the span, so a stray close in
a normal answer no longer suppresses a genuine leading call. The rehearse-then-
call case still skips the rehearsal. Test for the stray-close case.
* Studio: trim redundant comments (comment-only, AST-verified)
* studio: keep tool_healing importable on Python 3.9
_balanced_json_span was annotated -> int | None. With no
from __future__ import annotations, that PEP 604 union is evaluated at
import time, so on Python 3.9 (which the package still supports,
requires-python >=3.9, and where external inference servers import this
module standalone) the def raises TypeError and the whole module fails
to import before any parsing runs.
Add from __future__ import annotations so annotations stay lazy strings,
matching the prevailing convention across studio/backend. No behavior
change: the module has no runtime annotation introspection.
* Studio: gate the Anthropic tool-stream display strip on declared tools
The Anthropic streaming and non-streaming tool paths called
_strip_tool_xml_for_display without enabled_tool_names, so with the default
strip-all behavior a final answer that literally contains an inactive-name
NAME[ARGS]{json} (prose, not a call) lost those bytes in the delivered text.
The GGUF and safetensors paths already pass _display_tool_name_gate(tools);
these two sites were missed when that gate was threaded through.
Compute the gate from the declared tools and pass it at both sites (threading
openai_tools into _anthropic_tool_non_streaming and its caller), so an
inactive-name rehearsal survives while an active-name one is still stripped.
Add a regression test.
* Studio: hold a split unrestricted rehearsal prefix at the bracket
In unrestricted tool mode (tools=[]) the rehearsal-prefix regex required
[A after the bracket, so a chunk boundary landing right after NAME[ (e.g.
web_search[ then ARGS]{...}) failed the prefix check and streamed the
partial tool markup web_search[ to the client before the call drained.
Restricted mode already holds this via a startswith check. Make the bracket
and each ARGS letter individually optional so NAME[ is held too, matching
the documented intent. Add a regression test.
* Studio: gate rehearsal detection and history strip on the original tool set
Two display/loop gate fixes so a spent one-shot tool is handled consistently:
- Rehearsal DETECTION (safetensors and GGUF loops) now uses the ORIGINAL tool
list, matching the strip gate, instead of the post-removal active_tools. After a
one-shot tool (render_html) runs it is dropped from active_tools; a repeat
render_html[ARGS]{...} while another tool is still active was stripped from
display yet never detected, so it was not routed to the render_html_repeat no-op
and the turn ended as a blank continuation. Detection now fires for it.
- The GGUF assistant-history sanitiser forwards the enabled-tool-name gate (like
the live-response strip), so a prior turn documenting an inactive foo[ARGS]{...}
shape is preserved in the replayed prompt context instead of being deleted.
Add regression tests for both loops and the history strip.
* Studio: thread the tool-name gate through the remaining rehearsal/history sites
Follow-up to the rehearsal-detection and history-strip gate fixes, covering the
sibling sites that were missed:
- GGUF loop: the rehearsal-prefix and trailing-name hold checks now use the
original tool list (_detect_tools) like the detection path, so a spent one-shot's
split repeat (bare render_html then [ARGS]{...}) is held instead of flushed as
visible text.
- The safetensors and Anthropic assistant-history sanitisers and the Anthropic
non-streaming passthrough now forward the enabled-tool-name gate to
_strip_tool_xml_for_display, matching the GGUF history sanitiser and the live
strips, so a prior turn documenting an inactive foo[ARGS]{...} example is
preserved in the replayed prompt / final text instead of deleted.
Add regression tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tile bracket-call spans per array item and include the v11 closer
Two with_spans fixes for the Mistral bracket parser, both hit through the
client-tool passthrough healers:
- A multi-call [TOOL_CALLS] array carried its whole markup span on the first
call and zero-width spans after, so a consumer that filters promotions by
the declared tool set either re-emitted the full raw array as text next to
the promoted call or silently dropped a filtered call's bytes. The region is
now tiled across the call-producing items (each call's span covers its own
JSON object plus the separator bytes before it; the last span runs to the
region end), so promoted markup strips exactly once and a skipped call's
bytes stay visible.
- The v11 wrapper closer [/TOOL_CALLS] sat outside the reported span and
leaked as stray text after promotion; the region now extends over an
immediately-following closer.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: decouple healer signals from the loop signal set
The passthrough healer buffered on every TOOL_XML_SIGNALS entry, so the bare
[ARGS] rehearsal marker this branch adds for the loops (where it is gated on
active tool names) put legitimate prose like 'Use foo[ARGS] in templates'
into the holding state and stalled the stream until finalization. The healer
can never promote a bare rehearsal call, so it now buffers only on formats
its parser promotes: <tool_call>, <|tool_call>, <function=, [TOOL_CALLS].
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Condense comments in the Mistral tool-call rescue to contract essentials
* verify_import_hoist: exempt __future__ imports and same-diff relocations
Two false positives fired on this PR's refactor. A from __future__ import
is a compiler directive whose name never appears as a runtime load, so
HOISTED-IMPORT-UNUSED can never see it used, yet the file requires it for
PEP 604 annotations on Python 3.9. TARGET-CHANGED flagged the deliberate
move of the strip-pattern constants into core.inference.tool_call_parser
as a silent re-point even though the old module-level target was removed
and the new one added in the same diff. Both get narrow exemptions; a
re-point to a pre-existing target is still caught, and the self-test
negative controls all pass unchanged.
* Drain the whole Mistral [TOOL_CALLS] array in streaming passthrough healing
StreamToolCallHealer._drain promoted only the first parsed call per pass and
dropped the rest of the buffer past that one span. For a well-formed Mistral
parallel-tool-call array streamed through client-tool passthrough
([TOOL_CALLS][{...},{...}]), the per-item spans are contiguous, so after the
first call was promoted the residue began with ,{...}] (no leading signal) and
was flushed as raw text: every call after the first was lost.
_drain now walks the contiguous run of parsed calls (adjacent tiled spans =
one array), promoting each declared call and relaying undeclared ones as data,
and stops at the first gap (prose) or incomplete trailing block so separate
blocks still stream incrementally in document order. This mirrors the
non-streaming heal_openai_message / finalize promote-or-flush loop and the
server-side safetensors loop, which already handled multi-call arrays.
Added regression tests: 2-call array in one feed and char-by-char, an
undeclared middle call kept as text, and an array followed by trailing prose.
* Drain comma-less Mistral tool-call arrays and normalize null arguments
The array branch fed the whole body to a single json.loads, which rejects the
comma-less multi-call form the repo's own Mistral/Ollama templates render (the
range loop in ollama_template_mappers.py emits the objects with no separator) and
so dropped every call. Decode elements individually with the existing
comma-tolerant raw_decode helper, now _decode_array_items, which also returns the
objects, so all calls are recovered while the span tiling is unchanged.
Also normalize a non-object array argument such as arguments null to an empty
object, matching the wrapped tool_call path, instead of serializing None to the
string "null" that auto-heal would turn into a bogus query of "null".
* Gate safetensors reasoning prefill on the rendered generation prompt
reasoning_always_on fires on any paired <think></think> in the template,
including markup that only renders PAST assistant history (Kimi-K2-Thinking)
while the generation prompt opens no <think>. Starting the reasoning extractor
in prefilled mode there captured a normal answer entirely as reasoning_content
and returned blank visible content. Prefill only when rendering the generation
prompt actually leaves <think> open (DeepSeek-R1 / QwQ / Qwen3-Thinking);
history-only templates start the extractor in normal mode and parse the model's
own <think>...</think>. Adds a Kimi-shape regression test.
* Keep bare scalar Mistral array arguments raw instead of double-encoding
A scalar string argument in the canonical Mistral [TOOL_CALLS] array
(for example [TOOL_CALLS][{"name":"web_search","arguments":"weather"}])
was run through json.dumps, turning weather into the JSON string
"weather". The downstream argument healer then wrapped that quoted
form, so a single-string tool like web_search searched for the literal
"weather" with quotes. The <tool_call> path already keeps a scalar
argument raw; mirror it here so only a dict is serialized. Add a
regression test asserting both paths yield the same healed arguments.
* Tighten tool-call rescue and reasoning-prefill comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: tool calling for Llama-3, Mistral, Gemma 4 on safetensors + MLX (#5615)
Adds tool calling for Llama-3, Mistral (pre-v11 + v11+ + [ARGS]), and Gemma 4 to the safetensors / transformers and MLX backends. Parser patched against llama.cpp / vLLM / SGLang per-family parsers and normalises to OpenAI shape. 96 targeted unit tests + cross-OS staging CI (ubuntu / macos-14 / windows) green on the multi-format probe.
* studio: tool-call healing parity between safetensors / MLX and GGUF
After the multi-format parser landed in #5615, the safetensors / MLX
agentic loop and the GGUF loop still differed on healing behaviour.
This commit closes the gaps in both directions so the two backends
react the same way to identical model output.
Changes:
1. core/inference/llama_cpp.py -- the GGUF BUFFERING state machine
now wakes on every emission marker the shared parser knows. Was
("<tool_call>", "<function="); is now the five-tuple imported
from core.inference.tool_call_parser (Qwen / Qwen3.5 / Llama-3
<|python_tag|> / Mistral [TOOL_CALLS] / Gemma 4 <|tool_call>).
Stream cleanup is delegated to the same shared strip_tool_markup
so leaked markup from any family is removed from assistant
content.
2. core/inference/llama_cpp.py -- per-tool canonical heal key. When
a tool arguments field is a bare string and JSON parsing fails,
the GGUF path now heals to {"code": raw_args} for python,
{"command": raw_args} for terminal, and {"query": raw_args} for
everything else. Was hard-coded to {"query": raw_args}, which
silently routed every python / terminal emission through
web_search. Mirrors safetensors_agentic._CANONICAL_HEAL_ARG.
3. core/inference/safetensors_agentic.py -- re-prompt on plan-
without-action. When the model emits a short forward-looking
intent ("I'll search for that", "Let me check", "First, I
will...") and no tool call, the loop nudges the model to act
instead of silently returning a plan-only answer. Up to
_MAX_REPROMPTS=3 (matches GGUF). The intent regex, character
cap, and instruction text are byte-identical to the GGUF path.
The buffer-end fall-through is unified so a buffered intent
emission that never exits the BUFFERING state still triggers
the re-prompt.
4. core/inference/safetensors_agentic.py -- extra iteration slots
for re-prompts. The loop now budgets max_tool_iterations +
_MAX_REPROMPTS + 1 total iterations and tracks the tool-call
count separately, so a stalling model can be nudged 3x without
eating the caller's tool-call budget. Mirrors the _extra slot
reservation in the GGUF path.
Tests (14 new safetensors-side units; 5 GGUF parity pins):
TestLoopRePrompt -- intent-trigger, plain-answer,
no-tools, cap-at-three, budget
preserved, buffer-end intent.
TestLoopCanonicalHealKey -- python / terminal / unknown.
TestGGUFSafetensorsHealingParity -- shared markers used, shared
strip used, canonical heal keys
identical, intent regex matches
same phrases, _MAX_REPROMPTS
equal on both backends.
All 110 targeted tests pass locally; the broader tool / inference /
model-config / sandbox / anthropic / mlx suites stay green.
Why this matters
Without this parity, Llama-3.2 / Mistral / Gemma 4 emissions on Mac
(MLX) and Linux-safetensors stop the agentic loop as soon as the
model says "Let me...", because the GGUF re-prompt logic never
existed on these backends. The two-marker GGUF BUFFERING tuple also
let non-Qwen tool emissions stream out as plain prose when
llama-server's structured channel did not pick them up. Both paths
now drain the same way, heal the same way, and re-prompt the same
way -- so a tool call that works on GGUF works identically on
safetensors / MLX.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: fix tool-call parser bugs from gemini review on #5620
Three high-priority gemini findings on the tool-call parsing additions:
1. unicode_escape on UTF-8 bytes corrupts non-ASCII literals
(e.g. ✨ becomes â\x9c¨). Replace with json.loads on a quoted
string -- preserves emoji / CJK / RTL while still handling
\n \t \uXXXX escapes.
2. Llama-3 sentinel stripping is order-dependent. A leading
`<|eot_id|><|begin_of_text|>` left `<|begin_of_text|>` behind
because the loop had already passed that sentinel. Loop until
no sentinel matches at the start.
3. Mistral v11+ `[TOOL_CALLS] name { json }` regex uses non-greedy
`\{.*?\}` which truncates at the first `}` of a nested JSON
argument, leaking the tail (e.g. `}}`) into user-visible
streamed text. Same problem for the v0.3 array pattern with
nested brackets. Strip those with balanced brace/bracket
scanning via a new `_strip_mistral_closed_calls` helper called
from `strip_tool_markup`.
Also fix the inference routes' parallel `_TOOL_XML_RE`:
- Same nested-JSON truncation in the Mistral patterns; route the
strip through the parser's balanced-scan helper via a thin
`_strip_tool_xml` wrapper that all existing callers now use.
- Llama-3 `<|python_tag|>[^\n<]*` stopped at any `<`, leaking the
tail of any tool call whose argument contained a literal `<`
(queries, code snippets). Relax to `[^\n]*` which keeps the
strip confined to the actual end-of-line.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tool calling for DeepSeek (R1/V3/V3.1), GLM 4.x, Kimi K2
Adds three more emission-family parsers to tool_call_parser.py so the
shared safetensors / MLX / GGUF agentic loop covers the major open-
weight reasoning families. Patterns ported from llama.cpp
(common/chat-parser.cpp legacy pre-PEG branch), vLLM
(tool_parsers/deepseekv3*, glm4_moe, kimi_k2), and SGLang
(function_call/deepseekv31_detector, glm4_moe_detector, kimik2_detector).
All three references are MIT (llama.cpp) or Apache-2.0 (vLLM, SGLang).
Formats covered:
DeepSeek R1 <｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function
<｜tool▁sep｜>NAME\n```json\n{...}\n```<｜tool▁call▁end｜>
<｜tool▁calls▁end｜>
-- args wrapped in a Markdown json fence, ``function``
literal prefix per llama.cpp common_chat_parse_
deepseek_r1 (chat-parser.cpp:801-820)
DeepSeek V3/V3.1
<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>NAME
<｜tool▁sep｜>{json}<｜tool▁call▁end｜><｜tool▁calls▁end｜>
-- bare JSON, no code fence, no ``function`` prefix
per llama.cpp common_chat_parse_deepseek_v3_1
(chat-parser.cpp:822-879)
GLM 4.5/4.6/4.7 <tool_call>NAME\n<arg_key>k1</arg_key>
\n<arg_value>v1</arg_value>...</tool_call>
-- strings raw, non-strings JSON-encoded per
chat_template.jinja; multi-call is back-to-back
blocks. Per llama.cpp common_chat_parse_glm_4_5
(chat-parser.cpp:1040-1052)
Kimi K2 <|tool_calls_section_begin|><|tool_call_begin|>
functions.NAME:IDX<|tool_call_argument_begin|>{json}
<|tool_call_end|><|tool_calls_section_end|>
-- bare name recovered by stripping ``functions.``
prefix and ``:IDX`` suffix; full id preserved as
tool_calls[i].id so the roundtrip replays verbatim.
Per llama.cpp common_chat_parse_kimi_k2
(chat-parser.cpp:896-913)
Marker collisions
GLM uses the same ``<tool_call>`` opener as Qwen but with a bare
function name + ``<arg_key>`` body (Qwen has ``\s*{`` after the tag).
The dispatch keeps Qwen first; Qwen's _TC_JSON_START_RE returns no
matches on a GLM emission, so the fall-through to _parse_glm_tool_
calls handles it correctly. Existing Qwen tests confirm zero
regression.
Streaming buffer
TOOL_XML_SIGNALS extended from 5 markers to 12 so the BUFFERING state
machine wakes on every new family's section opener. Added the
DeepSeek alternative markers (ASCII underscores, short ``<｜tool▁calls｜>``
form) because real checkpoints emit those variants.
Strip patterns
_TOOL_CLOSED_PATS adds DeepSeek envelope (``<｜tool▁calls▁begin｜>...
<｜tool▁calls▁end｜>``) and Kimi section (``<|tool_calls_section_begin|>
...<|tool_calls_section_end|>``). _TOOL_ALL_PATS adds the same plus
the unclosed-tail variants so a truncated stream does not leak
markup.
Route gate
_detect_safetensors_features._PARSER_MARKERS grows to include
DeepSeek and Kimi markers plus ``<arg_key>`` (the unique GLM signal).
_TOOL_XML_RE (the route-layer markup-strip regex) gets DeepSeek and
Kimi closed-pair patterns. _TOOL_TEMPLATE_MARKERS in llama_cpp.py
adds ``message['role'] == 'tool'``, ``message['tool_calls']``, and
``tool_calls is defined`` so the classifier recognises DeepSeek's
subscripted-access template style (it has no top-level
``{% if tools %}`` block).
Tests (39 new):
TestParserDeepSeek (7) -- R1 fence, short-form opener, V3.1 bare,
multi-call, with-reasoning, strip,
signal-wakes-streaming
TestParserGLM (6) -- single, mixed types, multi-call,
unclosed-heal, no-Qwen-regression, strip
TestParserKimi (6) -- single, multi-call, dotted-name, unclosed,
strip, signal-wakes-streaming
TestParserCrossFormatRouting (2) -- dispatch routing, signal coverage
TestLoopBasic loop integration (3) -- DeepSeek / GLM / Kimi end-to-end
Capability advertise (3) -- DeepSeek / GLM / Kimi templates flip
supports_tools=True
All 398 targeted tests pass locally (115 safetensors + 27 capability
+ rest of tool / inference / sandbox / model-config suites). Builds
on PR #5620 (parser + healing parity for Llama-3 / Mistral / Gemma 4);
will rebase cleanly onto main once #5620 lands. PR opened as draft -
do not merge until validated against real models for each family.
Sources
- llama.cpp common/chat-parser.cpp lines 801-913, 1040-1052 (MIT)
- vLLM vllm/tool_parsers/deepseekv31_tool_parser.py (Apache-2.0)
- vLLM vllm/tool_parsers/glm4_moe_tool_parser.py (Apache-2.0)
- vLLM vllm/tool_parsers/kimi_k2_tool_parser.py (Apache-2.0)
- SGLang python/sglang/srt/function_call/{deepseekv31,glm4_moe,kimik2}_
detector.py (Apache-2.0)
- Live chat templates: deepseek-ai/DeepSeek-V3.1, zai-org/GLM-4.6,
moonshotai/Kimi-K2-Instruct, unsloth/DeepSeek-V3-0324,
unsloth/GLM-4.5-Air, unsloth/Kimi-K2-Instruct
* studio/routes: make python_tag strip multi-line aware
Earlier revisions of _TOOL_XML_RE in studio.backend.routes.inference
oscillated between two bug shapes:
5615 r"<\|python_tag\|>[^\n<]*" -- stopped at any literal "<"
so code='if x < 10: pass'
leaked '< 10: pass)' to the
user.
5620.1 r"<\|python_tag\|>[^\n]*" -- single-line only; the second
line of
python.call(code="a\nb")
leaked.
The full parser (_parse_llama3_python_tag) already handles both via
balanced-brace scanning, so the parsing path was fine; the LEAK was
in the streaming strip path that runs on every cumulative emission
while content is still arriving.
Switch to r"<\|python_tag\|>(?:[^<]|<(?!\|))*" so the strip consumes:
* any character that is not a "<" (newlines, JSON, code, ...),
* a "<" only when it is NOT followed by "|" (i.e. NOT a Llama-3
sentinel start like <|eot_id|>, <|eom_id|>, <|begin_of_text|>).
This means:
* code='if x < 10' stays inside the strip (5615 fix preserved),
* multi-line code stays inside the strip (5620 round 2),
* the strip terminates at the next Llama-3 sentinel so trailing
assistant content survives.
Tests: TestRoutesPythonTagStrip (8 cases)
pytest test_safetensors_tool_loop.py test_safetensors_capability_advertise.py
-> 118 passed in 1.81s (was 110).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: review follow-ups for DeepSeek / GLM / Kimi tool calling
Four fixes addressing review of the parent commit:
1. GLM <arg_value> coercion: tighten the
json.loads -> ast.literal_eval -> raw cascade to only deserialize
when the body unambiguously looks like a JSON literal (object,
array, JSON-encoded string, true/false/null, or numeric). Strings
like ``True`` / ``None`` (Python literals, not JSON) and arbitrary
prose now stay raw. The bare-numeric / bare-boolean ambiguity with
string args remains an inherent limitation of the template without
schema access -- documented in the new comment. Drops the ast
import entirely (closes Gemini's :1036 suggestion).
2. Kimi K2 bare-counter ids (e.g. ``<|tool_call_begin|>3``) are now
dropped rather than surfaced as a tool literally named "3". Matches
vLLM behaviour; SGLang's schema-infer fallback is out of scope at
the parse site. Real Kimi K2 emissions use ``functions.NAME:IDX``
so this is the exception path.
3. Restore the elaborate ``<|python_tag|>(?:[^<]|<(?!\|))*`` clause in
routes.inference._TOOL_XML_RE -- the simpler ``[^\n<]*`` form
regressed PR #5620's multi-line / literal-``<`` python_tag fix.
Restore ``TestRoutesPythonTagStrip`` (8 tests) adapted to call
``_TOOL_XML_RE.sub`` directly since the ``_strip_tool_xml`` helper
was inlined this PR.
4. Add the spaced and backslash-escaped DeepSeek opener variants
(``<｜tool calls begin｜>``, ``<｜tool\_calls\_begin｜>``) to
``TOOL_XML_SIGNALS`` for streaming-gate parity with
``_DEEPSEEK_BEGIN_RE``.
Also updates the llama.cpp / vLLM citations in the parser docstrings:
``common/chat-parser.cpp`` was split into ``common/chat.cpp`` +
``common/chat-peg-parser.cpp`` by llama.cpp PR #18675, and vLLM
moved the tool parsers from ``vllm/entrypoints/openai/tool_parsers/``
to ``vllm/tool_parsers/``. Pin to pre-refactor commit ``51fa458a92d6``
where the cited line numbers still resolve.
New regression tests in ``test_pr5624_regressions.py`` cover the GLM
coercion heuristic shapes, GLM literal-``<`` in arg_value, Kimi K2
dotted name, Kimi K2 bare-counter drop, DeepSeek V3.1 truncated
mid-stream, and routes-layer strip across all three new families.
Tests:
pytest studio/backend/tests/test_safetensors_tool_loop.py
studio/backend/tests/test_safetensors_capability_advertise.py
studio/backend/tests/test_pr5624_regressions.py -q
-> 170 passed in 1.91s
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten verbose comments in tool-call parser sections
Comments were narrating what the code already says. Cut historical
"earlier revisions used X, then Y" narratives down to one-line WHY
notes where the footgun still matters (canonical heal-key parity,
balanced-brace vs non-greedy regex, ``(?:[^<]|<(?!\|))*`` over
``[^\n<]*``/``[^\n]*``). Drop section-header banners.
No behaviour change. Re-ran:
pytest studio/backend/tests/test_safetensors_tool_loop.py \
studio/backend/tests/test_safetensors_capability_advertise.py -q
-> 118 passed.
Regression replay (parser + _coerce_arguments on the 5 #5615 inputs)
-> 21/21.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: GLM 4.7 no-newline emission + Kimi multi-section parity
Two fixes surfaced by triple-confirm verification against the live
HF chat templates and upstream llama.cpp / vLLM / SGLang parsers.
1. GLM 4.7 silent drop
``zai-org/GLM-4.7/chat_template.jinja`` line 65 uses
``{{- '<tool_call>' + tc.name -}}`` which Jinja strips trailing
whitespace from, so the first ``<arg_key>`` follows the function
name with NO ``\n`` between them. Real emissions look like
``<tool_call>get_weather<arg_key>city</arg_key><arg_value>London
</arg_value></tool_call>``. The previous ``_GLM_TC_OPEN_RE`` ended
the name with ``\n`` so GLM-4.7 calls were silently dropped
(parser returned ``[]``).
Fix: relax the name terminator to a lookahead that accepts EITHER
``\n`` OR the next ``<arg_key>``:
_GLM_TC_OPEN_RE = re.compile(
r"<tool_call>\s*([^\n<{][^\n<]*?)\s*(?=\n|<arg_key>)"
)
The first-char restriction ``[^\n<{]`` still excludes Qwen's
``<tool_call>{json}`` form so the Qwen-vs-GLM dispatch remains
mutually exclusive.
2. Kimi multi-section parity with vLLM / SGLang
``vllm/tool_parsers/kimi_k2_tool_parser.py`` and SGLang's
``kimik2_detector.py`` both use ``re.findall`` and so collect every
``<|tool_calls_section_begin|>...<|tool_calls_section_end|>`` block
in a single stream. The previous implementation stopped at the
first ``<|tool_calls_section_end|>``. Kimi K2 doesn't emit
multi-section in practice, but parity is cheap.
Fix: wrap the existing per-call body parser in an outer loop that
advances past each ``<|tool_calls_section_end|>`` and continues to
the next ``<|tool_calls_section_begin|>``. Body parsing extracted
to ``_parse_kimi_section_body`` for clarity. Truncated final
section is still surfaced via the existing in-body balanced-brace
walk.
Verified independently against the live HF templates:
* GLM-4.7 emission constructed from the live template parses to the
expected ``{name, arguments}`` shape.
* GLM-4.5 / 4.6 newline shape continues to parse (the lookahead also
matches ``\n``).
* Qwen ``<tool_call>{json}`` still dispatches to the Qwen path -- the
first-char restriction stops the GLM regex from biting JSON bodies.
* Kimi two-section stream surfaces both calls in order with full ids
preserved.
* Bare-counter Kimi ids still drop.
Tests added in ``test_pr5624_regressions.py``:
* ``test_glm_4_7_no_newlines_between_name_and_arg_key``
* ``test_glm_4_7_no_newlines_multi_call``
* ``test_glm_4_7_does_not_break_qwen_path``
* ``test_kimi_two_sections_in_one_stream_both_parse``
pytest studio/backend/tests/test_safetensors_tool_loop.py
studio/backend/tests/test_safetensors_capability_advertise.py
studio/backend/tests/test_pr5624_regressions.py -q
-> 174 passed in 1.93s
pytest studio/backend/tests/ -q -k 'not gpu and not llama_cpp_integration'
-> 2038 passed, 15 failed (pre-existing CI gaps).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: parser robustness fixes for PR #5620
Three surgical extensions to the multi-format tool-call parser, each
covering a real fine-tune / template emission shape that the current
parser silently drops. No path narrows; all changes widen what is
accepted.
1. `_parse_tool_call_json` now accepts both `arguments` and
`parameters` keys. A Hermes / Qwen `<tool_call>{json}</tool_call>`
wrapper around a Llama-3.2 fine-tune that emits the `parameters`
key was extracting the tool name and silently discarding the
args, producing a working-shaped call with an empty payload. The
bare-JSON and python_tag paths already accepted both keys; this
path now matches them.
2. `_TC_FUNC_START_RE`, `_TC_PARAM_START_RE`, and `_TC_PARAM_CLOSE_RE`
now also match the attribute form
`<function name="..."><param name="...">v</param></function>` used
by MiniCPM-5 and MiniMax-M2. Names land in either capture group,
and `</param>` is accepted as a short close.
3. `_parse_llama3_bare_json` sentinel-strip now consumes the role
label inserted between `<|start_header_id|>` and
`<|end_header_id|>` by Meta's official Llama-3.x chat template.
Without this, every assistant turn re-fed through the template
prefix `<|start_header_id|>assistant<|end_header_id|>\n\n{json}`
parsed to zero calls, so any history-with-tool-call round-trip
in production silently dropped.
Tests in `studio/backend/tests/test_safetensors_tool_loop.py`:
* `TestParserRobustness::test_tool_call_json_accepts_parameters_key`
* `TestParserRobustness::test_function_xml_attribute_form`
* `TestParserRobustness::test_function_xml_attribute_form_multi_param`
* `TestParserRobustness::test_function_xml_legacy_equals_form_still_works`
(regression guard for the existing `<function=name>` syntax)
* `TestParserRobustness::test_llama3_chat_template_round_trip`
* `TestParserRobustness::test_llama3_round_trip_all_roles`
* `TestParserRobustness::test_llama3_round_trip_with_eot_prefix`
`pytest studio/backend/tests/test_safetensors_tool_loop.py
studio/backend/tests/test_safetensors_capability_advertise.py -q`
goes from 118 to 125 passed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim verbose comments in tool-call parser sections for PR #5624
Pure comment / docstring tightening on top of the GLM 4.7 + Kimi
multi-section fixes. No behavioural change.
* Drop multi-paragraph prelude and post-refactor citation chatter in
the DeepSeek, GLM and Kimi parser docstrings; keep the shape and
upstream-commit pin.
* Collapse ``parse_tool_calls_from_text``'s 9 per-family blocks into
a single ordered loop with one combined comment.
* Tighten the GLM coercion, Kimi bare-counter and ``_TOOL_XML_RE``
comments to one or two lines each.
* Same trim pass on ``_PARSER_MARKERS`` and the regression-test
docstrings.
Tests:
pytest studio/backend/tests/test_safetensors_tool_loop.py
studio/backend/tests/test_safetensors_capability_advertise.py
studio/backend/tests/test_pr5624_regressions.py -q
-> 174 passed in 2.00s
* Fix O(N^2) DeepSeek V3.1 backtracking for PR #5624
Adversarial input ``<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>fn<｜tool▁sep｜>``
followed by a long body that does NOT contain a closing brace caused
the V3 path's ``([^\n<]+?)<｜tool▁sep｜>`` regex to backtrack
quadratically: at each position the lazy quantifier extends one char
at a time looking for a sep that isn't there, taking ~19s on 50k
chars.
Replace the regex search with ``str.find`` on the sep marker plus a
left-walk to recover the name. ``str.find`` is O(N); the walk stops
on ``\n`` (turn boundary), ``<`` (start of a tag), or ``>`` (end of
an optional ``<｜tool▁call▁begin｜>`` prefix). Same observable
behaviour as the regex on every canonical input.
Tests:
test_deepseek_v3_1_huge_truncated_body_is_linear (new) -- 50k chars
must parse in < 1s.
pytest studio/backend/tests/test_safetensors_tool_loop.py
studio/backend/tests/test_safetensors_capability_advertise.py
studio/backend/tests/test_pr5624_regressions.py -q
-> 175 passed in 1.97s
pytest studio/backend/tests/ -q -k 'not gpu and not llama_cpp_integration'
-> 2038 passed, 15 pre-existing failures unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: terminate function-XML body at </function>, not just </tool_call>
`_parse_function_xml` was looking for `</tool_call>` (the Hermes
wrapper) as the body terminator. When a model emits a standalone
`<function=NAME><parameter=K>v</parameter></function>` followed by
explanatory prose (which models routinely do), no `</tool_call>` is
present, so the body extended to end-of-string and the trailing
prose leaked into the LAST parameter value.
Pre-existing on main (the legacy `<function=NAME>` form had this
bug too). Same affects PR #5620's new attribute-form
`<function name="NAME"><param name="K">v</param></function>`
emission used by MiniCPM-5 / MiniMax-M2.
Fix: `_TC_END_TAG_RE` now matches either `</tool_call>` OR
`</function>`. The existing `_TC_FUNC_CLOSE_RE` / `_TC_PARAM_CLOSE_RE`
strips are unchanged. Multi-call inputs still bound each function
at the next `<function=` start, so no over-eager consumption.
New tests:
* `test_function_xml_followed_by_prose` (legacy form + prose)
* `test_function_attribute_xml_followed_by_prose` (attribute form + prose)
Existing `test_code_with_embedded_xml` still passes (a parameter
value containing literal `<a></a>` is preserved because the
embedded close tag is `</a>`, not `</function>`).
`pytest studio/backend/tests/test_safetensors_tool_loop.py
studio/backend/tests/test_safetensors_capability_advertise.py -q`
goes from 125 to 127 passed.
* Studio: tighten Llama-3.2 bare-JSON guard
A fuzz pass on PR #5811 turned up that ``_parse_llama3_bare_json``
accepted ``parameters`` as a string, contradicting the docstring's
"parameters or arguments is a dict" guard. Prose JSON like
``{"name":"foo","parameters":"a sentence"}`` would wrongly fire the
parser, which the agentic loop would then heal into a real
``foo(query="a sentence")`` call.
Same code lives on this branch, so the same fix applies here.
Tightened guard:
- ``parameters`` must be a dict (Llama-3 spec).
- ``arguments`` may be a dict, or a JSON-encoded string that
decodes to a dict (OpenAI shape, e.g.
``"arguments":"{\"q\":\"x\"}"``). Plain non-JSON strings or
JSON-strings of lists / scalars / null no longer pass.
Mirrors the fix landed in PR #5811 commit 615b8608. Adds the same
4 regression tests under TestParserMultiFormat.
Existing test suite stays green: 127 -> 131 passing.
* Studio: skip non-scalar args in python_tag JSON form
The JSON sub-path of ``_parse_llama3_python_tag`` was fabricating
``{"value": args}`` when the model emitted a non-dict / non-string
``arguments`` value (e.g. ``42``, ``[1,2,3]``, ``null``, ``true``).
This silently turned a malformed emission into a real tool call,
which the agentic loop would then execute with arguments the model
never intended.
Tightened: skip the call instead of fabricating. The same
behaviour now matches the bare-JSON guard tightened earlier
(strict-guard merge from PR #5620, inherited via merge here).
Added a regression test covering the four non-scalar shapes.
Pass count on this branch: 158 -> 159.
Sites in ``_parse_tool_call_json`` and ``_consume_mistral_call``
keep the existing looser behaviour for now; both are reached
only after explicit ``<tool_call>`` / ``[TOOL_CALLS]`` markers
so the false-positive surface there is much narrower.
* studio: fix safetensors tool-call parser gaps vs llama.cpp (Mistral CALL_ID / THINK, attribute-form signal)
Three GGUF-parity fixes to the safetensors tool-call parser, each matching
llama.cpp's reference behaviour:
- Mistral Small 3.2 emits [TOOL_CALLS]name[CALL_ID]<id>[ARGS]{json}. The
parser stopped after the name on seeing [CALL_ID] (neither [ARGS] nor {),
dropping the call. Skip an optional [CALL_ID]<id> segment in both the
parse and strip paths. llama.cpp parses this (test-chat.cpp:4785).
- Magistral wraps reasoning in [THINK]...[/THINK]. A [TOOL_CALLS] inside the
reasoning was parsed as a real call, producing a phantom call. Strip a
leading [THINK] block before scanning so only the post-reasoning call
counts (test-chat.cpp:2285); a literal [THINK] inside a later argument is
left intact.
- The standalone MiniCPM-5 / MiniMax-M2 <function name="..."> attribute form
parsed correctly but was absent from TOOL_XML_SIGNALS and the markup strip
patterns, so the streaming safety-net parse was gated off (dropping the
call) and markup leaked into displayed text. Add the signal and broaden
the strip regexes.
Adds regression tests for all three.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: fix GLM and Kimi K2 safetensors tool-call parser gaps vs llama.cpp
Four GGUF-parity fixes for the GLM and Kimi K2 families:
- GLM 4.7 zero-argument inline call <tool_call>name</tool_call> was dropped:
the open-tag lookahead only allowed \n or <arg_key> after the name. Allow
</tool_call> too so a no-arg call parses to empty args (vLLM / SGLang /
llama.cpp all parse it).
- GLM string argument values were stripped, losing significant leading /
trailing whitespace in code / diff arguments. Keep the raw value for the
string fallback and only strip the copy used to probe for a JSON literal,
matching vLLM glm4_moe which never strips string args.
- Kimi K2 calls emitted without the <|tool_calls_section_begin|> wrapper
were dropped. llama.cpp makes the section optional (Kimi can call a tool
straight after reasoning without opening a section); parse a bare
<|tool_call_begin|> when no section is present.
- Kimi K2 malformed / truncated JSON in one call dropped every later call in
the section. Skip the bad call and keep parsing so valid subsequent calls
are recovered (vLLM parity).
Adds regression tests for all four.
* studio: fire safetensors tool calls for the bare-JSON (Llama-3.2) form
The agentic loop's streaming safety-net parse was gated on
has_tool_signal(), which is False for the Llama-3.1 / 3.2 bare-JSON tool
form {"name":..,"parameters":..} (no XML marker). Real tool calls were
therefore dropped: the loop logged "model planned without calling tools",
re-prompted three times, then gave up with zero tool calls, while GGUF's
llama-server parses the same emission natively.
Run parse_tool_calls_from_text() unconditionally in the safety net. The
parser is strict (only fires on a valid tool-call shape) so plain answers
are unaffected. Reproduced on a real unsloth/Llama-3.1-8B-Instruct run:
the model emits {"name":"web_search","parameters":{...}} which now
executes the tool instead of being re-prompted into a no-op.
Adds a loop regression test for the bare-JSON form.
* studio: fire safetensors tool calls for Gemma 4 (native template + stripped parser)
Gemma-4 safetensors fired no tools while its GGUF fired reliably. Three gaps:
- The Studio swaps in the Unsloth "gemma-4" chat template, which does not
render the tools schema (the model's native template does), so the model
never saw the tools. Fall back to the model's native template when the
override template renders identically with and without tools. Same fix
helps any family whose override template drops tools.
- skip_special_tokens strips the <|tool_call> wrapper and <|"|> string
markers, so a streamed Gemma-4 call arrives as a bare call:NAME{k:v, ...}
with unquoted values. Parse that form, keeping commas/braces inside a
code or command value, normalising surrounding quotes, and stripping the
leaked markup from the final answer.
- Without a grammar a small model can loop, repeating one call for the whole
tool budget. Collapse exact-duplicate calls within a turn and force a final
answer after a turn that made no new tool progress (llama-server's lazy
grammar prevents this loop on the GGUF side).
Adds parser tests for the bare/stripped Gemma-4 form.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: complete strict-mode contract and fix parser import paths
Address review findings on the multi-format tool-call parser:
- Honor allow_incomplete=False in the remaining sub-parsers. The Llama-3
<|python_tag|>NAME.call(...) parser, the pre-v11 Mistral [TOOL_CALLS] array
parser, and the Gemma 4 <|tool_call> parser ignored strict mode, so a
truncated call (missing closing paren, ], or <tool_call|>) was still healed
and executed with Auto-Heal disabled. Thread strictness through and reject
the unclosed forms, matching the JSON and function-XML paths.
- Drop the duplicate tool_call_parser import block in llama_cpp.py and the
redundant un-aliased TOOL_XML_SIGNALS; only the _SHARED_TOOL_XML_SIGNALS
alias is used as a value.
- Import _strip_mistral_closed_calls from core.inference.tool_call_parser in
routes/inference.py instead of studio.backend.core... The self-contained
run.py launch mode only puts studio/backend on sys.path, so the absolute
package path raised ModuleNotFoundError on the server-tool strip path.
Add strict-mode regression tests for the truncated Llama-3 dot-call and the
unclosed Mistral array.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden DeepSeek/Kimi tool-call parsing and strip
Address review findings on the DeepSeek and Kimi parsers:
- Honor allow_incomplete=False for DeepSeek. An envelope with no closing
<｜tool▁calls▁end｜> is truncated mid-stream; reject it in strict mode
instead of healing the body out to EOF, matching the strict XML and Mistral
paths.
- Do not skip a following tool call when the current call's end marker is
missing. The DeepSeek V3 and Kimi loops advanced by searching forward for the
next <｜tool▁call▁end｜> / <|tool_call_end|>, which could land on a later
call's end marker and drop the call in between. Advance by the JSON end; the
loop re-locates the next call marker from there.
- Strip truncated DeepSeek and Kimi section blocks in the route-level display
regex. The patterns required the closing marker; add the end-of-text
alternative so a block truncated by EOS does not leak raw markup to the UI.
Add regression tests for the truncated DeepSeek envelope, and for DeepSeek and
Kimi multi-call recovery when the first call's end marker is missing.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: preserve XML param indentation and alias Mistral array parameters
Two parser-correctness fixes found by auditing against the model chat templates
and the SGLang / vLLM reference parsers:
- Qwen3.5 XML parameter values lost their leading indentation. The chat template
emits <parameter=k>\nVALUE\n</parameter>, but the parameter-start regex ate the
wrapping newline AND the value's first-line indentation with a trailing \s*,
then str.strip() removed the rest. Narrow the trailing class to horizontal
whitespace only and trim exactly one wrapping newline (via _trim_param_value),
preserving indentation in code/diff arguments. Matches SGLang's qwen3_coder
detector. Applies to both _parse_function_xml (tool_call_parser.py) and the XML
path in tool_healing.py.
- Mistral pre-v11 array objects keyed on parameters dropped their payload.
_consume_mistral_call read only the arguments key; alias parameters the same way
the JSON/XML paths and SGLang's base detector do.
Add regression tests for preserved multi-line indentation and the array
parameters alias.
* Studio: DeepSeek strip sync, Gemma nested args, GLM/Kimi strict mode
Parser-correctness fixes found by auditing DeepSeek/GLM/Kimi against vLLM,
SGLang, and the model chat templates:
- DeepSeek: the short <｜tool▁calls｜> opener (and the space / escaped-underscore
spellings) was parsed but never stripped, so a short-opener envelope leaked raw
markup to the UI. Share one opener alternation between _DEEPSEEK_BEGIN_RE and
the strip patterns (and the route-level display regex) so a signal we parse can
never be left un-stripped.
- Gemma wrapper-less stream: a nested object/array argument (loc:{city:NYC},
labels:[bug,ui]) was kept as a literal string. Parse it recursively when the
bare value is a balanced {} / [], falling back to the raw string for a
truncated value.
- GLM and Kimi ignored allow_incomplete. With Auto-Heal off, a GLM block with no
</tool_call>, a Kimi section with no <|tool_calls_section_end|>, or a Kimi call
with no <|tool_call_end|> are truncated and must be rejected, matching the
strict behavior of the JSON/XML/Mistral/DeepSeek paths and vLLM/SGLang.
Add regression tests for the short-opener strip, the Gemma nested args, and GLM /
Kimi strict-mode rejection.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten tool-call parser comments
Make the comments in the multi-format tool-call parser and its callers succinct:
compress verbose docstrings/blocks to one or two lines, drop ones that restate the
code, and trim the tiny balanced-scanner helpers. Correctness rationale and
upstream provenance (SGLang/llama.cpp parity, the strict-mode / Auto-Heal
contract, whitespace-preservation, and the Unicode / full-width-pipe notes) are
kept in compact form.
Comment-only: no code or behavior change (verified with comment_tools.py check
--strip-docstrings; parser suite green).
* Studio: tighten DeepSeek/GLM/Kimi parser comments
Compress the comments added for the DeepSeek/GLM/Kimi parsers and the Gemma
wrapper-less helpers to one or two lines, keeping the upstream provenance
(llama.cpp 51fa458a92d6), the O(N^2) / strict-mode rationale, and the vLLM parity
notes intact.
Comment-only: no code or behavior change (verified with comment_tools.py check
--strip-docstrings; parser suite green).
* Studio: make DeepSeek R1 / GLM parsing linear and close routes strip gaps
Review follow-up for the DeepSeek/GLM/Kimi parser:
- DeepSeek R1 detection used a greedy ``([^\n]+)\n```json`` regex that backtracks
O(N^2) on a fence-less truncated body; scan with str.find instead (mirrors the
V3 path).
- GLM arg pairs used a lazy-group finditer that rescanned to EOF from each bare
<arg_key> in an unclosed body (O(N^2)); walk pairs with str.find.
- The route display strip (_TOOL_XML_RE) accepted fewer DeepSeek openers than the
parser (missed the space / escaped-underscore spellings) and missed bare
section-less Kimi calls, so a call we parse could leak raw markup to the UI.
Reuse the parser's shared _DEEPSEEK_OPEN_RE_SRC and add a bare-Kimi arm.
Add ReDoS-linearity regressions for the R1 and GLM paths, a positive R1
fenced-json parse test, and routes-strip tests for the space/escaped DeepSeek
openers and the bare Kimi call.
* Studio: fix test_mcp_servers _TOOL_XML_RE reconstruction after _DS_OPEN_SRC reuse
The routes strip fix made _TOOL_XML_RE reference the module-level
_DS_OPEN_SRC variable. test_mcp_servers reconstructs the regex by exec-ing
the extracted compile() source in a namespace that only defined _re, so it
raised NameError. Inject _DS_OPEN_SRC into that namespace, matching the same
fix already applied in test_tool_xml_strip.
* Studio: make Llama-3 .call and Mistral-array healing parsing linear
Two more O(n^2) ReDoS paths in the multi-format parser, both reachable from
the agentic loop on a long truncated body with no length cap:
- _LLAMA3_KV_RE.finditer over a .call(...) body retried at every offset of a
long word run / unterminated quote (40K -> 14s). Replace with a hand-scan
that reuses the same key/number/literal sub-regexes via anchored match and
walks the string body by hand, so an unterminated quote is O(n). Verified
byte-identical to the old regex over 200K fuzzed inputs.
- _parse_mistral_array healing ran _balanced_brace_end from every { in the
body (20K -> 17s). Walk top-level objects, advancing past each balanced
{...}; this also drops the phantom call the old scan emitted from a nested
argument object.
Add adversarial-length linearity regressions plus positive .call kwargs and
unclosed-array recovery coverage.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strengthen #5624 regression assertions and strip-test harness guards
- test_strip_tool_markup_handles_deepseek_envelope used `A or B` where B was the
preservation property the next line already asserts, masking the real check.
Replace with an explicit assertion that the call name and args are stripped.
- The test_tool_xml_strip source-extraction harness reconstructs _TOOL_XML_RE and
_strip_tool_xml_for_display from routes/inference.py via lazy regexes that could
silently grab a shorter slice. Assert the extracted regex carries the DeepSeek /
bare-Kimi arms and the helper body reached the _TOOL_XML_RE.sub call.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: honor strict mode in safety-net, keep empty Gemma args, strip attribute-form function XML
- safetensors safety-net parser now forwards allow_incomplete=auto_heal_tool_calls,
matching the draining path, so a late incomplete tool call is not healed and
executed when Auto-Heal is off.
- Gemma empty bare value ({k:}) now serialises as "" instead of invalid {"k":},
which previously dropped the whole call.
- Route _TOOL_XML_RE also strips the <function name="..."> attribute form
(MiniCPM-5 / MiniMax-M2) so it no longer leaks to the UI.
* Studio: linearize wrapper-less Gemma nested-arg parsing and correct parser provenance
- _gemma_parse_value/_gemma_parse_mapping/_gemma_parse_array now parse nested
{}/[] in a single forward pass instead of pre-scanning each subtree with a
balanced-brace walk and re-parsing it. Deeply nested wrapper-less Gemma args
were O(n^2); they are now ~linear (and ~40x faster at depth 400).
- Correct the DeepSeek/GLM/Kimi provenance comments: the cited commit
51fa458a92d6 is unrelated, and GLM/Kimi were never standalone
common_chat_parse_* functions (llama.cpp uses common_chat_params_init_glm_4_5
plus a generalized XML parser, PRs #15904 / #16932).
- Add tests: Gemma deep-nesting linearity, nested object/array preservation,
same-turn distinct-call cap, and the native-template tool-render fallback.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: guard Gemma value parser against non-advancement and missing tokenizer
Addresses Gemini review:
- _gemma_parse_value now consumes one character when a stray }/]/, sits where a
value is expected, so _gemma_parse_array can never stall at the same index on
malformed input (a latent infinite loop).
- _render_with_native_template returns None when neither a tokenizer nor a
processor is present instead of raising AttributeError.
- Tests for both.
* Studio: fix attribute-form function-XML literal close tag and zero-arg strict call
Addresses Codex review of the <function name="..."> attribute form in
_parse_function_xml (MiniCPM-5 / MiniMax-M2):
- End the call body at the LAST </function> / </tool_call> within the call's
window, so a literal close tag inside a code/search argument (e.g.
print("</function>")) is preserved instead of truncating the call.
- Accept a closed call with no parameters as a valid zero-argument call in strict
mode (the function close is already required), instead of rejecting it as a
truncated call.
- Tests for both, mirroring the legacy <function=...> coverage.
* Studio: drop scratch review/planning artifacts from the branch
* Studio: fix tool-call parser/loop review findings on the multi-format path
Address the live code-review findings on the safetensors/MLX + GGUF tool path:
- routes: include the attribute form <function name="..."> in the safetensors
capability whitelist so MiniCPM-5 / MiniMax-M2 templates keep the tool pill
(parser already handles the form; the post-filter wrongly suppressed it).
- safetensors loop: build the plan-without-action re-prompt from the active
tools instead of a hardcoded web_search/python string, and gate it on
auto_heal_tool_calls, matching the GGUF loop.
- safetensors loop: hold a leading bare-JSON object ({"name":..,"parameters":..})
during BUFFERING until it closes, then drain it as a tool call instead of
streaming the raw JSON to clients. The DRAINING/STREAMING resolvers still
recover a plain JSON answer, so this can never drop content.
- parser: anchor the Llama-3 <|python_tag|>NAME.call(...) scan to the tag and
chain ; -separated calls, so all semicolon-separated built-ins parse and a
literal <|python_tag|>x.call(...) inside a JSON string argument no longer
fires the wrong tool.
- parser: consume the optional trailing </s> after a named Mistral
[TOOL_CALLS]name{json} call, mirroring the array shape.
- GGUF streaming strip: use the shared parser patterns (which know
[TOOL_CALLS] and <|python_tag|>) so a textual tool call entering DRAINING is
stripped instead of leaking the marker to streaming clients.
- routes: hoist the _strip_mistral_closed_calls import to module level.
Adds regression tests covering each fix; existing parser suite stays green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix DeepSeek/GLM/Gemma tool-call review findings
Address the live code-review findings specific to the DeepSeek / GLM / Kimi
and native-template additions:
- parser: in strict mode (Auto-Heal off) require the per-call
<｜tool▁call｜end｜> terminator for DeepSeek V3 calls instead of executing on
a bare balanced object closed only by the envelope end.
- parser: keep GLM string arguments that begin with a quote verbatim (drop
the leading-quote case from the JSON-decode probe) so a quoted search query
is not decoded down to its inner text.
- parser: reject a GLM call with an unclosed <arg_value> in strict mode, and
under Auto-Heal keep the partial value rather than dropping it to a no-arg
call.
- parser: add a balanced wrapper-less Gemma strip (call:NAME{...}) so a nested
object/array argument is removed whole instead of leaving a trailing brace;
run the balanced Mistral and Gemma strips on the streaming display paths too.
- safetensors loop: buffer a leading wrapper-less Gemma call:NAME{...} so it
drains and executes instead of streaming the raw call text.
- inference: render the native-template fallback on a shallow tokenizer copy
instead of mutating the shared tokenizer outside the generation lock, and
load the native template from base_model for LoRA adapters.
Adds regression tests for each; existing parser suite stays green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden multi-format tool-call detection from review findings
Apply five targeted fixes from the review pass over the multi-format tool
path:
- routes: route display strip delegates to _strip_tool_xml so Mistral
[TOOL_CALLS] blocks with nested JSON are removed from streamed display
text, not just the XML forms.
- tool_call_parser: skip function/parameter starts that fall inside an
already-open parameter block (_inside_open_parameter) so nested example
payloads are not mis-parsed as new calls; extract
strip_llama3_leading_sentinels so the bare-JSON guard is shared.
- safetensors_agentic: probe bare JSON through strip_llama3_leading_sentinels
before the balanced-brace check so a leaked header sentinel does not defeat
the guard.
- tool_healing: allow dotted tool names in the Gemma wrapped start pattern.
- llama_cpp (GGUF): buffer wrapper-less Llama-3.2 {"name":..} calls that carry
no XML signal, drain a complete object silently and hold an incomplete one,
and run the end-of-stream safety net unconditionally so markerless calls are
detected and never leak the raw JSON (including truncated fragments).
Adds regression tests for the GGUF bare-JSON streaming path and the Mistral
display strip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stop bare-JSON tool calls leaking at EOF, oversized, and into history
The second review pass flagged that the Llama-3.2 bare-JSON tool-call handling
still leaked raw JSON in several spots; ``strip_tool_markup`` only knows
XML/bracket markup, so the bare-JSON form survived it. Fix them symmetrically
across the safetensors and GGUF loops:
- Safetensors stream-end resolver now routes a held bare-JSON fragment to
DRAINING (mirroring GGUF) so a truncated ``{"name":..`` cut off by the end of
the stream is dropped instead of flushed as assistant content. The 7/10
reviewer finding.
- Both loops now drain (suppress) an oversized still-open bare-JSON call once it
passes ``_MAX_BARE_JSON_BUFFER`` instead of streaming the raw prefix, gated on
a ``"name"`` key so a giant plain JSON answer still streams; a complete
oversized call still executes via the safety net.
- Add a shared ``strip_leading_bare_json_call`` helper and apply it to the
content kept for the assistant turn in both loops, so an executed bare-JSON
call is not replayed as visible text or fed back as next-turn history.
Plain JSON answers without a ``"name"`` key are untouched throughout. Adds
regression tests for the EOF, oversized, and next-turn cases on both backends
plus unit tests for the helper.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: bound the Llama-3 python_tag strip on real control sentinels
The route display strip's <|python_tag|> arm ran to the next <| of any kind.
A tool-call argument carrying a literal <|...|> token (for example <|cite|>
inside a string value) truncated the strip early and leaked the call tail into
the visible response. Narrow the stop condition to the genuine Llama control
sentinels (eot_id, eom_id, python_tag, start/end_header_id, begin_of_text,
finetune_right_pad_id) so embedded markup and JSON are consumed while real
header/turn boundaries still bound the strip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden GLM/Gemma parsing, cap GGUF textual calls, share native-template fallback
GLM 4.x parser walked a body pre-bounded by the first </tool_call>, so a string
argument containing a literal </tool_call> (e.g. code that prints it) was
truncated. Walk arg_key/arg_value pairs against the full content instead, since
each <arg_value> is delimited by its own </arg_value> and the call's real close
is the </tool_call> that precedes the next <arg_key>.
Add a truncated wrapper-less Gemma pattern (call:NAME{... with no closing brace)
to the markup strip so a call cut off mid-arguments does not leak raw into the
visible stream. It runs after the closed form, so a complete call keeps trailing
prose.
Cap and dedup tool calls parsed from the GGUF TEXTUAL fallback at
_MAX_TOOL_CALLS_PER_TURN, mirroring the safetensors loop. Structured
delta.tool_calls are grammar-bounded by llama-server, but text parsed straight
from content is not, so one runaway turn could fan out into dozens of
executions.
Extract the native-chat-template fallback into chat_template_helpers
(render_native_template / render_with_native_template_fallback) so the
transformers and MLX text backends share one implementation. The MLX text path
now applies it too, so an Unsloth override template that drops the tools schema
no longer silently stops MLX from advertising tools. The MLX VLM path renders
via the processor for image tokens and is intentionally left on its own render.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate markerless bare JSON on enabled tools and close parser/strip asymmetries
The Llama-3.2 custom_tools bare-JSON form has no marker, so any JSON object with a
name key was read as a tool call. An ordinary JSON answer like
{"name":"Alice","parameters":{"age":30}} was misclassified as a call to a
disabled tool and dropped from the visible response. Gate the markerless form on
the enabled tool names (threaded through parse_tool_calls_from_text and
strip_leading_bare_json_call, supplied by both streaming loops): an object whose
name is not an enabled tool is ordinary content. The marker-based forms keep
their name-agnostic behaviour (an explicit signal is a real call attempt), and
unrestricted mode stays ungated.
Also fix two parser/strip asymmetries the parser already tolerated:
- A literal </function> inside a parameter value (print("</function>")) truncated
both the core and route strips at the first close, leaking the tail. Extend the
strip to the call's real close (last </function> before the next opener),
mirroring the parser, without merging separate calls.
- The single-object Mistral [TOOL_CALLS]{...} shape parsed but _strip_mistral_closed_calls
left it, leaking the raw object into display. Strip the balanced object while
keeping trailing prose, matching the array and name shapes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio tools: fix strip/parse symmetry and native-template token for DeepSeek/GLM/Kimi
Pass-3 review follow-ups on the multi-format tool parser:
- Bare Kimi call (<|tool_call_begin|>...<|tool_call_end|> with no section
wrapper) is accepted by the parser, so add it to the closed strip patterns
so the streaming (non-final) display strip removes it instead of leaking the
markup mid-generation.
- Route display strip now also runs the wrapper-less Gemma cleanup, so a
Gemma 4 call:NAME{..} no longer leaks into the visible answer.
- MLX model record carries base_model for a LoRA adapter so the native-template
fallback loads the base repo template rather than the adapter's
(often template-less) tokenizer.
- Native-template reload forwards the load-time HF token so a gated/private
model's repo template can still be fetched (transformers and MLX text paths).
- GGUF end-of-stream bare-call heuristic is gated on the enabled tool names so a
truncated ordinary JSON object ({"name":"Alice","age":) streams as the answer
instead of being dropped as a tool call.
Adds regression tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio tools: gate GGUF bare-JSON suppression on enabled tools and fix python-tag exponent parsing
Pass-4 review follow-ups on the GGUF tool loop and Llama-3 parser:
- The GGUF bare-JSON suppression sites still keyed off a raw "name" substring,
so an ordinary JSON answer whose name is not an enabled tool was dropped when
it was truncated, oversized, or reached the no-tool DRAINING fallback (the
parser, helper, and safetensors paths were already gated). All three sites now
use the shared enabled-name gate, and a held bare-JSON buffer that turns out not
to be an enabled call is shown as the answer instead of dropped at stream end.
- The Llama-3 python-tag numeric kwarg regex matched only the mantissa, so
scientific notation was truncated to its leading digits (1e-3 parsed as 1) and a
tool executed with the wrong value. The regex now accepts exponent and decimal
forms, and the int/float classification keys off the exponent too.
Adds regression tests for the truncated / oversized disabled-name JSON cases (and
a counterpart that a truncated enabled call still does not leak) plus the
scientific-notation kwargs.
* Studio: drop accidentally committed async worker transcripts
Eight generated reviewer / async-worker transcripts were committed under
studio/backend/async_task_outputs/. They are not imported or referenced by any
code and carry only internal task state, so they should never ship in the repo.
Remove them and gitignore the directory so they cannot be re-added.
* Studio tools: gate safetensors bare-JSON drain, fix nested-name gate and function-XML strip
Pass-4 review follow-ups on the shared parser / safetensors loop:
- The safetensors oversized and end-of-stream bare-JSON drain branches keyed off
a raw "name" substring, so a large or truncated ordinary JSON answer whose name
is not an enabled tool was drained instead of streamed. Both now use the shared
enabled-tool-name gate, matching the GGUF path.
- strip_leading_bare_json_call matched the first "name" anywhere, so a plain JSON
answer with a nested name equal to an enabled tool ({"result":{"name":"web_search"}})
was wrongly suppressed. It now extracts the TOP-LEVEL name only, walking past
nested objects/arrays and keeping the text when a top-level value is truncated.
- The function-XML display strip used a regex negative-lookahead that stopped at a
literal <function=...> opener inside a parameter value and then dropped the rest
of the answer to EOF. A scan-based strip mirrors the parser (ignores openers
inside an open <parameter> via _inside_open_parameter) and closes each call at its
real </function>, so trailing assistant text after such a call survives.
Adds regression tests for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep tools prompt when native-template probe raises; make helper tests hermetic
Pass-4 review follow-ups on the native-template fallback:
- render_with_native_template_fallback re-renders the live template with tools=None
to detect whether it dropped the schema. A template that requires tools can raise
on that probe; that must not discard the already-valid tools prompt. The probe is
now wrapped so any error returns the original formatted_prompt (transformers would
otherwise fall back to manual formatting and lose the schema; MLX would let the
exception escape).
- The native-template helper tests imported InferenceBackend just to reach the
thin wrapper, which pulls in unsloth and its optional vllm package metadata. They
now call the dependency-light render_native_template helper directly so they pass
in a backend/test environment without vllm. Adds a probe-raises regression test.
* Tool parsing: 3.9 import safety, disabled-Auto-Heal contract, capability gate
Round-2 review follow-ups on the multi-format tool-call parser:
- tool_call_parser: add `from __future__ import annotations`. The module
is dependency-light by design (external llama-server wrappers import it
standalone) and the package targets python >=3.9, where its PEP 604
`int | None` return annotations would raise TypeError on import.
- safetensors + GGUF drain fallback: gate the leading bare-JSON strip on
auto_heal_tool_calls. With Auto-Heal off, a truncated enabled-name
fragment that did not parse now stays visible, matching the XML strip
in the same branch and the disabled-Auto-Heal contract. With Auto-Heal
on it is still suppressed.
- safetensors capability gate: match the bare-JSON `{"name":` template
marker with a whitespace/escape-tolerant regex so a pretty-printed
`{ "name" :` or JSON-escaped `{\"name\":` template is not mis-classified
as tool-less. The parser already accepts that whitespace via
raw_decode, so the gate must too.
Regression tests added for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GLM tool-call display strip: treat literal close tag in arg value as data
Round-2 review follow-up on the GLM 4.x tool-call format.
The GLM call shape is <tool_call>NAME<arg_key>k</arg_key><arg_value>v
</arg_value>...</tool_call>. The parser was hardened to walk arg_key /
arg_value pairs so a literal </tool_call> inside an argument value (e.g.
print("</tool_call>")) is treated as data and the call's real close is the
</tool_call> that precedes the next <arg_key>. The display strips still used a
non-greedy <tool_call>.*?</tool_call> regex, which stopped at the literal and
leaked the call's tail into visible content and stale history.
Add _strip_glm_calls, a scan that mirrors the parser's close detection, and run
it before the regex arms in every strip pipeline: the core strip_tool_markup,
the route _strip_tool_xml display/history cleanup, and the safetensors + GGUF
streaming strips. Qwen / Hermes <tool_call>{json} has no NAME token after the
opener, so it is left to the regex arms unchanged.
Regression tests cover the literal-close-tag leak (core + route), normal GLM
calls, back-to-back GLM calls, zero-arg GLM, truncated GLM, and untouched Qwen.
* Tool parsing: symmetric "function" bare-JSON alias and route strip parity
Round-3 review follow-ups, all parser/strip symmetry fixes.
- Bare-JSON "function" alias: the markerless parser accepts a call name via
obj.get("name") or obj.get("function"), but the strip/gates only knew "name",
so a {"function":<enabled tool>} call executed while its raw JSON leaked. Teach
_top_level_bare_json_name the alias (with "name" precedence and the same nested
and truncated-name guards), and widen the guards in strip_leading_bare_json_call,
the safetensors and GGUF _looks_like_enabled_bare_json gates, and the route
capability marker regex.
- Route display/history cleanup: strip a tail-only </param> alias close (the
parser accepts <param name="...">...</param>), and run the parser's guarded
function-XML scan (_inside_open_parameter) before _TOOL_XML_RE so a literal
nested <function=...></function> inside an argument value does not truncate the
strip and leak the tail.
Regression tests added for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio tools: fix DeepSeek strict recovery, Kimi dotted names, Gemma spaced streaming
Round 3 review fixes for the DeepSeek / GLM / Kimi tool-call parsing path.
- DeepSeek R1 and V3/V3.1 strict parsing (Auto-Heal off): when a call is
truncated (missing closing fence or <tool_call_end> terminator), skip it
and keep scanning for later well-formed calls instead of breaking out and
dropping the rest of the envelope. This matches the Kimi strict parser's
recovery behaviour.
- Kimi dotted tool names: keep the full name after stripping only the
functions. prefix and :idx suffix, e.g. functions.mcp.server-list:0 stays
mcp.server-list. The previous split on "." truncated dotted MCP names to
their last segment. This matches current vLLM
(tool_id.split(":")[0].removeprefix("functions.")) and SGLang
(^(?:functions\.)?(?P<name>[\w.\-]+):(?P<index>\d+)$).
- Gemma wrapper-less call streaming: hold the whitespace-tolerant prefix
(call : NAME) in the streaming suppression buffer, matching the parser's
_GEMMA_BARE_TC_RE, so the spaced spelling split across chunks is buffered
instead of leaking as visible text. Applied to both the safetensors and
llama.cpp streaming paths.
- Remove dead _render_with_native_template method and the now-unused copy
import from inference.py; the live path uses render_with_native_template_fallback.
Adds regression tests for DeepSeek R1/V3 strict recovery, Kimi full dotted
name preservation, and the Gemma spaced-call streaming suppression.
* Studio tools: honor tool budget in GGUF loop and guard function-XML streaming strip
Round 4 review fixes. Both are asymmetric-fix bugs where the final/steady path got a
guard the analogous streaming/loop path did not.
- GGUF tool-call budget: the safetensors loop counts real tool-call turns against
max_tool_iterations (re-prompt stalls excepted), but the GGUF loop only bounded the
turn count by the enlarged range (max_tool_iterations + _MAX_REPROMPTS). Since this
PR raised _MAX_REPROMPTS from 1 to 3, a model that keeps making valid tool calls
could run up to three extra tool rounds (with max_tool_iterations=1, four rounds
instead of one). Add a _tool_iters_done counter that increments only when a tool
actually executed in the turn, and stop once the caller's budget is spent so the
post-loop final-answer nudge fires. A duplicate/disabled no-op turn is a correction
turn (like a plan-without-action re-prompt) and does not consume budget, preserving
the existing "already completed" re-prompt behavior.
- Streaming display strip: the final strip runs the guarded _strip_function_xml_calls
scanner (a literal <function=...> inside a parameter value is data, not a nested
call), but the GGUF and safetensors streaming strips still used only the open-ended
regex arms. When a tool-call argument contained literal function markup, the regex
tail ate everything to end-of-text and dropped the real trailing prose after the
call's true </function>. Run the guarded scanner (and the balanced Mistral strip)
before the regex arms in both streaming paths so streaming and final display agree.
Adds regression tests: GGUF valid tool calls respect max_tool_iterations, and the
streaming strip keeps trailing prose after a function-XML call with a literal marker.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio tools: safetensors tool budget counts only executed turns (GGUF parity)
Follow-up to the GGUF budget fix. The safetensors loop charged max_tool_iterations
per non-re-prompt iteration (iteration + 1 - reprompt_count), so a duplicate/disabled
no-op turn spent a budget slot even though no tool ran. With a small cap this dropped
real work: for max_tool_iterations=2, a model that made a valid call, repeated it (an
internal no-op correction turn), then made a distinct valid call executed only the
first -- the third turn was sent with no tools and the distinct call was ignored.
Track whether a turn actually executed a tool (set on record_result) and count only
those turns against the cap, matching the GGUF loop. A duplicate/disabled no-op is a
correction turn -- like a plan-without-action re-prompt -- and no longer consumes
budget, so the model still gets its "already completed" nudge and another tool-enabled
turn. Adds a regression test for the small-cap duplicate-then-distinct-call flow.
* Studio tools: fix stale Kimi dotted-name regression test
test_pr5624_regressions.py still expected functions.my.tool:0 to resolve to the last
segment (tool). The parser now preserves the full dotted name (my.tool) after removing
only the functions. prefix and :idx suffix, matching current vLLM/SGLang so dotted MCP
names like mcp.server-list survive. Update the assertion, name, and module docstring to
the corrected contract (the raw id is still preserved on the call).
* Studio: render the reasoning block for safetensors and MLX like GGUF
enable_thinking chat templates (Qwen3/Qwen3.5/GLM) prefill an unclosed <think>
into the generation prompt, so the model emits only the closing </think> then
the answer. The safetensors/MLX chat stream emitted that as plain content, so
the reasoning showed inline with no collapsible thinking block, while GGUF
(which surfaces reasoning via reasoning_content) rendered one. This brings
safetensors and MLX to parity.
- _ResponsesReasoningExtractor gains a reasoning_prefilled mode that starts
inside the reasoning block and splits on the first </think>; default False
keeps GGUF and every existing caller byte-identical. It suppresses a stray
re-emitted <think> and holds partial markers back across chunk boundaries.
- _sf_reasoning_prefill_mode gates the mode on reasoning being enabled for the
request, an enable_thinking or enable_thinking_effort style, and the template
actually using the standard <think>/</think> markers. Models with a bespoke
reasoning channel (e.g. gemma's <|think|>/<|channel>) are excluded so their
answer is never swallowed; gpt-oss (Harmony) and thinking-off requests are
excluded too.
- sf_tool_stream and stream_chunks (the latter also serves MLX) feed text
through the extractor, emitting reasoning_content then content deltas, with a
per-turn reset in the tool loop and a flush before each tool_start; only the
visible delta reaches the monitor reply. The two non-streaming drains split
reasoning_content the same way.
- Tests: extractor prefilled mode (streaming and edge cases), the gate matrix
including the gemma-style exclusion, and a route-replay of the tool-loop
reasoning stream.
* Studio: render the reasoning block for safetensors and MLX like GGUF
enable_thinking chat templates (Qwen3/Qwen3.5/GLM) prefill an unclosed <think>
into the generation prompt, so the model emits only the closing </think> then
the answer. The safetensors/MLX chat stream emitted that as plain content, so
the reasoning showed inline with no collapsible thinking block, while GGUF
(which surfaces reasoning via reasoning_content) rendered one. This brings
safetensors and MLX to parity.
- _ResponsesReasoningExtractor gains a reasoning_prefilled mode that starts
inside the reasoning block and splits on the first </think>; default False
keeps GGUF and every existing caller byte-identical. It suppresses a stray
re-emitted <think> and holds partial markers back across chunk boundaries.
- _sf_reasoning_prefill_mode gates the mode on reasoning being enabled for the
request, an enable_thinking or enable_thinking_effort style, and the template
actually using the standard <think>/</think> markers. Models with a bespoke
reasoning channel (e.g. gemma's <|think|>/<|channel>) are excluded so their
answer is never swallowed; gpt-oss (Harmony) and thinking-off requests are
excluded too.
- sf_tool_stream and stream_chunks (the latter also serves MLX) feed text
through the extractor, emitting reasoning_content then content deltas, with a
per-turn reset in the tool loop and a flush before each tool_start; only the
visible delta reaches the monitor reply. The two non-streaming drains split
reasoning_content the same way.
- Tests: extractor prefilled mode (streaming and edge cases), the gate matrix
including the gemma-style exclusion, and a route-replay of the tool-loop
reasoning stream.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: don't force a tool re-prompt on a negated intent (safetensors parity)
The safetensors _INTENT_SIGNAL claimed to mirror GGUF but was missing the
negative lookahead, so a refusal like "I will not search the web for that"
matched the "i will" intent and triggered the plan-without-action re-prompt
(STOP... you MUST call a tool), overriding a valid no-tool answer. GGUF already
excludes not/never. Add the same (?!\s+(?:not|never)\b) lookahead so both
backends agree. Extends the intent parity test with negated refusals.
* studio: parse the outer envelope before DeepSeek/Kimi markers embedded in its args
parse_tool_calls_from_text ran the DeepSeek/Kimi marker pre-pass before the shared
<tool_call>/<function=...> parser. When a Qwen/Hermes call's argument contained
literal Kimi/DeepSeek markup (for example a user asking the model to explain that
syntax), the pre-pass matched the embedded marker and returned it, executing the
wrong tool and dropping the real call. Skip the pre-pass when a <tool_call> or
<function=...> envelope opens before the first DeepSeek/Kimi marker, so the shared
parser takes the outer call; a genuine marker-led call (no leading envelope) still
goes through the pre-pass. Tests for the embedded-marker case and the control.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: trim redundant comments (comment-only, AST-verified)
* Studio: trim redundant comments (comment-only, AST-verified)
* Studio: prevent Gemma tool-parser DoS on stray delimiters
_gemma_parse_value returned the input index unchanged when text[i] was a
stray delimiter (,}]), so the list and mapping caller loops that advance
on the returned index spun forever at 100% CPU on malformed input such as
[},]. Advance past the delimiter so parsing always terminates.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strip Magistral [THINK] reasoning from final display/history
strip_tool_markup removed [TOOL_CALLS] and <function> markup but left a
leading Magistral [THINK]...[/THINK] block intact, so its bracket-form
reasoning (not the <think> the reasoning channel renders) leaked into the
safetensors display and conversation history while GGUF/llama.cpp routes
it natively. Drop the leading reasoning block at end-of-turn (final=True)
via the existing _strip_mistral_reasoning helper; streaming is untouched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep times in wrapper-less Gemma tool arguments
The wrapper-less Gemma value scanner used _GEMMA_KEY_RE = [\w.\-]+ for keys,
which also matches a digit-leading token, so a comma followed by a time or
ratio inside a value (call:web_search{query:meet at 10:00, 11:00 tomorrow})
was misread as a new 11: key, truncating the query and injecting a bogus
argument. Require keys to start with a letter or underscore, matching the
identifier-start rule the wrapped path already uses (_GEMMA_NEXT_KEY_RE).
Add a regression test.
* Studio: treat markers/close-tags inside tool-call arguments as data
Four parser correctness fixes where a valid argument string was mistaken for
structure:
- DeepSeek: find the envelope-end token outside JSON strings, so a query/code
argument containing the literal token no longer truncates the body and drops
the whole call.
- GLM: locate the real </arg_value> as the one whose next token is <arg_key> /
</tool_call> / end, so a value containing a literal </arg_value> (or
</tool_call>) is kept instead of executing the tool with corrupted arguments.
- Attribute-form <function name="..."> envelopes now count in the embedded-marker
guard, so a DeepSeek/Kimi marker inside a parameter value does not hijack the
outer call and run the wrong tool.
- Wrapper-less Gemma call:NAME{...} is gated on the enabled tool names (parse and
display strip), mirroring the Llama bare-JSON gate, so a disabled/example name in
prose is not stolen as a call and the real answer is preserved.
Add regression tests for each.
* Gate route Gemma wrapperless strip by enabled tools; make Kimi section-end search string-aware
Route-level display stripping now threads the enabled tool-name set into the
Gemma wrapperless-call strip, so prose that mentions a disabled tool
(call:foo{...}) is preserved while active tool calls are still stripped. This
mirrors the parser-level gate already used in tool_call_parser.
The Kimi section-end lookup now searches outside JSON string literals, so a
section-end marker appearing inside an argument string no longer triggers a
false truncation that drops a valid tool call.
* Run DeepSeek/Kimi pre-pass when a closed tool-call example precedes a real block
The marker pre-pass was skipped whenever any <tool_call>/<function> opener
appeared before the first DeepSeek/Kimi marker, even when that opener was a
CLOSED syntax example in prose that ends before the real block. In that case
parse_tool_calls_from_text skipped the DeepSeek/Kimi parsers and the genuine
tool call was dropped while a phantom tool named in the example ran instead.
Only treat a marker as embedded in a leading envelope when removing the closed
outer <tool_call>/<function> envelopes also removes every marker (the marker
actually sat inside one). A marker left standing is a real call, so the pre-pass
runs. The legitimate case of a marker inside a closed outer envelope's arguments
is preserved.
* Honor reasoning_effort none in safetensors prefill; strip Magistral reasoning while streaming
Two safetensors/MLX reasoning fixes surfaced in review:
_sf_reasoning_prefill_mode only checked enable_thinking, so an
enable_thinking_effort (GLM-5.2) request that disables thinking via
reasoning_effort=none (without enable_thinking=False) still began in
prefilled-<think> mode. A plain answer with no </think> was then swallowed
whole into reasoning_content and the visible response came back empty. Thread
reasoning_effort into the predicate and treat none as disabled, mirroring
_request_reasoning_kwargs.
strip_tool_markup_streaming stripped tool markup but not the leading Magistral
[THINK]...[/THINK] bracket block, so the raw chain-of-thought leaked into the
streamed safetensors content instead of the reasoning drawer (GGUF routes it
natively). Apply _strip_mistral_reasoning first, matching the final strip; an
unclosed [THINK] is held from the marker on so nothing flickers.
* Heal truncated outer tool envelopes and keep quoted Gemma args intact
Two follow-ups from review of the marker pre-pass and Gemma parsing:
The leading-envelope guard only removed CLOSED outer <tool_call>/<function>
envelopes before deciding whether a DeepSeek/Kimi marker was embedded, so a
truncated outer call missing its close tag (whose argument embeds a marker) was
treated as a standalone marker and the embedded sample ran instead of the
intended outer call being Auto-Healed. Decide on the last outer opener before the
marker and whether it closed before the marker instead, so a closed syntax
example still runs the pre-pass while a real closed-or-truncated outer call keeps
it.
The wrapper-less Gemma argument scan tracked bracket depth but not quotes, so a
quoted value containing a comma followed by a key-like token (a search query such
as "weather, location: Boston") was split mid-string, truncating the value and
fabricating an extra argument. Track quote state (with escapes) so the top-level
comma boundary is only taken outside quoted spans.
* Span outer envelopes to their real close when locating embedded markers
Locating the DeepSeek/Kimi marker relative to a leading outer envelope used the
FIRST close tag after the opener, so a literal </function> or </tool_call> inside
an argument value (for example python code that contains the text) was mistaken
for the envelope boundary. The marker after it was then treated as a standalone
call and the embedded sample ran instead of the intended outer call.
Match the closed outer envelopes with the shared patterns that already extend to
the real final close (a literal close inside a value is data), and treat a marker
that survives their removal as embedded only when a still-open (truncated) outer
opener precedes it, so Auto-Heal still repairs a truncated outer call. A closed
syntax example before a genuine block still runs the pre-pass.
* Span the tool_call outer envelope to its real close in the marker guard
The leading-envelope check reused the lazy <tool_call>.*?</tool_call> strip
pattern, so a Qwen/Hermes JSON argument containing a literal </tool_call> ended
the span early. A DeepSeek/Kimi sample later in that same string then survived
the closed-envelope removal, and the pre-pass executed the embedded call instead
of the outer <tool_call>. The <function> arm already spanned to its real close;
give <tool_call> the same real-close pattern (with the negative lookahead that
keeps back-to-back calls separate) so a literal close inside a value is data.
* Preserve no-tool Gemma prose and keep later R1 calls when healing a close
Two review follow-ups:
_gemma_strip_gate returned None when no tools were enabled, and None means
strip every markerless call:NAME{...} block, so a no-tool answer that documents
the syntax (or the Anthropic display path, which passes an empty tool list as
None) had that prose deleted. It is a display/history gate, so return the
enabled-name set instead -- an empty set when no tool is enabled, which strips
nothing because every call:NAME{...} is then prose.
The DeepSeek R1 heal path located the close fence with an unbounded forward
search, so when a first call had balanced JSON but omitted its fence the search
landed on a LATER call's terminator and pos advanced past that valid call,
dropping it. Match the close immediately after the JSON (whitespace-skipped) like
the strict path, and advance by just the JSON when it is absent, so a multi-call
turn keeps its later well-formed calls (heal is now a superset of strict).
* Resume wrapper-less Gemma scan past a consumed call's balanced body
The markerless call:NAME{...} scan used finditer, which resumes right after the
opening call: token, so a nested call:OTHER{...} mentioned inside the first
call's own quoted string argument (for example a web_search query that quotes the
Gemma tool syntax) was re-matched and returned as a spurious second tool call,
executing an unintended tool. Walk with a manual cursor that resumes after the
outer call's balanced body (brace matching already skips quoted braces), so a
call's arguments are never rescanned. Genuinely separate back-to-back calls and
disabled/example prose are unaffected.
* Mistral outer call wins over XML literals; align healer signals with its parser
Two follow-ups on the shared-parser ordering after the healing-passthrough
merge:
- A well-formed [TOOL_CALLS] call whose JSON arguments quote tool XML parsed
the literal instead of the outer call (executing the wrong tool). When the
first XML signal sits inside a leading balanced Mistral body it is argument
data, so the Mistral parser now runs first; an XML signal before the trigger
keeps the normal order, so a [TOOL_CALLS] literal inside an XML call's
arguments still stays data.
- passthrough_healing buffered streams on the parser module's broadened signal
list (now including <|python_tag|> and [TOOL_CALLS]) but promotes with
core.tool_healing, which does not parse those forms: a streamed Mistral or
Llama text call was held until finalization and flushed as prose. The healer
keeps its own signal list limited to the formats it can promote, restoring
immediate streaming for the rest.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: Gemma wrapper-less marker literals and quotes, GLM embedded close pair
- The Gemma fallback deferral now keys on an actual wrapped opener
(_GEMMA_TC_RE), not the wrapper literal anywhere in content: a wrapper-less
call whose argument merely mentions <|tool_call> has nothing tool_healing
can parse, and deferring it lost the call entirely (not executed and
stripped from display).
- New _gemma_body_brace_end boundary scanner honors single- and double-quoted
strings like _gemma_parse_stripped_body, shared by parse and strip, so a
quoted brace in a code argument (code:print('}')) no longer truncates the
executed arguments or the strip span.
- _glm_value_close now requires a structural </arg_value> to sit at balanced
quote state: the full pair </arg_value></tool_call> embedded inside a string
literal is data, not an early close. When no candidate balances, the first
token-valid close wins as before.
* Address review: leading envelopes win over rehearsed literals
- New _first_foreign_tool_signal shared by the leading-envelope guards adds
<|python_tag|> to the protected signal set: the spelled-out literal inside a
Mistral call's arguments (a query about Llama built-in tool syntax) executed
the inner literal instead of the outer call.
- New _xml_signal_inside_leading_bare_json guard, sibling of the Mistral one:
a leading bare-JSON call whose string argument quotes tool XML (a code value
citing <function=...>) had the literal promoted by the shared XML pass
before the bare-JSON parser ran.
- Magistral [THINK]...[/THINK] is dropped once at parse entry instead of only
inside the Mistral parser, so a call rehearsed in the think block in a
foreign format can no longer be promoted while the real call after the
block is lost. Parse now agrees with the display strip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: a disabled leading bare-JSON object keeps its literals as data
When the leading bare-JSON object is ordinary content (name not an enabled
tool), the guard proved the first tool signal sits inside it, so falling
through to the XML/python_tag passes promoted quoted string data as a real
call. Drop the object and parse only the tail: a real call after the object
still parses, nothing inside it can be promoted.
* Address review: apostrophes in raw Gemma values, GLM strict key contract, per-model template token
- Quote openers in the wrapper-less Gemma boundary and body scanners now
require value-start context (after : { [ ( , =): an apostrophe inside an
unquoted value (query:what's the weather) opened quote mode, swallowed the
real closing brace, and lost the whole call on common contraction queries.
Quoted values keep hiding delimiters as before.
- A GLM <arg_key> with no <arg_value> tag now rejects the call in strict
mode, matching the unclosed-value contract, instead of executing the tool
with the argument silently dropped; Auto-Heal keeps the lenient skip.
- The native-template fallback reads the hf_token stored on the model record
instead of the instance-wide last-load token, so a later token-less load
cannot break template fetches for a previously loaded gated model (both
the transformers and MLX backends).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: Mistral literals inside leading JSON, whitespace-tolerant wrapped Gemma opener
- The leading bare-JSON guard now treats the [TOOL_CALLS] trigger as a
foreign signal: the Mistral parser runs before the bare-JSON one, so a
literal quoted inside the leading object's strings was promoted over the
outer call (or over ordinary JSON content).
- tool_healing's wrapped Gemma opener tolerates whitespace around call and
the colon: sampling drift emits call: name{ and call : name{, and
rejecting those lost the call entirely because no fallback re-parses the
wrapped form. Strict mode still requires the closing tag.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: DeepSeek/Kimi markers inside leading JSON and Mistral envelopes stay data
The DeepSeek/Kimi pre-pass runs before the outer-call parsers, and
_marker_inside_leading_envelope only protected XML envelopes: a marker
quoted inside a leading bare-JSON or Mistral call's argument strings was
promoted as a separate no-arg call and the real outer call dropped. The
guard now recognizes those two leading envelopes as well; standalone
DeepSeek/Kimi calls keep parsing.
* Address review: accept dotted Gemma argument keys in the key-quoting scanner
The scanner quoted keys of [alnum_-] only, so a dotted key (user.name:...)
was left unquoted, json.loads failed, and the whole wrapped call was lost
(parse empty, strip wipes the markup). Dots now match the parser's own
key/name charset.
* Address review: a real DeepSeek/Kimi call after a disabled leading JSON object still parses
DeepSeek/Kimi markers are foreign signals for the leading bare-JSON guard
too: a marker literal inside a disabled leading object made the envelope
guard skip the pre-pass for the whole message, so a real DeepSeek/Kimi call
after the object was dropped. Routing the case through the guard's
drop-and-parse-the-tail recursion reaches the real call while the literal
inside the object stays data.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: leading Mistral call owns the turn, dotted keys after bare values
- A LEADING parseable [TOOL_CALLS] call now runs the Mistral parser first
unconditionally: literal XML in trailing prose after the call was promoted
by the earlier shared XML pass, executing the quoted example instead of
the real leading call. XML leading keeps the normal order.
- _GEMMA_NEXT_KEY_RE accepts dots so a dotted key after a bare value
(query:foo,user.name:bob) ends the value at the comma instead of being
swallowed into it, matching the round-earlier key-quoting charset.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: a leading wrapper-less Gemma call owns the turn
A quoted foreign literal inside a leading wrapper-less Gemma call's
argument (a query citing another tool syntax) was promoted by tool_healing
before the Gemma fallback ran, executing the quoted example and dropping
the outer call. New leading guard, sibling of the Mistral and bare-JSON
ones, gated on an enabled name since the form is markerless. Foreign markup
leading keeps the normal order.
* Fix merge resolution: restore both leading-guard test classes intact
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: markup quoted inside a nameless leading JSON answer stays data
The leading bare-JSON guard required a top-level name, so a structured JSON
answer quoting tool markup in its strings (a response_format turn
documenting a tool's syntax) had the literal promoted by the later passes.
A nameless leading object that parses as real JSON now routes through the
same decline-then-parse-the-tail path; non-JSON braced prose keeps the old
behaviour, and a real call after the answer still parses.
* Address review: JSON answers stay data, nested Gemma quotes, earliest envelope, no failure caching
- A whole-content JSON value is a structured answer: the markerless Gemma
scan and its strip no longer promote or strip a quoted example of an
enabled tool's syntax inside it.
- Nested stripped-stream Gemma values now unquote quoted string leaves
recursively, so {loc:{city:"New York"}} hands the tool New York, matching
the top-level coercion.
- The DeepSeek/Kimi pre-pass dispatches by earliest envelope opener, so a
leading real call wins over a trailing example of the sibling format in
either direction.
- A failed native-template fetch is no longer cached as no-template: the
next call retries after the model record's token is fixed or a transient
Hub error clears; only definitive loads are cached.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: closed calls precede the marker pre-pass, truncated Gemma scan stops, quoted nested delimiters
- A closed non-DeepSeek/Kimi call preceding the first DS/Kimi marker owns
the turn: a trailing syntax example, or one quoted inside a wrapped Gemma
argument, was promoted by the pre-pass and dropped the real leading call.
Wrapped Gemma joins the outer-envelope pattern sets.
- An unbalanced wrapper-less Gemma call now stops the scan (mirroring the
strip contract) instead of resuming inside its own argument text, where a
quoted enabled call would be promoted.
- Raw-quoted strings in nested stripped-stream Gemma values hide delimiters,
so {city:"New, York"} is one value instead of a split pair, returned
unquoted like the top-level coercion.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: string-marker literals in wrapper-less args, mid-value quoted phrases
- The wrapper-less deferral guard no longer keys on the <|"|> literal: a
real call whose argument merely mentions the string marker was deferred to
tool_healing, which has no wrapped opener to parse, losing the call. The
wrapped-opener check alone owns the deferral.
- Double quotes now also open at the start of a word, so a quoted phrase
mid-value (query:find "weather, location: Boston", limit:3) hides its
delimiters instead of splitting the value into garbage keys; apostrophes
keep the value-start-only rule so contractions stay prose.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: strict GLM refuses in-quote close fallback, Gemma guard covers preambles
- _glm_value_close gains a strict flag: a truncated value whose only close
candidates sit inside a string literal rejects the call in strict mode
(Auto-Heal keeps the lenient partial), restoring the strict contract the
quote-aware fallback had weakened.
- The leading wrapper-less Gemma guard no longer requires the call to open
the response: a visible preamble before call:NAME{...} is the normal
shape, and the quoted foreign literal inside the argument was promoted
again in that shape. An enabled balanced call beginning before the first
foreign signal owns it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: contextual GLM quote openers, disabled Gemma examples stay prose, JSON array answers
- The GLM value-close quote tracker uses the same contextual openers as the
Gemma scanners (single quote after punctuation context, double quote also
at word start), so strict mode accepts a normal apostrophe value again
while still rejecting a truncated value whose only close candidates sit
inside a string literal.
- A disabled wrapper-less Gemma call is prose by design, so a tool literal
quoted inside it no longer promotes: the span is dropped for parsing and
the tail parsed, mirroring the nameless-JSON guard.
- Leading JSON ARRAY answers join the leading-JSON envelope guard, so a
marker quoted inside a structured array response stays data.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Align closed-envelope regression test with the document-order contract
The test asserted the pre-round-13 behavior (trailing DeepSeek/Kimi block
wins over a leading closed envelope) while the shipped rule is document
order: the leading closed call owns the turn. Rename the test and assert
the leading call so the suite matches the contract exercised by
test_leading_xml_call_wins_over_trailing_kimi_example.
* Parse a leading Llama-3.2 bare-JSON call before the markerless Gemma scan
The bare-JSON form only ever matches a leading call object, and document
order says that call owns the turn. Running the Gemma wrapper-less scan
first let an enabled call:NAME{...} snippet quoted inside the leading
call's string arguments steal the turn when the JSON was not the whole
content (trailing prose or a second ;-separated call), executing the
quoted tool instead of the real one. Reordering cannot take a leading
Gemma call's turn since that content never starts with an object brace.
* Leading-call ownership: Mistral trigger in Gemma guards, closed bare JSON before markers, depth-aware nested Gemma values
Three parser gaps against the document-order contract:
The wrapperless Gemma leading guards did not count [TOOL_CALLS] as a
foreign signal, so a leading Gemma call quoting a Mistral snippet in its
argument lost the turn to the quoted literal. Both the enabled-call and
disabled-example guards now include the trigger, matching the bare-JSON
guard's local inclusion.
_marker_inside_leading_envelope required the DeepSeek/Kimi marker to sit
inside the first closed bare-JSON or Mistral call. A marker after that
closed call (a trailing example or data in a later ;-chained call's
strings) now also defers to the leading call, the same inside-or-after
rule the closed XML envelope patterns already applied.
The nested Gemma primitive value scan split on every comma, corrupting
arguments like opts:{code:print(1,2),lang:py}. It now applies the same
paren/brace depth, contextual quote openers, and comma-only-before-a-key
mapping rule as the top-level scan.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gemma leading guard: a closed enabled call preceding the signal owns the turn
The wrapperless Gemma guard only claimed the turn when the first foreign
signal sat inside the first enabled balanced call. When that call closed
before the signal (a second call quoting a Mistral or Kimi literal, or a
trailing prose example), the guard forfeited the turn and the foreign
parser promoted the quoted literal, dropping the real Gemma calls. Apply
the same inside-or-after ownership rule as the closed bare-JSON and
Mistral envelopes, gated on an enabled name so the name-agnostic legacy
path is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Marker guard: only an executable leading bare-JSON call owns the turn
The bare-JSON branch of the leading-envelope marker guard claimed the
turn for any NAMED leading object. A disabled-name object is prose by
design (the bare-JSON parser will not execute it), so deferring the
DeepSeek/Kimi pre-pass to it lost the real later call entirely. Gate the
ownership claim on the enabled set (or the name-agnostic None path). A
marker inside the disabled object's own strings stays data, matching the
tail-exclusion contract; a marker after it now falls through so the
pre-pass parses the real call. The Mistral branch stays ungated since
[TOOL_CALLS] parsing is never name-gated.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gemma scan skips leading JSON answers; GLM heal bounds values at structural tags
Two fixes to the document-order data contracts:
The markerless Gemma scan only exempted whole-content JSON, so a leading
JSON answer followed by prose had an enabled call:NAME{...} snippet
inside its strings promoted to a real executed call and stripped from
the displayed answer. Both the parse and strip scans now start after a
balanced json-valid leading value span, keeping parse and strip
mirrored. Real calls after the answer still parse; mid-prose JSON gets
no exemption.
The GLM heal fallback for a missing closing arg_value tag took the
entire remainder as the value, executing markup-contaminated arguments
like city="NYC</tool_call>" and swallowing trailing prose. The healed
value now stops at the next arg_key or tool_call close and the pair walk
resumes there. EOF-truncated values keep the partial heal, strict mode
still rejects, and closed values holding a literal close tag in quotes
are untouched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Compress docstrings in the multi-format tool parser to their contract essence
* Condense parser guard comments and test narration to contract essentials
* verify_import_hoist: exempt __future__ imports and same-diff relocations
Two false positives fired on this PR's refactor. A from __future__ import
is a compiler directive whose name never appears as a runtime load, so
HOISTED-IMPORT-UNUSED can never see it used, yet the file requires it for
PEP 604 annotations on Python 3.9. TARGET-CHANGED flagged the deliberate
move of the strip-pattern constants into core.inference.tool_call_parser
as a silent re-point even though the old module-level target was removed
and the new one added in the same diff. Both get narrow exemptions; a
re-point to a pre-existing target is still caught, and the self-test
negative controls all pass unchanged.
* Leading bare-JSON calls own the turn; function calls end at the first balanced close
The XML-signal guard for a leading bare-JSON call required the signal
strictly inside the object, so a trailing XML example stole the turn
from the leading call; it now applies the same inside-or-after rule as
the Mistral guard. Function-XML calls also ended at the LAST close tag,
which let prose after a closed call that mentions a literal close tag
get swallowed into the final parameter value; calls now end at the
first close tag that is not inside an open parameter, and the strip
mirrors the same rule so parse and strip agree.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Attribute-form calls end at the first balanced close; bare-JSON strip requires the call shape
The attribute form parser still kept the last close tag in the call
window, folding prose after a closed call into the final parameter
value. It now takes the first close not inside an open parameter, the
same rule the equals form and the strip already use.
The leading bare-JSON strip deleted any closed object whose top-level
name matched an enabled tool, including plain JSON answers the parser
correctly rejects as non-calls. The strip (and the drain gate that
delegates to it) now requires the parser's exact call shape, so answers
like {"name":"web_search","result":...} stream and display intact.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* False-alarm markers keep the answer; the bare-JSON strip consumes the whole chain
The trailing strip arms dropped everything from a bare marker to EOF,
so a normal answer that mentions [TOOL_CALLS] or another marker
literally was truncated (or fully swallowed when it started with the
literal) after the no-call drain fallback. Those arms now require a
call-shaped lookahead or marker-at-EOF before dropping; truncated real
calls still strip.
Chained bare-JSON turns executed both calls but stripped only the first
object, so the second call's raw JSON replayed into the next assistant
history message alongside the structured tool_calls. The strip now
consumes the entire chained run of call-shaped enabled objects while
non-call answers, disabled names, and trailing prose stay intact.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* DeepSeek and Kimi trailing strip arms require a call-shaped lookahead
Same false-alarm rule as the bare-word markers: a prose answer that
mentions a DeepSeek or Kimi marker literally keeps its tail, while
truncated real envelopes and bare end-of-text fragments still drop.
* Attribute-form containment, parameter-close-decides rule, preamble-tolerant Mistral guard, strict strip shape
Four document-order and containment fixes. A leading attribute-form
call now parses before the shared XML pass, so markup quoted in its
parameter stays data. The open-parameter scan lets the parameter's own
close tag decide, so any number of literal function closes inside one
value stay data, restoring the pre-close-scan behavior for multi-close
arguments. The leading-Mistral guard tolerates a visible preamble, with
the leading-bare-JSON guard running first so a trigger quoted inside a
leading JSON object stays data. The bare-JSON strip requires the
parser's top-level name in every mode, so nested-name JSON answers
survive name-agnostic stripping.
* Keep buffering long wrapper-less Gemma tool names instead of leaking the prefix
The streaming buffer stopped holding a call:NAME prefix at a fixed
32-char cap, so a Gemma wrapper-less call to a tool whose name exceeds
that (OpenAI allows 64 chars, MCP names run longer) streamed its raw
call:longname text as visible content before the end-of-turn parser
executed it. Hold the variable-length prefix while it still matches the
call: shape, bounded like the bare-JSON path and self-terminating into
prose, draining once the opening brace arrives.
* Keep prose that only mentions DeepSeek/Kimi markers in the route display strip
The route-level _TOOL_XML_RE DeepSeek/Kimi arms consumed from an opener up to
the end of text whenever the marker appeared, so an answer that merely refers
to a marker (for example "See <|tool_call_begin|> in the docs") had the rest
of the reply truncated. The parser-level _TOOL_ALL_PATS already gates these
arms with a call-shaped lookahead. Mirror it here so a marker is only stripped
when a real call follows it or it is a bare fragment at end of text.
* Tighten tool-calling parser and backend comments
* Pass trust_remote_code when reloading native tokenizers
The native-template fallback re-fetches a model's native chat template from
its repo when an Unsloth override template drops the tools schema. The
secondary AutoTokenizer.from_pretrained threaded hf_token but not
trust_remote_code, so for a model loaded with trust_remote_code=True whose
tokenizer repo carries custom code the reload raised, was swallowed, and the
request silently kept the tool-dropping prompt for a model that supports tools.
Store the loaded trust_remote_code on each backend's per-model info dict and
source it in render_native_template, so the reload re-uses exactly the consent
granted at load. For a LoRA adapter the reload targets the base model, whose
remote code was gated and loaded under the same stored flag, so re-passing it
executes no unconsented code. Falsy stored flag preserves the prior behaviour.
Adds a regression test that fails without the flag (custom-code reload raises,
returns None) and passes with it (tools-advertising native prompt returned).
* Treat <|python_tag|> as an outer marker envelope
A Llama-3 <|python_tag|> tool call (built-in NAME.call(...) or custom
{json} form) whose argument quotes a complete DeepSeek/Kimi example was
hijacked by the DeepSeek/Kimi marker pre-pass: the embedded example (for
example delete_all) executed instead of the real outer call. python_tag
is Llama-3's tool-call envelope, so a marker quoted inside its arguments
is data, the same as for <tool_call>, <function=...>, bare JSON, Mistral
and wrapper-less Gemma, which the guard already covers.
Add <|python_tag|> to _OUTER_ENVELOPE_OPEN_RE with a call-shaped
lookahead (mirroring the _TOOL_ALL_PATS python_tag arm) so the marker
pre-pass is suppressed when a python_tag call opens before the first
marker, while a bare prose <|python_tag|> mention is left untouched.
* Tighten tool-call parser comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: Daniel Han <info@unsloth.ai>
Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>
* Quote-aware Gemma strip, symmetric unstarted cleanup, ReDoS anchor
Address review findings on the tool-strip and streaming paths:
- strip_tool_call_markup stripped Gemma-native spans with a plain regex that
stops at the first <tool_call|>, so a literal close marker inside a
<|"|>-quoted argument truncated the span and leaked its suffix into visible
text. A brace/quote-aware _strip_gemma_native_spans now removes complete
spans (keeping an incomplete one unless final), matching the parser's own
balance logic.
- The Gemma close pattern this PR added (<\|tool_call>.*?<tool_call\|>) had no
\Z fallback, so a run of unclosed markers backtracked from every open
position (quadratic, and the streaming stripper re-scans per token). It is
now anchored to (?:<tool_call|>|\Z) like routes/inference.py's _TOOL_XML_RE,
linear with identical output on well-formed input.
- _SameTaskStreamingResponse added unstarted_cleanup for the OpenAI passthrough,
but the local GGUF/safetensors streams that enter _TrackedCancel before
returning only unregister in the generator finally, which never runs if the
client disconnects before the body iterator starts, leaking cancel-registry
entries. Each such stream now passes unstarted_cleanup to exit its tracker.
- __call__ reads _unstarted_cleanup via getattr so a response built through
__new__ (the cancel-timing test) without __init__ does not raise
AttributeError; the test also sets the attribute explicitly.
- Document that the verbatim /v1/chat/completions passthrough delegates
<think>/<|tool_call> splitting to llama-server (--jinja, --reasoning-format
auto) and is intentionally not re-parsed locally, noting the llama.cpp
dependency.
Adds a regression test for the close-marker-inside-quoted-argument strip.
* Tighten comments on the tool-strip and streaming paths
Compress the verbose comment blocks added with the Gemma tool-call / streaming
work to crisp one or two liners, drop restatements of obvious code, and shorten
docstrings, keeping the load-bearing rationale (ReDoS anchor, quote-aware strip,
unstarted-cleanup, llama.cpp passthrough dependency). Code is unchanged
(verified comment-only via AST/ast signature, docstrings stripped).
* Harden Gemma parse/strip: span-aware XML fallback and quote-aware streaming
- Security: the XML fallback in parse_tool_calls_from_text scanned the whole
content for <function=...> markers and only skipped those inside an open XML
parameter, not those inside a collected JSON/Gemma candidate span. A balanced
but unparsable Gemma call whose argument data contained XML tool markup
(<|tool_call>call:outer{code:<function=terminal>...}<tool_call|>) therefore
fell through to the fallback and returned an executable terminal call. The
fallback now also excludes <function=> markers inside any candidate span,
including ones that failed to parse.
- strip_tool_call_markup no longer skips the generic Gemma regex after running
the quote-aware _strip_gemma_native_spans, so a closed Gemma span the helper
cannot match (malformed, e.g. <|tool_call>{"name":"x"}<tool_call|>) is still
stripped instead of leaking its opener and payload into visible text.
- _strip_gemma_native_spans stops at the first unbalanced start instead of
re-scanning every later start to EOF, keeping it linear on a run of unclosed
markers rather than quadratic.
- The GGUF and safetensors streaming strippers run _strip_gemma_native_spans
before the regex patterns, so a well-formed streamed call whose quoted
argument contains a literal close marker no longer leaks its suffix into
incremental display.
Adds regression tests for the nested-XML escape and the malformed-span strip.
* Avoid remainder copy in _strip_gemma_native_spans
Match the Gemma close marker with re pos directly on the buffer instead
of slicing tail = text[brace_end + 1:] on every span. The streaming
strippers re-scan a growing cumulative buffer per token, so the per-span
remainder copy was quadratic. Behavior is unchanged.
* Exclude unclosed Gemma/JSON starts from the XML tool-call fallback
The nested-XML guard only skipped <function=> markers inside recorded
candidate spans, but a span is recorded only when the braces balance. An
unbalanced call such as <|tool_call>call:outer{code:<function=terminal>...
recorded no span, so the fallback still promoted the inner <function=> to
an executable terminal call. Treat unclosed JSON/Gemma starts as exclusion
spans through EOF before scanning. Standalone <function=> calls with no
preceding unclosed start still parse. Regression tests added.
* Skip doomed tool-strip passes to avoid quadratic rescans
The lazy closed-pair strip patterns (<tool_call>.*?</tool_call>,
<function=...>.*?</function>) rescan to EOF from every opener when their
close token is absent, which is O(n^2) and re-runs per streamed token. Add
strip_tool_patterns, which skips a pass whose close token is not present in
the text; output is identical to the per-pattern loop (verified by fuzz),
and a degenerate run drops from ~minutes to milliseconds. Used by
strip_tool_call_markup and the GGUF/safetensors streaming strippers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use full tool-call envelopes to close nested-XML escape variants
Key the parser and stripper off the full <|tool_call>...<tool_call|> /
<tool_call>...</tool_call> envelope (start to close marker, searched after
the braces; EOF if unclosed) instead of just the braces:
- XML between the closing brace and the close marker
(call:outer{broken:{x}}<function=terminal>...<tool_call|>) is now inside
the envelope, so the fallback no longer promotes it to a tool call.
- A balanced inner call inside an unclosed outer
(call:outer{code:<|tool_call>call:terminal{...}<tool_call|>) is skipped
via the envelope nested check, not just the XML fallback.
- strip_tool_call_markup searches for the close marker after the braces, so
junk before <tool_call|> is stripped through the close and text after it is
preserved instead of truncated to EOF; a no-close run stops early (linear).
Regression tests added; standalone XML and well-formed calls unaffected.
* Fix non-final Gemma strip and missing-close recovery for PR #6611
Split the nested-skip from the XML fallback exclusion: nesting is decided by
each marker's brace region, so a balanced call after one with a missing close
marker is recovered instead of being swallowed to EOF. Only the XML fallback
keeps the search-to-close envelope, so trailing nested markup still cannot
escape as an executable call.
Use a closed-only Gemma pattern in the non-final strip list so an incomplete
block is preserved (matching the JSON and function paths); the final list keeps
the close-or-EOF Gemma pattern in its original position, so streaming display
output is byte-for-byte unchanged.
Add regression tests for both cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Block gap-nested tool markers and fix XML strip order for PR #6611
Decide candidate nesting by a per-marker coverage region paired with a
per-format stack (a close after the braces pops the nearest still-open marker
of that format). A closed outer call now covers up to its own close marker, so a
JSON or Gemma tool marker smuggled between the outer braces and that close is
treated as data instead of being executed. An outer that balances but has no
close of its own covers only its brace region, so a later sibling after an
omitted close marker is still recovered (adjacent calls use an exclusive end
bound so the next call is not misread as nested).
Strip every closed pair (JSON, Gemma, function) before any to-EOF sweep, so a
closed function call whose parameter text contains a bare Gemma opener is
removed as a unit and the to-EOF sweep can no longer drop the visible text after
the close.
Add regression tests for both.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Strip closed tool blocks before the Gemma final sweep for PR #6611
The final display strip ran the quote-aware Gemma helper before the closed
JSON/function patterns. A closed <tool_call>...</tool_call> or
<function=...>...</function> block whose argument data held a call-form Gemma
opener (e.g. a "<|tool_call>call:t{" string) was read as an incomplete Gemma
span and truncated to EOF, dropping the block's close and any visible text after
it.
Strip closed JSON/function blocks first, so such a block is removed as a unit
before the helper runs. Centralize the final strip order in a shared
strip_tool_markup_final so strip_tool_call_markup and both streaming display
wrappers (safetensors, llama_cpp) stay in sync, and apply the same closed-block
pre-pass to the non-final path.
Add regression tests for the JSON and function variants.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Recover XML/JSON siblings after a close-less tool marker for PR #6611
Two fixes so the XML fallback and marker coverage recover a later valid call
after an earlier marker omits its close, matching the candidate loop:
Reuse the candidate marker-coverage in the XML fallback instead of a separate
search-to-close-or-EOF envelope. A balanced but close-less marker now covers
only its brace region there too, so a following <function=...> sibling is
recovered rather than filtered as nested data; an unbalanced marker still covers
to EOF and a closed one still covers through its close, so nested XML stays
blocked.
Ignore a close token that falls inside another call's balanced braces when
pairing closes in _marker_coverage. Such a token is that call's quoted argument
data, so it no longer pops an earlier close-less marker and extends its coverage
over a later valid sibling.
Add regression tests for both.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make the closed-block strip pre-pass Gemma-span-aware
The final display strip ran the closed JSON/function regex pre-pass before
removing Gemma-native spans, so a literal <function=...> quoted inside a Gemma
argument plus any later </function> (a real call's close or even prose) was
deleted across the Gemma boundary. That mangled the Gemma close marker, the
quote-aware helper then saw an unclosed opener, and the whole visible tail
after the call was truncated.
The pre-pass now skips matches that start inside a complete Gemma span (that
text is the span's argument data) and resumes scanning at the end of the
covering span, so a real function-XML call after the Gemma call is still
stripped. The original ordering rationale is preserved: a Gemma opener inside
a JSON or function argument still cannot truncate that block, covered by
regression tests for both directions.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments in the Gemma streaming and strip pipeline to essentials
* Tighten comments in the Gemma strip and streaming disconnect paths
* Fold marker-collection comment to two lines
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: tool calling for Llama-3, Mistral, Gemma 4 on safetensors + MLX (#5615)
Adds tool calling for Llama-3, Mistral (pre-v11 + v11+ + [ARGS]), and Gemma 4 to the safetensors / transformers and MLX backends. Parser patched against llama.cpp / vLLM / SGLang per-family parsers and normalises to OpenAI shape. 96 targeted unit tests + cross-OS staging CI (ubuntu / macos-14 / windows) green on the multi-format probe.
* studio: tool-call healing parity between safetensors / MLX and GGUF
After the multi-format parser landed in #5615, the safetensors / MLX
agentic loop and the GGUF loop still differed on healing behaviour.
This commit closes the gaps in both directions so the two backends
react the same way to identical model output.
Changes:
1. core/inference/llama_cpp.py -- the GGUF BUFFERING state machine
now wakes on every emission marker the shared parser knows. Was
("<tool_call>", "<function="); is now the five-tuple imported
from core.inference.tool_call_parser (Qwen / Qwen3.5 / Llama-3
<|python_tag|> / Mistral [TOOL_CALLS] / Gemma 4 <|tool_call>).
Stream cleanup is delegated to the same shared strip_tool_markup
so leaked markup from any family is removed from assistant
content.
2. core/inference/llama_cpp.py -- per-tool canonical heal key. When
a tool arguments field is a bare string and JSON parsing fails,
the GGUF path now heals to {"code": raw_args} for python,
{"command": raw_args} for terminal, and {"query": raw_args} for
everything else. Was hard-coded to {"query": raw_args}, which
silently routed every python / terminal emission through
web_search. Mirrors safetensors_agentic._CANONICAL_HEAL_ARG.
3. core/inference/safetensors_agentic.py -- re-prompt on plan-
without-action. When the model emits a short forward-looking
intent ("I'll search for that", "Let me check", "First, I
will...") and no tool call, the loop nudges the model to act
instead of silently returning a plan-only answer. Up to
_MAX_REPROMPTS=3 (matches GGUF). The intent regex, character
cap, and instruction text are byte-identical to the GGUF path.
The buffer-end fall-through is unified so a buffered intent
emission that never exits the BUFFERING state still triggers
the re-prompt.
4. core/inference/safetensors_agentic.py -- extra iteration slots
for re-prompts. The loop now budgets max_tool_iterations +
_MAX_REPROMPTS + 1 total iterations and tracks the tool-call
count separately, so a stalling model can be nudged 3x without
eating the caller's tool-call budget. Mirrors the _extra slot
reservation in the GGUF path.
Tests (14 new safetensors-side units; 5 GGUF parity pins):
TestLoopRePrompt -- intent-trigger, plain-answer,
no-tools, cap-at-three, budget
preserved, buffer-end intent.
TestLoopCanonicalHealKey -- python / terminal / unknown.
TestGGUFSafetensorsHealingParity -- shared markers used, shared
strip used, canonical heal keys
identical, intent regex matches
same phrases, _MAX_REPROMPTS
equal on both backends.
All 110 targeted tests pass locally; the broader tool / inference /
model-config / sandbox / anthropic / mlx suites stay green.
Why this matters
Without this parity, Llama-3.2 / Mistral / Gemma 4 emissions on Mac
(MLX) and Linux-safetensors stop the agentic loop as soon as the
model says "Let me...", because the GGUF re-prompt logic never
existed on these backends. The two-marker GGUF BUFFERING tuple also
let non-Qwen tool emissions stream out as plain prose when
llama-server's structured channel did not pick them up. Both paths
now drain the same way, heal the same way, and re-prompt the same
way -- so a tool call that works on GGUF works identically on
safetensors / MLX.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: fix tool-call parser bugs from gemini review on #5620
Three high-priority gemini findings on the tool-call parsing additions:
1. unicode_escape on UTF-8 bytes corrupts non-ASCII literals
(e.g. ✨ becomes â\x9c¨). Replace with json.loads on a quoted
string -- preserves emoji / CJK / RTL while still handling
\n \t \uXXXX escapes.
2. Llama-3 sentinel stripping is order-dependent. A leading
`<|eot_id|><|begin_of_text|>` left `<|begin_of_text|>` behind
because the loop had already passed that sentinel. Loop until
no sentinel matches at the start.
3. Mistral v11+ `[TOOL_CALLS] name { json }` regex uses non-greedy
`\{.*?\}` which truncates at the first `}` of a nested JSON
argument, leaking the tail (e.g. `}}`) into user-visible
streamed text. Same problem for the v0.3 array pattern with
nested brackets. Strip those with balanced brace/bracket
scanning via a new `_strip_mistral_closed_calls` helper called
from `strip_tool_markup`.
Also fix the inference routes' parallel `_TOOL_XML_RE`:
- Same nested-JSON truncation in the Mistral patterns; route the
strip through the parser's balanced-scan helper via a thin
`_strip_tool_xml` wrapper that all existing callers now use.
- Llama-3 `<|python_tag|>[^\n<]*` stopped at any `<`, leaking the
tail of any tool call whose argument contained a literal `<`
(queries, code snippets). Relax to `[^\n]*` which keeps the
strip confined to the actual end-of-line.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/routes: make python_tag strip multi-line aware
Earlier revisions of _TOOL_XML_RE in studio.backend.routes.inference
oscillated between two bug shapes:
5615 r"<\|python_tag\|>[^\n<]*" -- stopped at any literal "<"
so code='if x < 10: pass'
leaked '< 10: pass)' to the
user.
5620.1 r"<\|python_tag\|>[^\n]*" -- single-line only; the second
line of
python.call(code="a\nb")
leaked.
The full parser (_parse_llama3_python_tag) already handles both via
balanced-brace scanning, so the parsing path was fine; the LEAK was
in the streaming strip path that runs on every cumulative emission
while content is still arriving.
Switch to r"<\|python_tag\|>(?:[^<]|<(?!\|))*" so the strip consumes:
* any character that is not a "<" (newlines, JSON, code, ...),
* a "<" only when it is NOT followed by "|" (i.e. NOT a Llama-3
sentinel start like <|eot_id|>, <|eom_id|>, <|begin_of_text|>).
This means:
* code='if x < 10' stays inside the strip (5615 fix preserved),
* multi-line code stays inside the strip (5620 round 2),
* the strip terminates at the next Llama-3 sentinel so trailing
assistant content survives.
Tests: TestRoutesPythonTagStrip (8 cases)
pytest test_safetensors_tool_loop.py test_safetensors_capability_advertise.py
-> 118 passed in 1.81s (was 110).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten verbose comments in tool-call parser sections
Comments were narrating what the code already says. Cut historical
"earlier revisions used X, then Y" narratives down to one-line WHY
notes where the footgun still matters (canonical heal-key parity,
balanced-brace vs non-greedy regex, ``(?:[^<]|<(?!\|))*`` over
``[^\n<]*``/``[^\n]*``). Drop section-header banners.
No behaviour change. Re-ran:
pytest studio/backend/tests/test_safetensors_tool_loop.py \
studio/backend/tests/test_safetensors_capability_advertise.py -q
-> 118 passed.
Regression replay (parser + _coerce_arguments on the 5 #5615 inputs)
-> 21/21.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: parser robustness fixes for PR #5620
Three surgical extensions to the multi-format tool-call parser, each
covering a real fine-tune / template emission shape that the current
parser silently drops. No path narrows; all changes widen what is
accepted.
1. `_parse_tool_call_json` now accepts both `arguments` and
`parameters` keys. A Hermes / Qwen `<tool_call>{json}</tool_call>`
wrapper around a Llama-3.2 fine-tune that emits the `parameters`
key was extracting the tool name and silently discarding the
args, producing a working-shaped call with an empty payload. The
bare-JSON and python_tag paths already accepted both keys; this
path now matches them.
2. `_TC_FUNC_START_RE`, `_TC_PARAM_START_RE`, and `_TC_PARAM_CLOSE_RE`
now also match the attribute form
`<function name="..."><param name="...">v</param></function>` used
by MiniCPM-5 and MiniMax-M2. Names land in either capture group,
and `</param>` is accepted as a short close.
3. `_parse_llama3_bare_json` sentinel-strip now consumes the role
label inserted between `<|start_header_id|>` and
`<|end_header_id|>` by Meta's official Llama-3.x chat template.
Without this, every assistant turn re-fed through the template
prefix `<|start_header_id|>assistant<|end_header_id|>\n\n{json}`
parsed to zero calls, so any history-with-tool-call round-trip
in production silently dropped.
Tests in `studio/backend/tests/test_safetensors_tool_loop.py`:
* `TestParserRobustness::test_tool_call_json_accepts_parameters_key`
* `TestParserRobustness::test_function_xml_attribute_form`
* `TestParserRobustness::test_function_xml_attribute_form_multi_param`
* `TestParserRobustness::test_function_xml_legacy_equals_form_still_works`
(regression guard for the existing `<function=name>` syntax)
* `TestParserRobustness::test_llama3_chat_template_round_trip`
* `TestParserRobustness::test_llama3_round_trip_all_roles`
* `TestParserRobustness::test_llama3_round_trip_with_eot_prefix`
`pytest studio/backend/tests/test_safetensors_tool_loop.py
studio/backend/tests/test_safetensors_capability_advertise.py -q`
goes from 118 to 125 passed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: terminate function-XML body at </function>, not just </tool_call>
`_parse_function_xml` was looking for `</tool_call>` (the Hermes
wrapper) as the body terminator. When a model emits a standalone
`<function=NAME><parameter=K>v</parameter></function>` followed by
explanatory prose (which models routinely do), no `</tool_call>` is
present, so the body extended to end-of-string and the trailing
prose leaked into the LAST parameter value.
Pre-existing on main (the legacy `<function=NAME>` form had this
bug too). Same affects PR #5620's new attribute-form
`<function name="NAME"><param name="K">v</param></function>`
emission used by MiniCPM-5 / MiniMax-M2.
Fix: `_TC_END_TAG_RE` now matches either `</tool_call>` OR
`</function>`. The existing `_TC_FUNC_CLOSE_RE` / `_TC_PARAM_CLOSE_RE`
strips are unchanged. Multi-call inputs still bound each function
at the next `<function=` start, so no over-eager consumption.
New tests:
* `test_function_xml_followed_by_prose` (legacy form + prose)
* `test_function_attribute_xml_followed_by_prose` (attribute form + prose)
Existing `test_code_with_embedded_xml` still passes (a parameter
value containing literal `<a></a>` is preserved because the
embedded close tag is `</a>`, not `</function>`).
`pytest studio/backend/tests/test_safetensors_tool_loop.py
studio/backend/tests/test_safetensors_capability_advertise.py -q`
goes from 125 to 127 passed.
* Studio: tighten Llama-3.2 bare-JSON guard
A fuzz pass on PR #5811 turned up that ``_parse_llama3_bare_json``
accepted ``parameters`` as a string, contradicting the docstring's
"parameters or arguments is a dict" guard. Prose JSON like
``{"name":"foo","parameters":"a sentence"}`` would wrongly fire the
parser, which the agentic loop would then heal into a real
``foo(query="a sentence")`` call.
Same code lives on this branch, so the same fix applies here.
Tightened guard:
- ``parameters`` must be a dict (Llama-3 spec).
- ``arguments`` may be a dict, or a JSON-encoded string that
decodes to a dict (OpenAI shape, e.g.
``"arguments":"{\"q\":\"x\"}"``). Plain non-JSON strings or
JSON-strings of lists / scalars / null no longer pass.
Mirrors the fix landed in PR #5811 commit 615b8608. Adds the same
4 regression tests under TestParserMultiFormat.
Existing test suite stays green: 127 -> 131 passing.
* studio: fix safetensors tool-call parser gaps vs llama.cpp (Mistral CALL_ID / THINK, attribute-form signal)
Three GGUF-parity fixes to the safetensors tool-call parser, each matching
llama.cpp's reference behaviour:
- Mistral Small 3.2 emits [TOOL_CALLS]name[CALL_ID]<id>[ARGS]{json}. The
parser stopped after the name on seeing [CALL_ID] (neither [ARGS] nor {),
dropping the call. Skip an optional [CALL_ID]<id> segment in both the
parse and strip paths. llama.cpp parses this (test-chat.cpp:4785).
- Magistral wraps reasoning in [THINK]...[/THINK]. A [TOOL_CALLS] inside the
reasoning was parsed as a real call, producing a phantom call. Strip a
leading [THINK] block before scanning so only the post-reasoning call
counts (test-chat.cpp:2285); a literal [THINK] inside a later argument is
left intact.
- The standalone MiniCPM-5 / MiniMax-M2 <function name="..."> attribute form
parsed correctly but was absent from TOOL_XML_SIGNALS and the markup strip
patterns, so the streaming safety-net parse was gated off (dropping the
call) and markup leaked into displayed text. Add the signal and broaden
the strip regexes.
Adds regression tests for all three.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: fire safetensors tool calls for the bare-JSON (Llama-3.2) form
The agentic loop's streaming safety-net parse was gated on
has_tool_signal(), which is False for the Llama-3.1 / 3.2 bare-JSON tool
form {"name":..,"parameters":..} (no XML marker). Real tool calls were
therefore dropped: the loop logged "model planned without calling tools",
re-prompted three times, then gave up with zero tool calls, while GGUF's
llama-server parses the same emission natively.
Run parse_tool_calls_from_text() unconditionally in the safety net. The
parser is strict (only fires on a valid tool-call shape) so plain answers
are unaffected. Reproduced on a real unsloth/Llama-3.1-8B-Instruct run:
the model emits {"name":"web_search","parameters":{...}} which now
executes the tool instead of being re-prompted into a no-op.
Adds a loop regression test for the bare-JSON form.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: complete strict-mode contract and fix parser import paths
Address review findings on the multi-format tool-call parser:
- Honor allow_incomplete=False in the remaining sub-parsers. The Llama-3
<|python_tag|>NAME.call(...) parser, the pre-v11 Mistral [TOOL_CALLS] array
parser, and the Gemma 4 <|tool_call> parser ignored strict mode, so a
truncated call (missing closing paren, ], or <tool_call|>) was still healed
and executed with Auto-Heal disabled. Thread strictness through and reject
the unclosed forms, matching the JSON and function-XML paths.
- Drop the duplicate tool_call_parser import block in llama_cpp.py and the
redundant un-aliased TOOL_XML_SIGNALS; only the _SHARED_TOOL_XML_SIGNALS
alias is used as a value.
- Import _strip_mistral_closed_calls from core.inference.tool_call_parser in
routes/inference.py instead of studio.backend.core... The self-contained
run.py launch mode only puts studio/backend on sys.path, so the absolute
package path raised ModuleNotFoundError on the server-tool strip path.
Add strict-mode regression tests for the truncated Llama-3 dot-call and the
unclosed Mistral array.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: preserve XML param indentation and alias Mistral array parameters
Two parser-correctness fixes found by auditing against the model chat templates
and the SGLang / vLLM reference parsers:
- Qwen3.5 XML parameter values lost their leading indentation. The chat template
emits <parameter=k>\nVALUE\n</parameter>, but the parameter-start regex ate the
wrapping newline AND the value's first-line indentation with a trailing \s*,
then str.strip() removed the rest. Narrow the trailing class to horizontal
whitespace only and trim exactly one wrapping newline (via _trim_param_value),
preserving indentation in code/diff arguments. Matches SGLang's qwen3_coder
detector. Applies to both _parse_function_xml (tool_call_parser.py) and the XML
path in tool_healing.py.
- Mistral pre-v11 array objects keyed on parameters dropped their payload.
_consume_mistral_call read only the arguments key; alias parameters the same way
the JSON/XML paths and SGLang's base detector do.
Add regression tests for preserved multi-line indentation and the array
parameters alias.
* Studio: tighten tool-call parser comments
Make the comments in the multi-format tool-call parser and its callers succinct:
compress verbose docstrings/blocks to one or two lines, drop ones that restate the
code, and trim the tiny balanced-scanner helpers. Correctness rationale and
upstream provenance (SGLang/llama.cpp parity, the strict-mode / Auto-Heal
contract, whitespace-preservation, and the Unicode / full-width-pipe notes) are
kept in compact form.
Comment-only: no code or behavior change (verified with comment_tools.py check
--strip-docstrings; parser suite green).
* Studio: make Llama-3 .call and Mistral-array healing parsing linear
Two more O(n^2) ReDoS paths in the multi-format parser, both reachable from
the agentic loop on a long truncated body with no length cap:
- _LLAMA3_KV_RE.finditer over a .call(...) body retried at every offset of a
long word run / unterminated quote (40K -> 14s). Replace with a hand-scan
that reuses the same key/number/literal sub-regexes via anchored match and
walks the string body by hand, so an unterminated quote is O(n). Verified
byte-identical to the old regex over 200K fuzzed inputs.
- _parse_mistral_array healing ran _balanced_brace_end from every { in the
body (20K -> 17s). Walk top-level objects, advancing past each balanced
{...}; this also drops the phantom call the old scan emitted from a nested
argument object.
Add adversarial-length linearity regressions plus positive .call kwargs and
unclosed-array recovery coverage.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: honor strict mode in safety-net, keep empty Gemma args, strip attribute-form function XML
- safetensors safety-net parser now forwards allow_incomplete=auto_heal_tool_calls,
matching the draining path, so a late incomplete tool call is not healed and
executed when Auto-Heal is off.
- Gemma empty bare value ({k:}) now serialises as "" instead of invalid {"k":},
which previously dropped the whole call.
- Route _TOOL_XML_RE also strips the <function name="..."> attribute form
(MiniCPM-5 / MiniMax-M2) so it no longer leaks to the UI.
* Studio: fix attribute-form function-XML literal close tag and zero-arg strict call
Addresses Codex review of the <function name="..."> attribute form in
_parse_function_xml (MiniCPM-5 / MiniMax-M2):
- End the call body at the LAST </function> / </tool_call> within the call's
window, so a literal close tag inside a code/search argument (e.g.
print("</function>")) is preserved instead of truncating the call.
- Accept a closed call with no parameters as a valid zero-argument call in strict
mode (the function close is already required), instead of rejecting it as a
truncated call.
- Tests for both, mirroring the legacy <function=...> coverage.
* Studio: fix tool-call parser/loop review findings on the multi-format path
Address the live code-review findings on the safetensors/MLX + GGUF tool path:
- routes: include the attribute form <function name="..."> in the safetensors
capability whitelist so MiniCPM-5 / MiniMax-M2 templates keep the tool pill
(parser already handles the form; the post-filter wrongly suppressed it).
- safetensors loop: build the plan-without-action re-prompt from the active
tools instead of a hardcoded web_search/python string, and gate it on
auto_heal_tool_calls, matching the GGUF loop.
- safetensors loop: hold a leading bare-JSON object ({"name":..,"parameters":..})
during BUFFERING until it closes, then drain it as a tool call instead of
streaming the raw JSON to clients. The DRAINING/STREAMING resolvers still
recover a plain JSON answer, so this can never drop content.
- parser: anchor the Llama-3 <|python_tag|>NAME.call(...) scan to the tag and
chain ; -separated calls, so all semicolon-separated built-ins parse and a
literal <|python_tag|>x.call(...) inside a JSON string argument no longer
fires the wrong tool.
- parser: consume the optional trailing </s> after a named Mistral
[TOOL_CALLS]name{json} call, mirroring the array shape.
- GGUF streaming strip: use the shared parser patterns (which know
[TOOL_CALLS] and <|python_tag|>) so a textual tool call entering DRAINING is
stripped instead of leaking the marker to streaming clients.
- routes: hoist the _strip_mistral_closed_calls import to module level.
Adds regression tests covering each fix; existing parser suite stays green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden multi-format tool-call detection from review findings
Apply five targeted fixes from the review pass over the multi-format tool
path:
- routes: route display strip delegates to _strip_tool_xml so Mistral
[TOOL_CALLS] blocks with nested JSON are removed from streamed display
text, not just the XML forms.
- tool_call_parser: skip function/parameter starts that fall inside an
already-open parameter block (_inside_open_parameter) so nested example
payloads are not mis-parsed as new calls; extract
strip_llama3_leading_sentinels so the bare-JSON guard is shared.
- safetensors_agentic: probe bare JSON through strip_llama3_leading_sentinels
before the balanced-brace check so a leaked header sentinel does not defeat
the guard.
- tool_healing: allow dotted tool names in the Gemma wrapped start pattern.
- llama_cpp (GGUF): buffer wrapper-less Llama-3.2 {"name":..} calls that carry
no XML signal, drain a complete object silently and hold an incomplete one,
and run the end-of-stream safety net unconditionally so markerless calls are
detected and never leak the raw JSON (including truncated fragments).
Adds regression tests for the GGUF bare-JSON streaming path and the Mistral
display strip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stop bare-JSON tool calls leaking at EOF, oversized, and into history
The second review pass flagged that the Llama-3.2 bare-JSON tool-call handling
still leaked raw JSON in several spots; ``strip_tool_markup`` only knows
XML/bracket markup, so the bare-JSON form survived it. Fix them symmetrically
across the safetensors and GGUF loops:
- Safetensors stream-end resolver now routes a held bare-JSON fragment to
DRAINING (mirroring GGUF) so a truncated ``{"name":..`` cut off by the end of
the stream is dropped instead of flushed as assistant content. The 7/10
reviewer finding.
- Both loops now drain (suppress) an oversized still-open bare-JSON call once it
passes ``_MAX_BARE_JSON_BUFFER`` instead of streaming the raw prefix, gated on
a ``"name"`` key so a giant plain JSON answer still streams; a complete
oversized call still executes via the safety net.
- Add a shared ``strip_leading_bare_json_call`` helper and apply it to the
content kept for the assistant turn in both loops, so an executed bare-JSON
call is not replayed as visible text or fed back as next-turn history.
Plain JSON answers without a ``"name"`` key are untouched throughout. Adds
regression tests for the EOF, oversized, and next-turn cases on both backends
plus unit tests for the helper.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: bound the Llama-3 python_tag strip on real control sentinels
The route display strip's <|python_tag|> arm ran to the next <| of any kind.
A tool-call argument carrying a literal <|...|> token (for example <|cite|>
inside a string value) truncated the strip early and leaked the call tail into
the visible response. Narrow the stop condition to the genuine Llama control
sentinels (eot_id, eom_id, python_tag, start/end_header_id, begin_of_text,
finetune_right_pad_id) so embedded markup and JSON are consumed while real
header/turn boundaries still bound the strip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate markerless bare JSON on enabled tools and close parser/strip asymmetries
The Llama-3.2 custom_tools bare-JSON form has no marker, so any JSON object with a
name key was read as a tool call. An ordinary JSON answer like
{"name":"Alice","parameters":{"age":30}} was misclassified as a call to a
disabled tool and dropped from the visible response. Gate the markerless form on
the enabled tool names (threaded through parse_tool_calls_from_text and
strip_leading_bare_json_call, supplied by both streaming loops): an object whose
name is not an enabled tool is ordinary content. The marker-based forms keep
their name-agnostic behaviour (an explicit signal is a real call attempt), and
unrestricted mode stays ungated.
Also fix two parser/strip asymmetries the parser already tolerated:
- A literal </function> inside a parameter value (print("</function>")) truncated
both the core and route strips at the first close, leaking the tail. Extend the
strip to the call's real close (last </function> before the next opener),
mirroring the parser, without merging separate calls.
- The single-object Mistral [TOOL_CALLS]{...} shape parsed but _strip_mistral_closed_calls
left it, leaking the raw object into display. Strip the balanced object while
keeping trailing prose, matching the array and name shapes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio tools: gate GGUF bare-JSON suppression on enabled tools and fix python-tag exponent parsing
Pass-4 review follow-ups on the GGUF tool loop and Llama-3 parser:
- The GGUF bare-JSON suppression sites still keyed off a raw "name" substring,
so an ordinary JSON answer whose name is not an enabled tool was dropped when
it was truncated, oversized, or reached the no-tool DRAINING fallback (the
parser, helper, and safetensors paths were already gated). All three sites now
use the shared enabled-name gate, and a held bare-JSON buffer that turns out not
to be an enabled call is shown as the answer instead of dropped at stream end.
- The Llama-3 python-tag numeric kwarg regex matched only the mantissa, so
scientific notation was truncated to its leading digits (1e-3 parsed as 1) and a
tool executed with the wrong value. The regex now accepts exponent and decimal
forms, and the int/float classification keys off the exponent too.
Adds regression tests for the truncated / oversized disabled-name JSON cases (and
a counterpart that a truncated enabled call still does not leak) plus the
scientific-notation kwargs.
* Studio tools: gate safetensors bare-JSON drain, fix nested-name gate and function-XML strip
Pass-4 review follow-ups on the shared parser / safetensors loop:
- The safetensors oversized and end-of-stream bare-JSON drain branches keyed off
a raw "name" substring, so a large or truncated ordinary JSON answer whose name
is not an enabled tool was drained instead of streamed. Both now use the shared
enabled-tool-name gate, matching the GGUF path.
- strip_leading_bare_json_call matched the first "name" anywhere, so a plain JSON
answer with a nested name equal to an enabled tool ({"result":{"name":"web_search"}})
was wrongly suppressed. It now extracts the TOP-LEVEL name only, walking past
nested objects/arrays and keeping the text when a top-level value is truncated.
- The function-XML display strip used a regex negative-lookahead that stopped at a
literal <function=...> opener inside a parameter value and then dropped the rest
of the answer to EOF. A scan-based strip mirrors the parser (ignores openers
inside an open <parameter> via _inside_open_parameter) and closes each call at its
real </function>, so trailing assistant text after such a call survives.
Adds regression tests for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tool parsing: 3.9 import safety, disabled-Auto-Heal contract, capability gate
Round-2 review follow-ups on the multi-format tool-call parser:
- tool_call_parser: add `from __future__ import annotations`. The module
is dependency-light by design (external llama-server wrappers import it
standalone) and the package targets python >=3.9, where its PEP 604
`int | None` return annotations would raise TypeError on import.
- safetensors + GGUF drain fallback: gate the leading bare-JSON strip on
auto_heal_tool_calls. With Auto-Heal off, a truncated enabled-name
fragment that did not parse now stays visible, matching the XML strip
in the same branch and the disabled-Auto-Heal contract. With Auto-Heal
on it is still suppressed.
- safetensors capability gate: match the bare-JSON `{"name":` template
marker with a whitespace/escape-tolerant regex so a pretty-printed
`{ "name" :` or JSON-escaped `{\"name\":` template is not mis-classified
as tool-less. The parser already accepts that whitespace via
raw_decode, so the gate must too.
Regression tests added for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tool parsing: symmetric "function" bare-JSON alias and route strip parity
Round-3 review follow-ups, all parser/strip symmetry fixes.
- Bare-JSON "function" alias: the markerless parser accepts a call name via
obj.get("name") or obj.get("function"), but the strip/gates only knew "name",
so a {"function":<enabled tool>} call executed while its raw JSON leaked. Teach
_top_level_bare_json_name the alias (with "name" precedence and the same nested
and truncated-name guards), and widen the guards in strip_leading_bare_json_call,
the safetensors and GGUF _looks_like_enabled_bare_json gates, and the route
capability marker regex.
- Route display/history cleanup: strip a tail-only </param> alias close (the
parser accepts <param name="...">...</param>), and run the parser's guarded
function-XML scan (_inside_open_parameter) before _TOOL_XML_RE so a literal
nested <function=...></function> inside an argument value does not truncate the
strip and leak the tail.
Regression tests added for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio tools: honor tool budget in GGUF loop and guard function-XML streaming strip
Round 4 review fixes. Both are asymmetric-fix bugs where the final/steady path got a
guard the analogous streaming/loop path did not.
- GGUF tool-call budget: the safetensors loop counts real tool-call turns against
max_tool_iterations (re-prompt stalls excepted), but the GGUF loop only bounded the
turn count by the enlarged range (max_tool_iterations + _MAX_REPROMPTS). Since this
PR raised _MAX_REPROMPTS from 1 to 3, a model that keeps making valid tool calls
could run up to three extra tool rounds (with max_tool_iterations=1, four rounds
instead of one). Add a _tool_iters_done counter that increments only when a tool
actually executed in the turn, and stop once the caller's budget is spent so the
post-loop final-answer nudge fires. A duplicate/disabled no-op turn is a correction
turn (like a plan-without-action re-prompt) and does not consume budget, preserving
the existing "already completed" re-prompt behavior.
- Streaming display strip: the final strip runs the guarded _strip_function_xml_calls
scanner (a literal <function=...> inside a parameter value is data, not a nested
call), but the GGUF and safetensors streaming strips still used only the open-ended
regex arms. When a tool-call argument contained literal function markup, the regex
tail ate everything to end-of-text and dropped the real trailing prose after the
call's true </function>. Run the guarded scanner (and the balanced Mistral strip)
before the regex arms in both streaming paths so streaming and final display agree.
Adds regression tests: GGUF valid tool calls respect max_tool_iterations, and the
streaming strip keeps trailing prose after a function-XML call with a literal marker.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio tools: safetensors tool budget counts only executed turns (GGUF parity)
Follow-up to the GGUF budget fix. The safetensors loop charged max_tool_iterations
per non-re-prompt iteration (iteration + 1 - reprompt_count), so a duplicate/disabled
no-op turn spent a budget slot even though no tool ran. With a small cap this dropped
real work: for max_tool_iterations=2, a model that made a valid call, repeated it (an
internal no-op correction turn), then made a distinct valid call executed only the
first -- the third turn was sent with no tools and the distinct call was ignored.
Track whether a turn actually executed a tool (set on record_result) and count only
those turns against the cap, matching the GGUF loop. A duplicate/disabled no-op is a
correction turn -- like a plan-without-action re-prompt -- and no longer consumes
budget, so the model still gets its "already completed" nudge and another tool-enabled
turn. Adds a regression test for the small-cap duplicate-then-distinct-call flow.
* Studio: render the reasoning block for safetensors and MLX like GGUF
enable_thinking chat templates (Qwen3/Qwen3.5/GLM) prefill an unclosed <think>
into the generation prompt, so the model emits only the closing </think> then
the answer. The safetensors/MLX chat stream emitted that as plain content, so
the reasoning showed inline with no collapsible thinking block, while GGUF
(which surfaces reasoning via reasoning_content) rendered one. This brings
safetensors and MLX to parity.
- _ResponsesReasoningExtractor gains a reasoning_prefilled mode that starts
inside the reasoning block and splits on the first </think>; default False
keeps GGUF and every existing caller byte-identical. It suppresses a stray
re-emitted <think> and holds partial markers back across chunk boundaries.
- _sf_reasoning_prefill_mode gates the mode on reasoning being enabled for the
request, an enable_thinking or enable_thinking_effort style, and the template
actually using the standard <think>/</think> markers. Models with a bespoke
reasoning channel (e.g. gemma's <|think|>/<|channel>) are excluded so their
answer is never swallowed; gpt-oss (Harmony) and thinking-off requests are
excluded too.
- sf_tool_stream and stream_chunks (the latter also serves MLX) feed text
through the extractor, emitting reasoning_content then content deltas, with a
per-turn reset in the tool loop and a flush before each tool_start; only the
visible delta reaches the monitor reply. The two non-streaming drains split
reasoning_content the same way.
- Tests: extractor prefilled mode (streaming and edge cases), the gate matrix
including the gemma-style exclusion, and a route-replay of the tool-loop
reasoning stream.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: don't force a tool re-prompt on a negated intent (safetensors parity)
The safetensors _INTENT_SIGNAL claimed to mirror GGUF but was missing the
negative lookahead, so a refusal like "I will not search the web for that"
matched the "i will" intent and triggered the plan-without-action re-prompt
(STOP... you MUST call a tool), overriding a valid no-tool answer. GGUF already
excludes not/never. Add the same (?!\s+(?:not|never)\b) lookahead so both
backends agree. Extends the intent parity test with negated refusals.
* Studio: trim redundant comments (comment-only, AST-verified)
* Studio: prevent Gemma tool-parser DoS on stray delimiters
_gemma_parse_value returned the input index unchanged when text[i] was a
stray delimiter (,}]), so the list and mapping caller loops that advance
on the returned index spun forever at 100% CPU on malformed input such as
[},]. Advance past the delimiter so parsing always terminates.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strip Magistral [THINK] reasoning from final display/history
strip_tool_markup removed [TOOL_CALLS] and <function> markup but left a
leading Magistral [THINK]...[/THINK] block intact, so its bracket-form
reasoning (not the <think> the reasoning channel renders) leaked into the
safetensors display and conversation history while GGUF/llama.cpp routes
it natively. Drop the leading reasoning block at end-of-turn (final=True)
via the existing _strip_mistral_reasoning helper; streaming is untouched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor reasoning_effort none in safetensors prefill; strip Magistral reasoning while streaming
Two safetensors/MLX reasoning fixes surfaced in review:
_sf_reasoning_prefill_mode only checked enable_thinking, so an
enable_thinking_effort (GLM-5.2) request that disables thinking via
reasoning_effort=none (without enable_thinking=False) still began in
prefilled-<think> mode. A plain answer with no </think> was then swallowed
whole into reasoning_content and the visible response came back empty. Thread
reasoning_effort into the predicate and treat none as disabled, mirroring
_request_reasoning_kwargs.
strip_tool_markup_streaming stripped tool markup but not the leading Magistral
[THINK]...[/THINK] bracket block, so the raw chain-of-thought leaked into the
streamed safetensors content instead of the reasoning drawer (GGUF routes it
natively). Apply _strip_mistral_reasoning first, matching the final strip; an
unclosed [THINK] is held from the marker on so nothing flickers.
* Mistral outer call wins over XML literals; align healer signals with its parser
Two follow-ups on the shared-parser ordering after the healing-passthrough
merge:
- A well-formed [TOOL_CALLS] call whose JSON arguments quote tool XML parsed
the literal instead of the outer call (executing the wrong tool). When the
first XML signal sits inside a leading balanced Mistral body it is argument
data, so the Mistral parser now runs first; an XML signal before the trigger
keeps the normal order, so a [TOOL_CALLS] literal inside an XML call's
arguments still stays data.
- passthrough_healing buffered streams on the parser module's broadened signal
list (now including <|python_tag|> and [TOOL_CALLS]) but promotes with
core.tool_healing, which does not parse those forms: a streamed Mistral or
Llama text call was held until finalization and flushed as prose. The healer
keeps its own signal list limited to the formats it can promote, restoring
immediate streaming for the rest.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: leading envelopes win over rehearsed literals
- New _first_foreign_tool_signal shared by the leading-envelope guards adds
<|python_tag|> to the protected signal set: the spelled-out literal inside a
Mistral call's arguments (a query about Llama built-in tool syntax) executed
the inner literal instead of the outer call.
- New _xml_signal_inside_leading_bare_json guard, sibling of the Mistral one:
a leading bare-JSON call whose string argument quotes tool XML (a code value
citing <function=...>) had the literal promoted by the shared XML pass
before the bare-JSON parser ran.
- Magistral [THINK]...[/THINK] is dropped once at parse entry instead of only
inside the Mistral parser, so a call rehearsed in the think block in a
foreign format can no longer be promoted while the real call after the
block is lost. Parse now agrees with the display strip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: a disabled leading bare-JSON object keeps its literals as data
When the leading bare-JSON object is ordinary content (name not an enabled
tool), the guard proved the first tool signal sits inside it, so falling
through to the XML/python_tag passes promoted quoted string data as a real
call. Drop the object and parse only the tail: a real call after the object
still parses, nothing inside it can be promoted.
* Address review: Mistral literals inside leading JSON, whitespace-tolerant wrapped Gemma opener
- The leading bare-JSON guard now treats the [TOOL_CALLS] trigger as a
foreign signal: the Mistral parser runs before the bare-JSON one, so a
literal quoted inside the leading object's strings was promoted over the
outer call (or over ordinary JSON content).
- tool_healing's wrapped Gemma opener tolerates whitespace around call and
the colon: sampling drift emits call: name{ and call : name{, and
rejecting those lost the call entirely because no fallback re-parses the
wrapped form. Strict mode still requires the closing tag.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: accept dotted Gemma argument keys in the key-quoting scanner
The scanner quoted keys of [alnum_-] only, so a dotted key (user.name:...)
was left unquoted, json.loads failed, and the whole wrapped call was lost
(parse empty, strip wipes the markup). Dots now match the parser's own
key/name charset.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: leading Mistral call owns the turn, dotted keys after bare values
- A LEADING parseable [TOOL_CALLS] call now runs the Mistral parser first
unconditionally: literal XML in trailing prose after the call was promoted
by the earlier shared XML pass, executing the quoted example instead of
the real leading call. XML leading keeps the normal order.
- _GEMMA_NEXT_KEY_RE accepts dots so a dotted key after a bare value
(query:foo,user.name:bob) ends the value at the comma instead of being
swallowed into it, matching the round-earlier key-quoting charset.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: markup quoted inside a nameless leading JSON answer stays data
The leading bare-JSON guard required a top-level name, so a structured JSON
answer quoting tool markup in its strings (a response_format turn
documenting a tool's syntax) had the literal promoted by the later passes.
A nameless leading object that parses as real JSON now routes through the
same decline-then-parse-the-tail path; non-JSON braced prose keeps the old
behaviour, and a real call after the answer still parses.
* Compress docstrings in the multi-format tool parser to their contract essence
* verify_import_hoist: exempt __future__ imports and same-diff relocations
Two false positives fired on this PR's refactor. A from __future__ import
is a compiler directive whose name never appears as a runtime load, so
HOISTED-IMPORT-UNUSED can never see it used, yet the file requires it for
PEP 604 annotations on Python 3.9. TARGET-CHANGED flagged the deliberate
move of the strip-pattern constants into core.inference.tool_call_parser
as a silent re-point even though the old module-level target was removed
and the new one added in the same diff. Both get narrow exemptions; a
re-point to a pre-existing target is still caught, and the self-test
negative controls all pass unchanged.
* Leading bare-JSON calls own the turn; function calls end at the first balanced close
The XML-signal guard for a leading bare-JSON call required the signal
strictly inside the object, so a trailing XML example stole the turn
from the leading call; it now applies the same inside-or-after rule as
the Mistral guard. Function-XML calls also ended at the LAST close tag,
which let prose after a closed call that mentions a literal close tag
get swallowed into the final parameter value; calls now end at the
first close tag that is not inside an open parameter, and the strip
mirrors the same rule so parse and strip agree.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Attribute-form calls end at the first balanced close; bare-JSON strip requires the call shape
The attribute form parser still kept the last close tag in the call
window, folding prose after a closed call into the final parameter
value. It now takes the first close not inside an open parameter, the
same rule the equals form and the strip already use.
The leading bare-JSON strip deleted any closed object whose top-level
name matched an enabled tool, including plain JSON answers the parser
correctly rejects as non-calls. The strip (and the drain gate that
delegates to it) now requires the parser's exact call shape, so answers
like {"name":"web_search","result":...} stream and display intact.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* False-alarm markers keep the answer; the bare-JSON strip consumes the whole chain
The trailing strip arms dropped everything from a bare marker to EOF,
so a normal answer that mentions [TOOL_CALLS] or another marker
literally was truncated (or fully swallowed when it started with the
literal) after the no-call drain fallback. Those arms now require a
call-shaped lookahead or marker-at-EOF before dropping; truncated real
calls still strip.
Chained bare-JSON turns executed both calls but stripped only the first
object, so the second call's raw JSON replayed into the next assistant
history message alongside the structured tool_calls. The strip now
consumes the entire chained run of call-shaped enabled objects while
non-call answers, disabled names, and trailing prose stay intact.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Attribute-form containment, parameter-close-decides rule, preamble-tolerant Mistral guard, strict strip shape
Four document-order and containment fixes. A leading attribute-form
call now parses before the shared XML pass, so markup quoted in its
parameter stays data. The open-parameter scan lets the parameter's own
close tag decide, so any number of literal function closes inside one
value stay data, restoring the pre-close-scan behavior for multi-close
arguments. The leading-Mistral guard tolerates a visible preamble, with
the leading-bare-JSON guard running first so a trigger quoted inside a
leading JSON object stays data. The bare-JSON strip requires the
parser's top-level name in every mode, so nested-name JSON answers
survive name-agnostic stripping.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Let a leading <|python_tag|> call own the turn over quoted XML literals
The leading-call ownership contract (a leading executable call owns the turn;
foreign markup quoted in its string arguments or trailing prose stays data) was
enforced for the bare-JSON, Mistral and attribute-form leading calls but not
for the Llama-3 <|python_tag|> form. The shared tool_healing XML pass runs
before _parse_llama3_python_tag and does not recognise <|python_tag|>, so a
<function=...> / <tool_call> / [TOOL_CALLS] literal quoted inside a
<|python_tag|> .call(...) string argument (or its JSON parameters) was promoted
and the wrong tool executed. Well-formed single-format examples:
<|python_tag|>web_search.call(query="... <function=foo> ...") -> foo
<|python_tag|>python.call(code="<function=render_html>..</function>") -> render_html
both returned the phantom inner tool instead of the real leading call.
Add a leading-<|python_tag|> guard mirroring the other leading-call guards:
when the tag is the first tool signal, parse it before tool_healing so quoted
foreign markup stays data. A foreign signal before the tag keeps normal
document order. Added TestPythonTagOuterOverXmlLiteral (7 cases).
* studio: tighten tool-calling comments to be shorter and clearer
* studio: shorten tool-format comments in changed files
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: Daniel Han <info@unsloth.ai>
Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>
* Fix external drive custom folder selection
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/backend/tests/test_linux_external_media_paths.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep legacy media scan validation strict
* Apply sensitive-dir denylist to legacy folder browser for PR #6799
The legacy /api/models browse endpoint gained the new /run/media mount
roots in its allowlist but not the credential/config guard that scan-folder
registration and the Hub browser already enforce. Filter sensitive names
during enumeration and reject them in _resolve_browse_target so .ssh, .aws,
.config, etc. under allowlisted roots stay unbrowseable, matching the Hub
browser. Add a public contains_sensitive_path_component helper and cover the
legacy resolver with a regression test.
* Trim redundant comments in PR #6799 changes
* Skip sensitive Linux media roots
* Reject sensitive dirs at exact browse roots for PR #6799
Both _resolve_browse_target functions only checked contains_sensitive_path_component
while walking descendant parts, so requesting an allowlisted root itself (empty
relative path) returned it unchecked. A pre-existing scan-folder row under ~/.ssh,
~/.aws, ~/.config, etc. (registerable before the denylist was added) is re-added to
the allowlist on upgrade and could then be browsed. Check the resolved target once
before returning in both the legacy and Hub browsers, and cover the root case in
both test suites.
* fix: avoid unused path helper reexports
* fix: import sensitive path helpers directly
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: flush passthrough stream headers before upstream prefill stalls
* Studio: clean up delayed passthrough send failures
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: close passthrough preheader cleanup gaps
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: retry delayed passthrough overflow truncation
* Studio: close completed passthrough send responses
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: expose full compressed-tensors scheme set in an export formats dropdown
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: multi-select export formats, portable torchao FP8/INT8, GGUF LoRA, source parity
Export page overhaul on top of the formats dropdown:
- Unify merged precision into one sorted multi-select list (16-bit first, then
8-bit, then 4-bit). Drop "vLLM" from labels, add INT8 (W8A8), INT8 (W8A16),
INT4 (W4A16), MXFP4, MXFP8. Quick formats render as toggle pills; the rest live
in a multi-select "More formats" dropdown, so several formats export in one run.
- Add a portable torchao FP8/INT8 save path (Float8WeightOnlyConfig /
Int8WeightOnlyConfig) that needs no NVIDIA GPU to produce and loads in vLLM.
FP8 serializes to safetensors, INT8 to .bin. Wired into save_pretrained_merged
and push_to_hub_merged via a TORCHAO_EXPORT_SCHEMES registry and
_unsloth_save_torchao, parallel to the compressed-tensors path.
- Hide NVIDIA-only compressed-tensors formats when no NVIDIA GPU is present; keep
16-bit and portable FP8/INT8. The backend also rejects a compressed request on
non-NVIDIA hardware so it stays authoritative.
- Relax merged export to non-PEFT models so Local Model and Hugging Face sources
get the same 16-bit / compressed / portable options.
- GGUF: send the whole quant list in one call (merge once, quantize many).
- LoRA: add a GGUF adapter option (convert_lora_to_gguf.py) with an outtype
select (f16/bf16/f32/q8_0/auto), alongside the safetensors adapter.
- Thread the new fields through models, routes, orchestrator, and worker; extend
the export tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate export by accelerator with a torch-aware reason; fix export save dir naming
Export runs through Unsloth, which requires a compute accelerator (NVIDIA/AMD/Intel
GPU or Apple MLX) and has no CPU code path, so a bare-CPU host cannot export even
with PyTorch installed. Add export_capability() in utils/hardware that reports
export_supported plus a precise reason so the UI stops showing a generic "no GPU":
- pytorch_not_installed: a --no-torch install (even a physical GPU is unusable)
- no_accelerator: PyTorch present but no supported accelerator (bare CPU)
- mlx_unavailable: Apple Silicon where the MLX stack is missing or too old
Expose the fields on /api/system/hardware and /api/system, and guard the mutating
export routes (load-checkpoint, export/merged|base|gguf|lora) with HTTP 400 and the
reason, leaving read-only endpoints usable so the Export page still renders.
Make core/export/export.py import without PyTorch and without a usable accelerator
(the Unsloth import is caught) so the export worker degrades to a clear message
instead of crashing at import.
Frontend: keep /export reachable on chat-only hosts and gray out the method and
format options with the backend reason (Alert plus disabled MethodPicker) instead
of silently redirecting to /chat, so users see why export is unavailable.
Also fix the export save directory producing "model/null" for Local Model and
Hugging Face sources that have no run/checkpoint, naming the folder from the model id.
* CI: validate Studio export capability gating on Linux, Windows and macOS
Add a small pytest matrix that runs studio/backend/tests/test_export_capability.py
on ubuntu-latest, windows-latest and macos-latest. It confirms, on each real OS,
that hardware.export_capability() reports the right decision and reason
(pytorch_not_installed, no_accelerator, or mlx_unavailable) and that the export
backend imports without PyTorch and degrades to a clear message instead of crashing.
Hosted runners have no GPU/MLX, so this covers the "export unavailable, here is why"
path a Mac/Windows user without an accelerator sees; a real accelerator export is
validated separately. The job installs only a CPU PyTorch plus the backend import
deps (no unsloth, triton, or llama.cpp), so it runs in seconds with no GPU.
* Studio export: address Codex review (source-aware gating, GGUF LoRA token/MLX/guard)
Frontend (export-page):
- Gate LoRA and quantized-model restrictions on the active source. isAdapter /
isQuantized come from the selected checkpoint; in Local Model / Hugging Face
("model") source mode they were stale, so LoRA stayed wrongly enabled for a
direct base model (backend then rejects "No adapter to export") and a stale
"quantized" flag disabled every method for an unrelated, exportable model. Add
effectiveIsAdapter / effectiveIsQuantized (false outside checkpoint mode) and use
them in the method-reset effect and the MethodPicker disabled state.
- Hide the GGUF LoRA option on a macOS/MLX host (the backend rejects GGUF LoRA on
MLX), so users no longer pick it, wait through the load, and always fail. Disable
the "GGUF adapter" button on a Mac host and never send loraGguf there.
Backend (core/export/export.py):
- Pass the HF token into the GGUF LoRA conversion (save_pretrained_gguf), so a
gated/private base model's config fetch in convert_lora_to_gguf.py is
authenticated; without it the load can succeed but the conversion fails.
- Guard the save_pretrained_gguf capability check with getattr so an older Unsloth
model that lacks the method returns the clean "not supported" message instead of
an AttributeError that surfaces as a generic 500.
* Studio export: address 2nd Codex review (CI index, empty merged, test import)
- studio-export-capability-ci.yml: add --extra-index-url https://pypi.org/simple to
the torch install so torch's transitive deps still resolve; --index-url alone
replaces PyPI with only the CPU wheel index, which does not serve all of them.
- export-page handleStart: reject an empty merged selection (mirrors canExport), so
clicking the panel's Start button with every precision pill deselected no longer
submits mergedSelections: [] and launches an unintended default 16-bit export.
- test_export_imatrix_compressed: the torchao-registry test now reads unsloth/save.py
as text (like the other ast/string checks) instead of `import unsloth.save`, which
raised ModuleNotFoundError in the CPU studio-backend suite that has no unsloth
installed.
* Studio export: make comments succinct across the export changes
* Studio export: use load token for local GGUF LoRA export of gated bases
* Studio export: harden portable torchao path and gate multi-format Hub push
torchao (_unsloth_save_torchao):
- merge to an isolated temp staging dir so a co-selected 16-bit output at save_directory is not deleted
- narrow VLM detection to vision_config / ForVisionText2Text so T5/BART/Whisper are not misrouted
- forward trust_remote_code (from auto_map) to the reload so custom-code models export
Export UI:
- hide portable torchao formats on macOS/MLX (backend rejects quantized export there)
- restrict a Hub merged export to a single format (each writes to the repo root)
* Studio export: torchao tokenizer remote-code + XPU offload, scale GGUF timeout
torchao (_unsloth_save_torchao):
- honor auto_map in the staged tokenizer/processor configs (not just model.config) when
deriving trust_remote_code, so custom-code tokenizers reload after the merge
- offload single-device XPU models to CPU (and empty the XPU cache) before the reload, matching
the CUDA path, so an Intel GPU that fits the model once does not OOM on the second copy
Export orchestrator:
- scale the GGUF wait timeout by the number of requested quants so a multi-quant list export of a
large model does not time out at a flat 3600s
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio export: show portable torchao formats only on non-NVIDIA (CPU) hosts
Portable torchao FP8/INT8 is the fallback for hosts without the NVIDIA compressed-tensors path.
On an NVIDIA GPU the compressed-tensors FP8/FP4/INT formats are the intended path (llm-compressor
auto-installs), so hide the portable duplicates there; keep them on CPU / non-NVIDIA hosts and
continue hiding them on macOS/MLX.
* Studio export: report all output folders and the exported formats
- Multi-format merged export now collects every sibling output directory (one per selected
precision) instead of only the last; the success banner lists them all.
- Show the selected precision formats in the run summary (a Formats row, like GGUF Quantizations),
so the panel says what is being exported rather than just 'Merged Model'.
- Persist the selected formats in the run summary and seed them on mount, so navigating away and
back (or toggling the export method) restores the selection instead of resetting to 16-bit.
* Studio export: list all output formats, add GGUF LoRA target, default Q8_0, auto-select newest checkpoint
- Progress/summary panel now shows a Formats row with the selected merged
formats, and the success banner lists every output folder a multi-format
merged run creates (one line per format) instead of only the last one.
- Merged format selection is seeded from the active run, so navigating away
and back (or switching method cards) no longer resets it to 16-bit.
- GGUF / Llama.cpp now offers an Export target toggle (Full model or LoRA
adapter) for adapter checkpoints, reusing the LoRA GGUF export path.
- Removed the Auto GGUF LoRA output type and defaulted to Q8_0 in the UI,
the request model, and the backend defaults; the outtype list is now
Q8_0/F16/BF16/F32. Core save.py still accepts auto for external callers.
- When a finetune has no checkpoint selected, auto-select the newest one.
* Studio torchao export: robust reload class + optional VLM import
Two fixes to the portable torchao FP8/INT8 export reload, from review of the
narrowed VLM detection:
- Encoder-decoder seq2seq checkpoints (T5/BART/Whisper) are not causal LMs.
With the narrowed is_vlm test they now correctly skip the image-text class,
but fell through to AutoModelForCausalLM and failed to reload after the merge.
Reload them with their own architecture class from the config instead.
- AutoModelForImageTextToText was imported unconditionally at the top of the
torchao path, so on Transformers builds without that class the import aborted
every torchao export (even text-only). Import it lazily only for a VLM, with
the AutoModelForVision2Seq fallback used elsewhere in Unsloth.
* Studio: enable FP8/FP4 compressed export for newer-transformers models
The shipped llm-compressor 0.10.x pins transformers<=4.57.6, so FP8/FP4 export failed
for models needing a transformers 5.x sidecar (Qwen3.5, Gemma-4, Qwen3-Next): the
quantization subprocess crashed importing the removed TORCH_INIT_FUNCTIONS.
Run the quantization against a dedicated llm-compressor-main "shadow": a --target
package dir (transformers 5.10.2 + llm-compressor main + compressed-tensors) layered
over the existing torch. It installs --no-deps so torch is never touched (works on any
Studio torch build), is provisioned lazily and fingerprint-cached, and can be turned
off with UNSLOTH_DISABLE_LLMCOMPRESSOR_MAIN.
- transformers_version.py: provision + validate .venv_llmcompressor.
- export.py: route all compressed exports through the shadow when available; else keep
the workspace 0.10.x path and fail fast past its transformers ceiling.
- save.py: launch _compressed_quantize.py with a clean PYTHONPATH = shadow.
- _compressed_quantize.py: skip linear_attn / vision tower / MTP modules (matches the
RedHatAI and NVIDIA reference quants, and is required by the grouped schemes).
Verified all four schemes (fp8, w8a8, w4a16, mxfp4) on Qwen3.5-9B and Llama-3.2-1B, and
fp8 on Gemma-4, end to end through Studio.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix GGUF LoRA export tests
* Fix export CI expectations
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <112766706+wasimysaid@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* inference: add passthrough tool-call healing core (heal_gate, heal_openai_message, StreamToolCallHealer, nudge helpers)
Small GGUF models often emit tool calls as text (<tool_call>{...}</tool_call>,
Gemma <|tool_call>, <function=> XML) instead of structured tool_calls. Studio's
enable-tools loop already heals these, but the client-tool passthrough
(unsloth run --disable-tools, unsloth start agents) relays them verbatim, so
the agent sees prose and the turn dies.
This module is the shared response-side repair layer the passthrough routes
will call: promote parsed text-form calls to structured calls, but only for
function names the client actually declared; coerce arguments through the same
canonical-key healing as the tool loop; never touch the upstream request body
(llama-server KV/slot reuse stays byte-identical). StreamToolCallHealer is the
streaming buffer-and-repair state machine: prose forwards immediately, only a
partial-signal tail or a suspected tool block is held, false alarms flush
verbatim, and a 64 KiB bound caps memory. nudge_should_retry/nudge_messages
support an opt-in single-retry nudge for non-streaming routes (wired later).
Kill-switch: UNSLOTH_DISABLE_TOOL_CALL_HEALING=1. Reuses
core/tool_healing.parse_tool_calls_from_text, strip_tool_call_markup, and
tool_loop_controller.coerce_tool_arguments unchanged.
* inference: heal text-form tool calls on the OpenAI and Responses passthrough
Wire the passthrough healing core into /v1/chat/completions and /v1/responses,
default ON whenever the request declares client tools:
Non-streaming: heal_openai_message runs inside the existing response-mutation
loop; a promoted call flips finish_reason to tool_calls and nulls the content,
and the verbatim-bytes fast path still applies when nothing was healed.
/v1/responses non-streaming inherits this through openai_chat_completions.
Streaming: a StreamToolCallHealer per stream. Ordinary prose relays
byte-for-byte (a fast path keeps upstream bytes when the healer passes a chunk
through whole); once a tool signal appears, content is held, and at the
finish/[DONE] boundary either synthetic delta.tool_calls chunks replace the
markup (finish_reason rewritten to tool_calls, including the synthetic-finish
path) or a false alarm flushes the held text verbatim. Structured upstream
deltas put the healer to sleep after flushing anything held, so grammar-mode
responses stay byte-identical. The Responses stream feeds healed calls through
the same per-call state machinery as structured deltas (indexes live in a
disjoint range so a healed call can never merge into a structured call's
state), and the visible/reasoning split runs first so reasoning text is never
promoted. parallel_tool_calls=false caps healed calls on every path.
The upstream request body is never touched and healing issues no extra
generation, so llama-server slot/KV-cache reuse is unchanged. Opt-out per
request with auto_heal_tool_calls=false (Responses reads it from the
extra-body); requests without tools relay verbatim.
* inference: heal text-form tool calls on the Anthropic /v1/messages passthrough
Streaming: AnthropicPassthroughEmitter.enable_healing(allowed_tools) routes
content deltas through the shared StreamToolCallHealer. A promoted call closes
any open text block (only the safe prose prefix ever streamed into it), opens a
synthetic tool_use block with a fresh toolu_* id, carries one input_json_delta,
and closes; finish() then forces stop_reason to tool_use unless a truncation
(max_tokens) wins. Structured upstream deltas flush anything held and put the
healer to sleep, so grammar-mode responses are untouched, as is every stream
where enable_healing is never called (Studio's own loop, no-tools requests).
disable_parallel_tool_use caps healed calls too.
Non-streaming: the OpenAI message dict is healed BEFORE block building, so the
existing tool_use promotion loop and stop_reason line treat promoted calls
exactly like native ones (finish_reason length still maps to max_tokens). The
legacy tool-XML strip still runs on remaining text, so opted-out requests keep
today's cleanup behavior byte-for-byte.
auto_heal_tool_calls is now a typed field on AnthropicMessagesRequest
(default True, mirroring Chat Completions) and threads into both passthrough
calls. Healing never touches the upstream request body.
* inference: opt-in single-retry tool-call nudge on the non-streaming passthrough
When the model clearly tried to call a tool (a tool signal in the text) but
healing produced nothing usable, re-ask once: the retry body is the original
body plus an assistant turn (the model's own failed text) and a short user
nudge naming the declared tools. The prompt prefix stays byte-identical, so
llama-server reuses the slot's KV cache and only the two-message suffix is
prefilled. The retry replaces the original response only when it actually
yields a promotable or structured call; on any error or still-garbage output
the original response is returned unchanged. Exactly one retry, non-streaming
OpenAI and Anthropic passthroughs only (a stream has already emitted bytes).
OPT-IN per user decision: nudge_tool_calls=true per request (typed on both
ChatCompletionRequest and AnthropicMessagesRequest, lifted from the Responses
extra-body), or UNSLOTH_TOOL_CALL_NUDGE=1 to flip the process default.
auto_heal_tool_calls=false disables healing AND the nudge.
Also align the non-streaming heal on allow_incomplete=True: the response is
final, so a trailing unclosed tool block is a model failure worth repairing,
matching the enable-tools loop's drain semantics.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: never assume the upstream response shape in the nudge helpers
llama-server error bodies can carry message: null (or no choices at all), and
_last_assistant_text / response_has_promotable_calls / nudge_should_retry
called .get() on the message without a dict check, so a malformed upstream
response raised an AttributeError the surrounding except tuples did not catch,
failing the request instead of degrading to 'nothing to heal'. Route the shape
probing through one _first_choice_message helper that returns None for any
non-dict message, and add a parametrized test over the malformed shapes.
* inference: constrain healing by tool_choice, preserve length finish_reason, keep healed event order in Responses streams
Three review findings on the passthrough healer:
- heal_gate now honors the request's tool_choice: "none" disables healing
outright and a forced function narrows the promotion allowlist to that
one function, so healing can never contradict the request's tool-choice
constraint. Wired through the OpenAI chat (stream and non-stream),
Responses, and Anthropic (converted shape) passthroughs.
- The OpenAI non-streaming heal only upgrades finish_reason "stop" to
"tool_calls"; a truncated generation keeps "length" (the healed call
stays attached) matching the streaming and Anthropic paths.
- The Responses stream emits healer events in order instead of collapsing
all text ahead of the healed calls, so text after a healed call no longer
jumps ahead of the function_call item and output indexes are claimed in
the order the model produced them.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: all-or-nothing promotion when a response mixes declared and undeclared text-form calls
Promoting a subset used to strip ALL tool markup from the content, which
silently deleted the text of any call naming an undeclared tool. The heal
now declines entirely when any parsed call is unpromotable, so the whole
message relays verbatim (pre-PR behavior) and no bytes are ever lost. In
streaming, a declared call that completed before an undeclared one arrived
is already emitted; the late undeclared markup still flushes as raw text.
The nudge helpers mirror the same contract via a shared predicate.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests: wrap long lines in the Responses healing tests to the project style
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: span-exact healing, disjoint healed stream indexes, per-call Responses message items, allowlisted nudge acceptance
Four review findings on the passthrough healer:
- parse_tool_calls_from_text gains an optional with_spans return so healing
removes EXACTLY the promoted calls' markup. This supersedes the previous
all-or-nothing rule: declared calls promote and every unpromoted byte
(undeclared calls, unparseable closed blocks, suppressed alternate
formats such as a <function=...> block after a JSON call) relays as text.
The stream healer also processes one block per pass, so text between two
healed calls keeps its document position instead of trailing them.
- The OpenAI chat stream shifts native tool-call delta indexes past any
already-emitted healed calls; clients merge deltas by index, so a healed
call and a later native call can no longer merge into one.
- A healed call in the Responses stream closes the open message item and
trailing text opens a fresh one with a later output index, matching the
native stream shape; response.completed snapshots every message item
with its own text.
- The nudge retry only replaces the original response when the retry's
structured call names a DECLARED tool; a hallucinated undeclared call is
not an improvement.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stop the heal path folding trailing prose into a closed function call
parse_tool_calls_from_text(allow_incomplete=True) cut a <function=...> body only
at an end-anchored </function>, so a fully closed call followed by trailing prose
(<function=..>..</parameter></function> words) folded </parameter></function> and
the prose into the tool argument and deleted the prose from visible content. The
strict path (allow_incomplete=False) already cut at the real </function> via rfind.
Do the same in both modes: trim the body at the real </function> when present and
end the removal span there, falling back to the end-anchored strip and body_end
only when the call is genuinely truncated. Add a regression test.
* inference: one shared single-call budget for healed and native calls
Codex round 5: the parallel-call caps counted healed and native calls
separately, so a healed text-form call followed by a native structured
delta double-emitted on all three streaming surfaces when the client
disabled parallel calls.
- OpenAI SSE: once a healed call went out with parallel_tool_calls
false, native tool_call deltas are dropped instead of index-shifted.
- Anthropic emitter: native deltas skip block allocation when the
healed-plus-native count already filled the single slot, and healed
emission counts open native states too.
- Responses stream: native deltas that survived the chunk-level cap are
skipped once a healed call claimed the slot.
Also adds a span assertion for the closed-</function> trailing-prose
parse fixed in the previous commit.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: relay undeclared text-form calls as text on Anthropic non-streaming
heal_openai_message promotes only declared text-form tool calls and
span-trims just their markup, deliberately leaving every unpromoted byte
(undeclared text-form calls included) in the content to relay as text.
The Anthropic non-streaming builder then ran a blanket _TOOL_XML_RE strip
over that content unconditionally, deleting the undeclared block before
building the text part, so Anthropic clients silently lost a call the
OpenAI non-streaming path preserves. The strip was harmless when healing
was all-or-nothing but became data loss once healing turned span-exact.
Gate the legacy strip on whether healing promoted a call, matching the
OpenAI passthrough and the intent already stated in the comment above.
Add a route-level regression test for the mixed declared+undeclared case.
* inference: require fully declared nudge retries; keep unpromoted Anthropic text
Codex round 6, two findings:
- response_has_promotable_calls accepted a nudge retry when any one
structured call named a declared tool, so a mixed retry (hallucinated
undeclared call plus a declared one) replaced the original and the
caller forwarded the undeclared call, or with parallel_tool_calls
false could keep only it. All structured retry calls must be declared.
- The Anthropic non-streaming builder still ran the legacy _TOOL_XML_RE
strip after span-exact healing, deleting undeclared or malformed call
text that healing deliberately preserved. The legacy strip now runs
only when healing is off (no declared tools, or opted out), matching
the OpenAI passthrough.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: keep unpromoted Anthropic text whenever healing is active
The previous commit skipped the legacy strip only when a call was
actually promoted, so an undeclared-only (or malformed-only) response
was still silently emptied: exactly the dead-turn shape this path
exists to fix, and inconsistent with the OpenAI passthrough, which
relays those bytes verbatim. Gate the strip on healing being active
instead; opt-out and no-tools requests keep the legacy strip.
* Fix schema-aware tool healing for PR #6801
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix passthrough healing ordering for PR #6801
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix stream finish ordering for PR #6801
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: wasimysaid <112766706+wasimysaid@users.noreply.github.com>
oobabooga