* refactor(studio): move chat model picker into features/model-picker
Relocate model-selector + its support files from components/assistant-ui
into a self-contained features/model-picker feature (own barrel), mirroring
the modular Hub layout. Pure move + import repoint; no behaviour change.
* feat(model-picker): add per-model config persistence layer
Superset PerModelConfig (customContextLength, kvCacheDtype, speculativeType,
specDraftNMax, tensorParallel, chatTemplateOverride, trustRemoteCode) persisted
to localStorage (unsloth_model_configs) with schema versioning + LRU budget.
KV-dtype and speculative value sets match main's sidebar (no q4_0/ngram-simple).
Reuses features/hub/lib/model-identity for normalization; adds storage-key layer
and applyPerModelConfigToRuntime (sets tensorParallel, which the old PR omitted).
* feat(picker): modular backend for chat-template validate + default fetch
New studio/backend/picker package (schemas/service/routes) mounted at /api/picker:
- POST /api/picker/validate-chat-template (Jinja syntax validation, no false positives)
- GET /api/picker/chat-template/{model_name} (default template from tokenizer_config.json,
reusing get_cache_path/resolve_cached_repo_id_case; graceful null, no model-code exec)
Frontend api/templates.ts client + hooks/use-model-defaults lazy cache. No backend
changes to the existing inference load route (per-model load fields already supported).
* feat(model-picker): bind picker on-device list to shared hub inventory
Picker now sources cached + local models from useHubInventory (the Hub's shared
store) via a thin adapter, replacing its own /api/models/* fetchers + module
caches. Hub, download manager, and picker now share one source of truth, so
completed downloads reflect in the picker automatically. Partial/live-download
rows are filtered from the cached lists (unchanged rendering). Local naming/search
preserved via additive LocalInventoryRow modelId/displayName. Variant expander,
scan-folder management, recommended-fit, search, external providers untouched.
Known minor: cached 'Downloaded date' sort tiebreak degrades to alphabetical
(hub cached rows carry no mtime); default 'recent' (load-time) sort preserved.
* feat(model-picker): per-model config step inside the picker
Picking a (non-external) model now opens an in-picker config view built from
main's current load controls (context length, KV cache dtype, speculative
decoding, draft tokens, tensor parallel) plus a chat-template editor backed by
the picker validate/default endpoints. 'Remember for this model' persists the
config per model+variant; Run forwards the config to the existing load flow via
meta.config. External models bypass the step. Two-view orchestration lives in
model-selector (single interception point); pickers.tsx call sites untouched.
trustRemoteCode dropped from PerModelConfig to preserve main's per-load consent.
* feat(chat): apply/persist per-model config through the load flow
handleCheckpointChange threads meta.config into the selection; stageOrLoad and
the autoload/Hub-run paths now apply the picker config (explicit pick or saved
remembered config) via applyPerModelConfigToRuntime before staging/loading, with
keepSpeculative set so a remembered speculative mode survives the model switch.
Replaces the old remembered-load-settings seeding (resolveInitialConfig now the
single source). SelectedModelInput carries config.
* refactor(chat): remove per-model load config from the right sidebar
The load knobs (context, KV cache, speculative, draft tokens, tensor parallel)
and the chat-template editor now live only in the picker config step. The sheet's
Model section keeps the staged Load/Cancel flow (config is applied at pick time);
sampling params, system prompt, and RAG are unchanged. Deletes the superseded
remembered-load-settings module + the store's applyRememberedLoadSettings action,
removes the now-dead sheet state/imports, and points the settings reset at
unsloth_model_configs. Delete-cleanup deferred (stale config is LRU-capped).
* fix(model-picker): remove leftover sidebar-staging cogwheel + empty Model section
The downloaded-variant gear (ModelLoadSettingsAction) staged a model straight
into the right-sidebar Run-settings flow -- the old 'configure before load' path
now fully replaced by the in-picker config step. Removed the gear + its component.
Also gate the sheet's 'Model' section to staged picks only (pendingSelection):
after the load-knob strip its content is staged-only, so it was rendering an
empty section header whenever a model was merely loaded.
* chore(chat): remove dead per-model-config setters + modelControlsDisabled
After the load-config UI moved into the picker, the store's per-model setters
(setKvCacheDtype/setSpeculativeType/setSpecDraftNMax/setTensorParallel/
setCustomContextLength/setChatTemplateOverride) had zero callers
(applyPerModelConfigToRuntime writes via setState), and the sheet's
modelControlsDisabled was unreferenced. Verified dead across the whole tree.
* fix(chat): config-step Load actually loads (ignore Load-on-selection)
Root cause: with Settings > Chat > 'Load on selection' turned OFF, the config
step's load went down the deferred-staging path -- opening the right sidebar with
'<model> is staged, not loaded yet / Choose Load model'. The in-picker config step
IS the deliberate load action, so its Load now loads immediately (or downloads +
auto-loads when not cached) regardless of the toggle. Renamed the button
'Run model' -> 'Load model' to match. Native/dropped picks still honor the toggle.
* refactor(chat,hub): retire 'Load on selection' — config step is the only load flow
The in-picker config step (and the Hub Run button) now fully supersede the old
stage-to-sidebar flow, so the Load-on-selection toggle is removed everywhere:
- chat stageOrLoad: every pick loads immediately, or downloads + auto-loads when
not cached (the previous default behaviour, now universal).
- hub Run: drops the stage branch; downloaded GGUFs load directly with their saved
per-model config (no collision with the chat config step — both end at selectModel).
- store: removed loadOnSelection field/setter/key/default; Settings>Chat toggle and
its settings-reset entry removed.
- staged sidebar section is now a download-progress view (auto-loads on completion).
No manual staging remains; stageModel is used only for background auto-load downloads.
* feat(model-picker): default chat template from GGUF + thread variant through config flow
Read the embedded tokenizer.chat_template from GGUF files (read_gguf_chat_template
in gguf_metadata) and use it as the per-model default. Plumb gguf_variant through
the picker service, /api/picker/chat-template route, frontend templates API, and
use-model-defaults so the right variant's template is fetched.
Also refine the picker config-page/model-selector wiring, drop the dead
ggufNativeContextLength runtime path, and add the per-model-config storage keys to
the settings prefs export.
* feat(model-picker): read safetensors chat template + hide editor where it has no effect
Resolve the default chat template for safetensors models: prefer the modern
chat_template.jinja, fall back to the tokenizer_config.json chat_template field,
then chat_template.json (multimodal processor), then the GGUF embedded template.
Applied to local dirs, the HF cache snapshot scan, and the HF remote fetch.
Hide the chat-template editor in the picker for safetensors models — the override
is only applied at load by the GGUF/llama.cpp backend, so editing it on safetensors
currently has no effect. GGUF keeps the editor. Nothing removed; the dialog stays
for when the safetensors apply path is wired up in a later branch.
* fix(model-picker): set legacy-migration flag only after the write succeeds
Set unsloth_model_configs_migrated only once writeMap confirms the migrated
map persisted, so a quota/storage failure no longer marks migration done and
silently drops the user's pre-existing remembered settings — the next load retries.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* MVP model picker fixes
* MVP picker config fix
* MVP safetensors config
* MVP max seq config
* MVP max seq fix
* Fix static max tokens cap ignoring model context
* Fix picker GGUF scan parity
* fix(studio): harden model picker config loading
Apply remembered per-model configs consistently from picker and Hub loads, keep default configs from overriding standing speculative settings, add config access for direct local GGUF files, and support saving or forgetting active model settings without a reload.
* Fix model picker config flow
* Fix model picker config loads
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Avoid recursive per-model config migration reads
* Apply the displayed context length when loading a GGUF
* Fix template validation, cached template lookup, and failed load rollback
- Validate chat templates with the loopcontrols extension so templates
that use break or continue tags pass the picker validator, matching the
inference renderer that already accepts them.
- Read the default chat template from the newest cache snapshot rather than
an arbitrary iterdir order, so an older cached revision no longer prefills
a stale template.
- Capture the runtime per-model config before a load and reapply it when the
load fails, so a failed switch leaves the active model context, KV cache,
template, and speculative settings as they were.
* Make chat template view only for safetensors models
Custom chat template overrides are applied at inference only for GGUF
models, which pass the template to llama-server. The safetensors backend
renders with the model built-in template and ignores the override, so
editing it would save a value that never loads. For safetensors the
config page now opens the template as a read-only preview with a note
that editing is not available yet. This can become editable once
inference support for custom safetensors templates lands in main.
* Fix model picker config edge cases
- Restore prior runtime config when a load no-ops for the active model
- Cap the picker validator request body via the protected prefixes
- Keep the GGUF context slider max above the loaded context
- Fetch subfolder chat templates for uncached Hub repos
- Show the compare side config when reopening the picker
* Keep saved GGUF context above the fallback ceiling
* Show the model config in the run settings sidebar
* Fix model config sidebar reset and context slider
- Stack the remember toggle and action buttons in the sidebar
- Reset the config to defaults instead of the loaded values
- Fetch the native context so the slider max is not the loaded value
* Fix model picker config and download regressions
- Run picker chat template routes off the event loop
- Depth and root guard local template directory scans
- Restore download manager flow for uncached hub picks
- Apply per model context length on reload
- Import model picker symbols from the feature barrel
* Fix model picker config and cached download sorting
- Restore load settings when a Hub run is rejected mid load
- Reuse one NumericValueInput instead of a duplicate copy
- Fix double decode of the model name in the template route
- Remove the unused reset-to-loaded settings action
- Fix cached model download sorting
* Fix model picker per-model config edge cases
Honor a saved or typed max seq length above the model's native context so
RoPE extended values are no longer clamped and silently overwritten. Allow
typing past native while the slider keeps native as a soft ceiling.
Guard the fetch success paths in use-model-defaults against an aborted
signal, and refetch when the HF token changes.
Hash the chat template content in the sidebar remount key instead of its
length. Enable reset for a GGUF whose native context is unknown, and floor
the context slider max so it can never fall below the min.
* Fix GGUF context auto-fit and gated model config token
Stop forcing a 32768 context when a GGUF native context is unknown so the backend auto-fits to VRAM again, while still honoring an explicit context edit.
Send the HF token as a query param so gated safetensors models resolve their max position embeddings.
Derive model default state during render to drop the set-state-in-effect calls.
* Fix native GGUF context ceiling and guard picker template reads
Restore the native context store field so the sidebar slider keeps the
full ceiling for drag and drop GGUFs. Limit local chat template reads to
the browse allowlist, skip malformed repo ids, and drop unused model
picker exports.
* Fix model picker lint boundaries
* Fix model picker review findings
Chat template editor never seeded its draft. Radix only calls onOpenChange
from internal events, so the seed in the nextOpen branch was dead and a model
with a saved override opened empty. Saving then cleared the override. Drop the
dead branch, treat draft as an untouched sentinel, and reset it on every close.
Uncached Hub picks could auto load a model after the user left the chat. Main
detached the staged pick on route exit and on chat context change. Carry the
context key on the pending pick and skip the load when it no longer matches.
Also clear configTarget when the picker closes, restore the onUpdated ref so
variant rows stop resubscribing on every parent render, skip the LRU write when
the entry is already most recent, import NumericValueInput relatively, and drop
the unused ModelUpdateAction barrel export.
* Preserve GGUF context on active reload
* Fix model picker per-model config regressions
- Stop reloading the already loaded model on re-pick
- Hide infra models from the chat picker
- Detect vision support on cached GGUF repos
- Honor saved maxSeqLength on auto load
- Restore default chat template for local GGUFs
- Warn on save failure and revert config on cancel
- Refetch picker inventory on open
- Persist read only per model config safely
* Fix stale model auto load
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix model picker numeric input sizing and constraints
Size value inputs to their content so long context lengths are not clipped,
restrict them to numeric characters, and stop the speculative decoding label
from truncating in the sidebar.
* Fix picker CI tests and harden chat template resolution for PR #6647
- tests: point the descender guard at the moved model-selector.tsx path
- tests: exclude the disabled Reload model button from the regenerate locator so .first targets the real Regenerate
- picker/service.py: reject symlinked template/gguf leaves that resolve outside the browse allowlist (HF cache reads unchanged)
- compare mode: resolve each pane's own remembered chat template instead of inheriting the other pane's from the store
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Protect future-schema per-model configs from deletion for PR #6647
savePerModelConfig already refuses to overwrite a stored config whose schema version is newer than this client understands, but deletePerModelConfig did not. Unchecking Remember on an older client therefore silently destroyed a newer client's saved config. Apply the same guard on delete and surface the blocked case through the existing saveFailed toast.
* Protect future-schema per-model configs from quota eviction for PR #6647
The save and delete guards already refuse to touch a stored config whose schema version is newer than this client understands, but the quota-eviction path did not, so a full store on an older client could still evict a newer client's config. Skip future-schema entries when evicting and fail the save if the budget cannot be met without them.
* Fix GGUF context persistence, compare context, and rollback settings for PR #6647
Persist a GGUF context override from the user's intent instead of collapsing it against the loaded context, which reintroduced the context-reset (f4838782cb reverted the native-baseline fix). model-config-page now collapses the saved value against native, and use-chat-model-runtime and chat-adapter retain the requested context on load so re-saving another setting keeps the override; a null request stays null so a VRAM auto-fit never becomes a stored override.
shared-composer: a compare pane with no explicit GGUF context now loads at native (0) like single-view, not the session maxSeqLength that silently shrank the shown context.
use-chat-model-runtime: restore the previous model's KV cache dtype and chat template on a failed-load rollback so it runs as it was, not with backend defaults.
* Preserve native path token when reloading the active model for PR #6647
handleReloadActiveModel rebuilt the selection without the store's activeNativePathToken, so reloading a file-picked GGUF after a settings change validated the display label as a repo/path and failed. Thread the active native token through the reload selection so native-loaded models reopen correctly.
* Make picker template validation resilient and accept HF generation tags for PR #6647
Import Jinja lazily inside validate_chat_template so a backend without the optional jinja2 package (GGUF-only installs) still starts instead of raising ModuleNotFoundError at import time. Register a no-op extension for the Transformers {% generation %} assistant-mask tag so pasting a valid HF chat template validates, matching the renderer, rather than being rejected as an unknown tag.
* Honor remembered compare config and parse processor chat_template.json for PR #6647
* Fix failed-load rollback context and processor template map fallback for PR #6647
* Restore speculative decoding config on failed-switch rollback
When a model switch fails after the previous model was unloaded, the
rollback reload restored tensor_parallel, KV cache dtype and the chat
template override, but omitted speculative_type and spec_draft_n_max and
cleared their loaded shadows to null. The previous model therefore came
back running at backend defaults (speculation off) while the UI still
showed it enabled, and the status resync confirmed the off state. Resend
the previous model's speculative settings in the rollback load and keep
the store's active and loaded speculative fields in sync with them.
* Reset max sequence length when a model has no saved config
applyPerModelConfigToRuntime reset every per-model field except
maxSeqLength, which it only wrote when the incoming config had one.
maxSeqLength is the sole field carried on store.params, so selecting a
model with no remembered config left the previous model's value in place
and later loaded the new model at that leaked length. Fall back to the
standing default so an unremembered model loads at its own default.
* Surface a message when a variant update cannot start
startManagedUpdate handled the conflict and error start outcomes but let
busy fall through as if the update began, so the confirm dialog closed
with no job created and the cached variant stayed stale. Show an info
message when the repo is busy with a sibling transfer so the click is
not silently dropped.
* Keep per-model speculative choices out of the global default
A staged load with a per-model or one-off config sets keepSpeculative,
which already skips reading the global speculative preference. The
matching save still ran unconditionally, so the model-specific choice was
written to the global unsloth_chat_speculative_type and a later model with
no saved config started from it instead of Auto. Skip saveSpeculativeType
when keepSpeculative so the per-model choice stays isolated.
* Seed non-active model settings from the app default max length
The Run settings page captured initialMaxSeqLength from the loaded
model's runtime params and fell back to it for a model with no saved
config. Opening settings for a different, unloaded model and clicking
Load then sent the active model's context (for example 64k) instead of
the 4096 default, risking validation failures or OOMs. Seed the default
for non-active models and keep the runtime value only for the active one.
* Prefer sidecar tokenizer chat template over the GGUF copy for variants
_chat_template_from_dir returned the embedded GGUF template first when a
variant was selected, reversing the tokenizer-first precedence of the
no-variant path. A model whose chat_template.jinja or tokenizer_config.json
supersedes a stale embedded template then got the wrong template on
variant selection. Keep tokenizer files first regardless of variant; the
variant only picks which GGUF is the fallback. Adds regression tests for
both the tokenizer-wins and gguf-fallback cases.
* Keep per-model speculative choices load-local in autoload and compare
The interactive load path treats a per-model speculative choice as
load-local and skips writing it to the global default. Autoload and
generalized compare still called saveSpeculativeType unconditionally, so a
remembered off or ngram setting leaked into unsloth_chat_speculative_type
and later models with no saved config inherited it. Persist the global
preference only when the value came from the global settings.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: record the compare pane's loaded context in runtime state so the active model's settings and any reload or save use it, not the previous context
* Studio: notify the user when a Hub autoload can't start because another download for the model is already running, instead of silently dropping it
* Studio: drop the merge's orphaned staged-model store helpers and unused alert imports
The main merge left isPendingGguf and pendingSelectionMatches referencing the
removed PendingModelSelection type, and the alert-dialog/alert imports unused
after the permission-mode dropdown replaced the bypass dialog, so tsc -b failed.
* Studio: cache a null default chat template so the viewer stops re-fetching it
A model with no sidecar or embedded template resolves to a terminal null, but
that result was never cached, so reopening the template viewer re-ran the
backend and Hugging Face lookup every time.
* Studio: detect direct-file GGUFs in run settings so Max Tokens uses their context
A GGUF loaded from a local file or custom folder has no variant label, so the
run-settings panel treated it as non-GGUF and clamped Max Tokens to the session
max_seq_length instead of the loaded GGUF context. Detect it via the reported
GGUF context and the .gguf checkpoint suffix, matching the chat page.
* Studio: prompt to re-select a local model file when its lease expired before reload
A file-picked GGUF is reachable only through a native path token that the
desktop host prunes after a TTL. Reloading reused that token blindly, so a
reload long after the initial load failed with an opaque error. Track the
token's expiry and, when it has passed, ask the user to re-select the file
instead of attempting a doomed reload.
* Fix descender-clipping test to tolerate sidebar layout utilities
The sidebar account-block div carries layout utilities (min-w-0, flex-1)
between 'flex' and 'flex-col', so the descender-clipping guard's regex,
which required 'flex' immediately followed by 'flex-col', no longer matched
and the test failed to locate the account-block div. Generalize the prefix
to allow intervening flex utilities while still capturing the leading-*
class before the collapsible visibility utility and asserting leading-tight,
so the guard against clipped glyph descenders is fully preserved.
* Harden picker chat-template resolution
Enforce the 64 KiB chat-template contract at the validate endpoint's request
model so a direct caller cannot submit a template far larger than the frontend
allows (MaxBodyMiddleware only bounds the whole request body, not this field);
oversized templates now return a clean 422.
Apply sidecar-over-GGUF template precedence globally across cached snapshots
instead of per snapshot. A repo with multiple cached revisions previously
returned the first snapshot's template, so a newer GGUF-only revision could
win over an older revision's maintained chat_template.jinja sidecar, which
contradicted the documented intent that sidecars supersede the embedded copy.
* Guard per-model config against future-schema and lossy migration
Two forward-compatibility gaps in the versioned per-model config store:
- The load/apply path returned and normalized a stored record without checking
its schema version, so a record written by a newer client was reinterpreted
under the current schema and applied to a live model load, even though save,
delete and eviction all refuse to touch future-schema records. Reject
future-schema records on load too.
- The one-time legacy migration enforced the storage budget without protecting
the entries it had just migrated and set the completion flag unconditionally.
When storage was already full of future-schema records (which are unevictable
by an older client), the migrated entries were the only evictable ones and
could be dropped while migration was still marked complete. Protect the
migrated keys during eviction and only mark migration complete when they
survive, so it retries once space frees up.
* Discard chat-template validation results after the dialog closes
Server-side template validation is async, but closing or cancelling the editor
did not abort it, so a late-arriving valid response still called onSave and
applied a template the user had already dismissed. Track a validation token
that is bumped on close and ignore any validation result whose token is stale.
* Record native lease expiry when loading a picked GGUF from the chip
The pending-native-model chip loaded via stageOrLoad directly, bypassing
loadNativeModelIntent, so activeNativePathExpiresAtMs was never recorded for a
chip-loaded file. A later reload then either skipped the lease-expiry guard
entirely (expiry left null) or compared against a previously loaded file's
stale expiry, so reload could reuse an already-pruned token or wrongly block a
still-valid one. Route the chip through loadNativeModelIntent, which builds the
same selection and records the expiry.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Prefer sidecar template for a directly selected local GGUF file
A direct .gguf file path read its embedded chat template without checking the
parent directory for a maintained sidecar (chat_template.jinja /
tokenizer_config.json), while directory and variant selections already prefer
the sidecar. That let the config editor preview or save a stale embedded
template for the same model depending on how it was selected. Check the parent
directory sidecars first, then fall back to the embedded copy, and cover both
paths with tests.
* Resolve cached chat template per revision, newest first
The earlier change searched every cached snapshot for a sidecar before
considering any snapshot's embedded GGUF template, which let an obsolete sidecar
from an older revision override the newest revision's template. Restore
per-snapshot resolution (newest first): a revision's sidecar still supersedes
its own embedded GGUF copy, but a newer revision is no longer overridden by an
older revision's sidecar.
* Preserve autoload transport conflicts and surface background busy downloads
- When a Hub autoload hits a transport conflict, keep pendingHubAutoLoad bound
instead of clearing it. Clearing it re-keyed the download surface and its
cleanup cancelled the conflict the toast tells the user to resolve, so the
Hub resume affordance was gone the moment it appeared. Return early on
conflict, mirroring the started branch, so resolving it from the Hub still
auto-loads on completion.
- The background-download branch handled started and conflict but silently
dropped a busy outcome, leaving the user with no feedback when a peer variant
of the same repo was already downloading. Surface the same busy toast the
autoload path uses.
* Fix context length, GGUF template, fetch state and lease expiry bugs
Keep explicit context length values instead of collapsing to null at
native. The collapse made the slider jump back at the native maximum
and made Reload load the previous context instead of the chosen one.
Prefer the first split when resolving a GGUF without a variant. Later
splits carry no chat template metadata, so picking the largest file
could return no template for a sharded model.
Clear stale fetch state when template and metadata lookups retry, so
a previous terminal error is not shown while a new fetch is running.
Record native path lease expiry together with the token when a load
commits. The expiry was written by only one load path and even when
the load did not start, so a reload could be blocked with an expired
file message for a still valid token.
* fix(model-picker): resolve review findings across config, inventory, and templates
- Apply remembered per-model config in the training-compare chat handoff so a
prior model's customContextLength no longer leaks into the next load
- Match GGUF variant labels with the inventory extractor too, so cached
no-quant-token files resolve their default chat template
- Show "Auto" instead of a fabricated 32768 when native context is unknown
- Reuse the identical staged auto-load object on same-pick so a re-pick during
download pre-flight no longer disarms auto-load via "busy"
- Union supports_vision when deduping cross-cache inventory rows
- Serve hidden-model needles from a new GET /api/hub/hidden-models endpoint and
merge them client-side, covering runtime-configured RAG embedders
- Clamp GET chat templates to MAX_CHAT_TEMPLATE_BYTES (route + jinja sidecar),
matching the validate endpoint's contract
- Lower-clamp stored customContextLength to shared CONTEXT_LENGTH_MIN
- Wipe unsloth_chat_load_on_selection in Settings "Reset all"
- Drop stale pendingHasContext comment describing deleted staging machinery
* Fix stale defaults cache, token in query string and rounded up context ceiling
Refresh cached chat template and max position data when a model update
completes. Send the HF token for model config requests in the dedicated
header instead of the URL. Snap the native sequence length ceiling down
to the nearest step so the slider cannot exceed the declared maximum.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix compare pane reverting active checkpoint on non-GGUF load
Re-read runtime params after setCheckpoint so the fresh checkpoint is
kept instead of being overwritten by the pre-setCheckpoint snapshot.
* Send the HF token via header for the vision and embedding checks
checkVisionModel and checkEmbeddingModel still passed the HuggingFace
token as a ?hf_token= query parameter, so it landed in server access
logs, proxy logs, and browser history. Move them to the
X-Unsloth-HF-Token header like getModelConfig already does, and accept
the header on the check-vision and check-embedding routes with the
existing query parameter kept as a fallback for older clients.
* Cap the chat template on the model load path
The load endpoint accepted an unbounded chat_template_override, so a
direct caller could hand llama.cpp an arbitrarily large Jinja template
even though the frontend, the validate endpoint, and the read paths all
enforce the 64 KiB limit. Reuse MAX_CHAT_TEMPLATE_BYTES in the
LoadRequest validator, rejecting oversized templates with a fast
character-count check before the exact UTF-8 byte check.
* Protect existing per-model configs during legacy migration
When the one-time legacy import pushes the store over budget, eviction
now protects the entries the user already has and drops only the
just-migrated legacy entries, so importing old load settings can never
discard a newer per-model config.
* Reset clears the context override instead of pinning the native value
Reset wrote the discovered native context into customContextLength for
GGUF models, but isDefaultConfig treats any non-null customContextLength
as an explicit pin, so Reset with Remember enabled persisted a fixed
context and future loads stopped using the native auto context. Reset
now restores the full default (customContextLength null); the native
value is still shown through the existing display fallback.
* Bound chat-template sidecar reads to a size limit
The chat_template.json, tokenizer_config.json, and Hub-downloaded sidecar
readers decoded and json-parsed the whole file before the extracted
template hit the 64 KiB response cap, so an oversized metadata file could
exhaust memory. Read them through a bounded reader (4 MiB envelope) that
returns None when the file is larger, matching the existing chat_template.jinja
size guard. Adds tests for oversized tokenizer_config.json and chat_template.json.
* Keep the native-path token and lease expiry in sync
Rollback after a failed reload restored the previous token but left the
failed load's expiry in the store, so a later reload could be falsely
blocked as expired (token A paired with load B's lease). Restore the
previous lease alongside the token, and clear the expiry wherever the
token is cleared on a non-GGUF transition, so the two never diverge.
* Clear the native file lease on compare-pane loads
* Studio: add regression tests for the model-picker per-model-config
Guard the specific regressions that reverted the predecessor change:
- backend pytest (studio/backend/tests/test_model_picker_regression.py):
infra-model hiding, HF token via header with query fallback, and the
chat-template byte caps.
- source contracts (tests/studio/test_model_picker_contracts.py): the token
stays out of the URL, the context ceiling is floored, the native lease is
cleared on compare-load and restored on rollback, the default caches key on
the inventory version, and the hidden needles stay present.
- Playwright E2E (tests/studio/playwright_model_config.py) wired into
studio-ui-smoke.yml on port 18898: Context Length persists across a reload,
Reset clears the stored override, and infra models are absent from the picker.
- optional GPU-gated inference smoke (tests/studio/test_gpu_inference_smoke.py)
that auto-skips on GPU-less CI and stays short on a GPU.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: model pinning, row menus, hub inference settings, and inventory filters
Pinning
- Add a pinned models store (localStorage) with repo and per-quant pins
- Pinned section in the model selector's On Device list and the hub inventory,
with newest pins first so Pin to top lands on top
- Deleting a repo drops its pins
Row menus
- Replace loose row icons with a shared 3-dots menu (pin, reveal in file
manager, copy identifier, copy path, delete) on picker rows, hub quant rows,
the hub run bar, and on-device inventory rows
- Menus only render for models actually on disk; platform-aware reveal labels
- Backend: cached-model-path and reveal-cached-model endpoints resolving
managed HF-cache repos only
Hub inference settings
- Gear in the GGUF run bar opens an Inference settings dialog reusing the chat
page's controls: model config (context length, KV cache, speculative
decoding, chat template), system prompt, reasoning, sampling, tools and
retrieval
Inventory
- Model-type filter (text, vision, embedding, STT, TTS, diffusion) beside the
sort pill, both with a sort icon, capped widths and truncation so the
On device heading never wraps
- Unsloth-owned repos without an upstream provider logo fall back to the
Unsloth mascot avatar
- Discover / On Device tabs widened; hub search bar narrowed to match
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: revert the Unsloth mascot avatar fallback
Unsloth-owned repos without an upstream provider match go back to the
colored-initial tile, and unslothai is no longer a relabeled owner.
* Studio: run-bar options on single models, and aligned type/capability filters
- Give single-model (non-GGUF) run bars the same 3-dots options menu and
settings gear as GGUF, at repo level
- Drop Pin to top from the run-bar menus; pinning stays in the On Device list
- Add an Image to text (diffusion) capability with detection, and surface it
in both the hub Discover capability filter and the On Device type filter
- Align the On Device type filter with the Discover capability options and
share the same detection so both dropdowns match
* Studio: apply hub inference config on reload, eject action, and run-bar polish
- Fix inference settings not applying: the hub dialog now writes the config to
the runtime before reload, matching the chat page (selectModel reads runtime
state, not the selection)
- Order the settings gear before the 3-dots menu in the run bars
- Replace the loaded-model run-bar action (New Chat) with Eject, wired through
the inspector to the hub's ejectModel
- Truncate the results heading so a long search query clips instead of
overlapping the header pills in split view
- Use a plain magnifying-glass icon for the no-results empty state
* Studio: fix GPU settings loss, load guards, pins, filters, and cached paths
Reloading a model from the chat sidebar or the hub gear dialog rebuilt the
per-model config without the GPU memory fields, so manual GPU layers, MoE
placement, and the GPU pick were reset on every reload and could be saved
over a remembered config. The active config now comes from a shared
useActiveModelConfig hook that carries the GPU fields for GGUF models, and
the sidebar remount signature tracks them through a shared gpuFieldsSignature
helper.
The in-flight load guard lived in a ref inside each useChatModelRuntime
instance, so the chat page, hub page, and gear dialog could not see each
other's loads. A load started from the gear dialog left the hub page free to
eject the model mid-reload or start a second concurrent load. The runtime
store now records the loading pick, selectModel checks it across instances,
and ejectModel refuses with a toast while any load is in flight.
The cached-model-path endpoint matched GGUF files by basename and excluded
only mmproj, so Copy path and Reveal could return an MTP drafter for a quant
and returned 404 for directory layouts like BF16/model-00001.gguf. Variant
files are now resolved from snapshot-relative paths with the same drafter,
mmproj, and big-endian exclusions as the load path, shared through a new
_main_variant_gguf_label helper.
Hub and picker fixes:
- rename the diffusion capability label from "Image to text" to
"Image generation", since it detects image generators
- validate pinned quants through the cached variant listing, keep the last
verified set while revalidating, and drop deleted quants immediately
- pass a measured scroll margin to the on-device virtual list so rows past
the overscan stay visible below the pinned block
- keep the delete menu for stopped partial safetensors downloads
- give the inventory type filter a reset in Clear filters, a truthful empty
state with a Show all types action, and hide it on the datasets view
- order picker pinned rows by pin recency, include pinned matches in the
empty-state check, and sync pins across browser tabs
- count only the visible rows in the On device list header
Tests: contract checks for each fix in test_model_picker_contracts.py and a
backend test for the variant label selection.
* Studio: reveal cached models in Windows Explorer under WSL
The reveal endpoint only branched on macOS, Windows, and generic Linux.
Under WSL the Linux branch spawned xdg-open, which is missing on a stock
distro without a Linux desktop, so the request failed with a 500 and the
UI showed a failed to open file manager error.
WSL is now detected with the existing helper and the path is converted
with wslpath before opening explorer.exe, selecting the file the same
way native Windows does. Directories open directly. When interop is
unavailable the old xdg-open fallback still runs. The macOS, native
Windows, and native Linux branches are unchanged, and the Tauri app is
covered since its hub reveal calls this same local endpoint.
Tests: platform guards for the WSL reveal, the interop fallback, and
the unchanged native Linux behavior in tests/studio/test_reveal_file_manager.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Adjust model picker row spacing and cogwheel hover consistency
* Studio: exact hidden model ids and newest revision cached paths
A custom RAG embedder repo was published to the frontend as a basename
substring needle, so a generic name like org/model could hide unrelated
models in the pickers. The hidden-models endpoint now sends full repo ids
that are matched exactly.
Copy path and Reveal picked a GGUF variant from an arbitrary cache
revision when the same file existed in more than one. The newest revision
now wins, matching the whole repo lookup.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix model picker GPU config, metadata, and cache selection
Load each compare model with its saved GPU memory mode, GPU layers, CPU MoE layers, and selected GPU IDs. Reconcile saved GPU IDs with the current hardware. Include the active native GGUF path token in metadata checks. Search all Hugging Face cache roots when resolving cached models and select the largest visible cache entry. Remove obsolete barrel exports and the staging-only GPU memory helper.
* Studio: hide hub inference settings gear for now
The cogwheel in the hub download cards is out of scope for this PR. The
dialog component stays in place and a TODO marks where the button
returns in a future PR.
* Refresh hidden model matchers
* Fix GGUF detection, compare context pin, and picker delete staleness
Treat any pick with a GGUF variant as GGUF in selectModel so the first
load after downloading an uncached quant validates and sizes with the
right GPU settings instead of unloading the current model on a wrong
preflight. Variant picks now also set isGguf on their selection meta.
Stop compare panes from inheriting the active model's context pin when
their own saved config says Auto. Null context in a remembered config
now means no pin, matching how the pane settings are shown.
Route picker deletes through the hub inventory client, which
invalidates the HF cache scan and the variants cache. The legacy
delete route left the scan cache warm, so deleted models reappeared
in the picker until the TTL expired. Removed the now unused legacy
delete client and updated the contract test to match.
* Studio: fix stale GGUF load-marker ordering test
The load-in-flight marker still precedes the hub-download guard and the
unload, but the llama_extra_args inheritance that used to sit between the
marker and the guard now runs ahead of the GGUF branch, so it is no
longer a landmark inside the sliced source. Drop it from the ordering
assertion and keep the marker -> guard -> unload invariant.
* Studio: fix per-model config edge cases in compare loads and saved defaults
- chat-settings-sheet: gate the MTP fallback note and context/VRAM warning on
the broader isGguf (variant, loaded gguf context, or .gguf suffix) instead of
isLoadedGguf, so direct-file and custom-folder GGUF loads still surface
those diagnostics.
- shared-composer: a compare pane's context now comes from its own config only
(a saved pin, else null for Auto/native). It no longer inherits the active
model's shared snapshot, which resolveFitMaxSeqLength treated as an explicit
pin and could load a pane at another model's context (VRAM/OOM), matching the
single-model load path.
- model-config-page: when an auto-fit GGUF is saved with fixed GPU layers
(Manual) and Remember, pin the displayed fitted context so a later fresh load
keeps the placement instead of sending native/0 and recreating the OOM.
- per-model-config: treat Auto GPU memory mode and Auto/default speculative type
as follow-global defaults; do not persist them as per-model overrides so later
global preference changes keep applying.
* Studio: gate vision capability on GGUF projectors and bound remote template downloads
- cache_inventory: only mark a cached repo vision-capable when it holds an actual
GGUF mmproj projector, not any file whose name merely contains "mmproj" (e.g.
mmproj_config.json), matching the runtime's GGUF-only projector detection.
- picker/service: pre-check the remote file size before downloading an uncached
repo's chat template / tokenizer config, so a maliciously large sidecar is
skipped instead of fetched and retained in full, mirroring the size gate the
local-file path already applies.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add source-contract guards for the per-model-config edge-case fixes
Guard the four per-model-config fixes against silent regression in CI:
- local GGUF diagnostics gate on the broad isGguf, not the variant-only isLoadedGguf
- fixed-layer GGUF saves pin the displayed context
- Auto GPU mode and Auto/default speculative are not persisted as per-model overrides
- a compare pane's context comes from its own config, not the active model's snapshot
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: clear manual GPU knobs on Default and resolve local embedders before repo-id
- model-config-page: switching GPU Memory back to Default now clears the Manual-only
knobs (gpuLayers/nCpuMoe/selectedGpuIds); otherwise a remembered config kept stale
pins that a later load re-applied when the global GPU preference was Manual, despite
the page showing Default.
- routes/models hidden_model_matchers: resolve an existing local path before the repo-id
regex, mirroring is_hidden_model, so a local embedder shaped like "models/embedder" is
hidden by exact path instead of leaking as a chat model.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add _is_mtp_drafter to the model_config stub in the export-paths test
routes/models.py imports _is_mtp_drafter from utils.models.model_config at module
load, but the lightweight stub in test_export_absolute_paths.py did not provide it,
so loading the module under the stub raised ImportError on Backend CI. Add the stub.
* Studio: read a picked GGUF's chat template through the native path lease
The picker chat-template GET has no native-path-lease plumbing, so a
desktop-picked (drag-drop) GGUF could not show its default chat template
in Run Settings until the model was loaded: the endpoint only receives
the display label, not the leased file path.
Read the embedded template through the existing lease-aware
/api/inference/validate probe instead. A new include_chat_template flag
resolves the granted canonical path and returns the GGUF's own embedded
template, never a sibling sidecar (the grant authorizes just that one
file); it skips the training guard like include_context_length and is
bounded by MAX_CHAT_TEMPLATE_BYTES. The frontend fetch mints a one-shot
validate-model lease when a native token is present and keeps the plain
GET path for HF and allowlisted local models.
Adds backend and source-contract regression tests.
* Studio: call worker.direct_wheel_url in the ROCm wheel-url test
The ROCm Mamba/SSM test referenced worker.py's private _direct_wheel_url,
but the worker imports the wheel helper under its public name
direct_wheel_url (utils.wheel_utils). When the worker module loads (its
imports resolve in CI), worker_mod._direct_wheel_url raised AttributeError;
the test only masked it by skipping when the worker could not be imported.
Call the name that actually exists so the assertion runs; it still returns
None for an empty cuda_major (ROCm).
* Studio: reset max sequence length to the app default, not the loaded value
For a non-GGUF active model, the per-model config seeds maxSeqLength from
the loaded runtime value so the panel opens showing the running context.
Reset set config.maxSeqLength to null, but the null fallback resolved back
to that captured runtime value, so the field kept showing the old custom
length and the config saved/reloaded it again. A remembered or active
max-length override therefore could not be cleared from Run settings.
Fall the null/default case back to the app default (clamped to the model's
native ceiling) instead of the active runtime snapshot, so Reset actually
clears the override. The initial view is unaffected: an active model's
config.maxSeqLength is already non-null, so it still shows the loaded value.
Adds a source-contract regression guard.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: persist default max length, refresh deleted quants, hide non-chat locals
Three follow-up fixes from review of the per-model-config picker:
- Max sequence length: the persisted per-model record now keeps config's
maxSeqLength (null after Reset) so isDefaultConfig can clear a remembered
override; the resolved app-default is substituted only into the load
request, never the saved record. Previously Reset saved the concrete
default and left the model pinned/remembered.
- GGUF variant expander: deleting a downloaded quant from a repo that still
has other cached quants now bumps the expander refresh key, so the removed
quant stops showing as downloaded and clickable (which would try to reload
the deleted file) until the repo is collapsed and reopened.
- Local picker rows: require capabilities.canChat before listing a local
models-folder / LM Studio row. A weightless folder (only config.json) is
classified non-chat, and toLocalModelInfo drops capabilities, so selecting
such a row would try to load a path the inventory already marked non-chat.
Adds source-contract regression guards for all three.
* Fix compare-pane and Reset context defaults in model picker
Two related per-model-config default regressions:
- A non-GGUF compare pane with no saved maxSeqLength fell back to the
active model's shared runtime snapshot, so comparing a saved 128K model
against an unconfigured pane loaded the latter at 128K and could OOM. It
now falls back to the shared app default (DEFAULT_MAX_SEQ_LENGTH), the
same fallback the single-model config path uses.
- contextAtDefault treated an explicit customContextLength equal to the
native ceiling as a default, which wedged the Reset button disabled for
a deliberate pin-to-native. It now counts as default only when there is
no override at all.
DEFAULT_MAX_SEQ_LENGTH becomes a single exported constant in
per-model-config.ts so the single-model config and the compare path share
one source of truth. Adds source-contract guards for both fixes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Skip over-cap remote Jinja templates so the tokenizer template wins
The remote chat-template resolver bounded raw chat_template.jinja downloads
only by MAX_TEMPLATE_METADATA_BYTES (4 MiB), then returned the first
non-empty Jinja unconditionally. The picker route drops any template larger
than MAX_CHAT_TEMPLATE_BYTES (64 KiB), so an uncached repo whose
chat_template.jinja sits between 64 KiB and 4 MiB returned no template at
all, even when a valid smaller tokenizer_config.json template existed. The
local path already skips oversized .jinja files and falls through.
Gate the extracted Jinja on MAX_CHAT_TEMPLATE_BYTES and continue searching
when it exceeds the cap, matching _chat_template_from_jinja_file. The 4 MiB
download bound stays for JSON files that merely embed a small template. Adds
a regression test that a big Jinja plus a valid tokenizer config resolves to
the tokenizer template.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard legacy per-model-config migration idempotency
The v1->v2 localStorage migration (unsloth_load_settings ->
unsloth_model_configs) runs on every store read, so it must migrate exactly
once and never re-run, duplicate, or clobber a newer per-model config on a
reload or restart. That was covered only by a manual proof, so add durable
guards:
- Source-contract test pinning the three idempotency layers (the in-memory
legacyMigrationChecked guard, the persistent unsloth_model_configs_migrated
flag set in every terminal branch, and the non-overwriting Object.hasOwn
merge-skip) plus the readMap invocation. Reddens if any layer is dropped.
- Playwright model-config E2E: promote the legacy-migration step to a gating
check (soft_fail, which gates under the CI STUDIO_UI_STRICT=1) that the
migrated value is preserved and the flag is set, then reload again with a
fresh legacy seed present and assert the stored key set is unchanged, so a
second reload cannot re-migrate, duplicate, or clobber.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Note the migration E2E now gates idempotency under STUDIO_UI_STRICT
* Tighten model-picker per-model-config code comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: shimmyshimmer <shimmyshimmer@users.noreply.github.com>
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
* Studio: GPU memory dropdown — llama.cpp --fit on and manual gpu-layers/cpu-moe
* Studio: simplify GPU memory changes (reuse ParamSlider, GPU_LAYERS_ALL, loadedGpuMemoryFields helper)
* Studio: GPU picker — choose which GPUs a GGUF model loads on (gpu_ids)
* Studio: simplify GPU picker (share /api/system fetch, validate gpu_ids)
* Studio: GPU picker review fixes (gate relative indices, no cross-model leak, validate, types)
* Studio: group GPU controls under a collapsible GPU section
* Studio: GPU feature review fixes (fix fit-ctx test, behavior-test the floor, comment accuracy)
* Studio: make GPU a top-level settings section (not nested under Model)
* Studio: flatten GPU controls into the Model section, group by GPU/context/generation
* Studio: move GPU Memory to the bottom of Model with its dependent controls beneath it
* Studio: move GPU Memory below Tensor Parallelism and GPUs below GPU Memory
* Studio: tighten GPU Memory and GPU Layers tooltip copy
* Studio: fix fit-mode context slider track-click, restore GPU Memory tooltip, shorten fit dropdown label
* Studio: GPU Memory tooltip one mode per line, briefer
* Studio: note HIP_VISIBLE_DEVICES (ROCm) in the GPUs picker tooltip
* Studio: narrow the GPU Memory dropdown to fit the shortened label
* Studio: use 'llama.cpp --fit' in the GPU Memory tooltip for consistency
* Studio: allow Tensor Parallelism in Manual GPU mode
* Studio: graduated MoE-on-CPU offload (--n-cpu-moe) replacing the all-or-nothing toggle
* Studio: size the MoE-offload slider for staged (deferred-load) models
* Studio: share one GGUF header walk for the context-length and MoE-count readers
* Studio: size the GPU Layers slider for staged models (one staged-header read)
* Studio: move Tensor Parallelism below the GPUs picker
* Studio: GPU split (--tensor-split) per-GPU model share in Manual mode
* Studio: tolerate whitespace in GPU split input, move it below GPU Layers
* Studio: rename the GPU split control to "Split ratio"
* Studio: Split ratio sends explicit even input; fix blank=free-VRAM (not even) copy
* Studio: tighten llama.cpp --fit VRAM margin with --fit-target 512
* Studio: GPU memory review fixes (rollback re-baseline, single-GPU TP gate, accurate copy)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: move Split ratio below MoE Layers on CPU
* Studio: address PR review (fix GPU-info hydration race, share fit context-length across load paths)
* Studio: address codex review (manual single-GPU TP guard, GPU-aware spec defaults in fit/manual, GGUF-only context/preference)
* Studio: address codex review round 2 (gpu_present seed, single-GPU tensor-split guard, staged manual-knob reset, strip inherited offload flags)
* Studio: address codex review round 3 (strip inherited --n-cpu-moe, CPU-fallback warning in Manual mode)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address codex review round 4 (preserve pinned fit context across a later Apply)
* Studio: address codex review round 5 (honor GPU picker for diffusion GGUFs, clear fit pin on cross-model switch)
* Studio: preserve the pending GPU Memory mode when staging a model
* Studio: pin diffusion GPU device order and reset GPU-memory state for diffusion loads
* Studio: address codex review round 6 (fit-Auto rollback context, preserve manual non-tensor split modes, persist GPU mode on load not select)
* Studio: persist the applied GPU Memory mode, not the requested one (skip diffusion loads)
* Studio: replace Manual-mode split-ratio field with per-GPU layer sliders
* Studio: clarify per-GPU layer split hint for tensor-parallel mode
* Studio: address codex review round 7 (allow GGUF gpu_ids past the legacy guard, replay GPU-memory fields on respawn)
* Studio: address codex review round 8 (size the validate preflight like the load in fit mode, across both load paths)
* Studio: skip the training-OOM guard for llama.cpp --fit GGUF loads (they spill to RAM)
* Studio: drop the now-redundant compare-path validate sizing (the --fit guard skip makes it moot)
* Studio: address codex review round 9 (keep the training guard for fit loads, forward gpu_ids to validate, strip inherited manual tensor-split)
* Studio: address codex review round 10 (gate GPU-memory adoption on is_gguf, record manual knobs only in Manual mode)
* Studio: handle diffusion GGUFs symmetrically in the GPU Memory controls (preserve the standing mode preference, hide the inapplicable mode/TP controls)
* Studio: remember the GPU Memory settings per model
* Studio: consolidate --fit mode and Manual mode into a single Manual mode
* Studio: preserve the per-GPU layer split across GPU Layers changes
* Studio: trim overly long GPU Memory comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* address GPU memory config review comments
* trim redundant GPU memory tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reconcile manual-mode TP drops with the #6659 drop-site invariants
* Preserve quantized KV in manual --fit, charge GGUF companions in full, reconcile GPU pick on load
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Clear stale GPU baseline on non-GGUF loads so it can't read as dirty
* Fix no-context-shift test for the conditional -c flag
* Credit manual GPU-layer offload for cached HF GGUFs
* Reset per-model load knobs on GGUF quant switch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Strip inherited tensor-split when manual ratio is cleared
* Match auto-load validation to safetensors placement
* Reset editable manual knobs after Auto GGUF loads
* Record a single device for diffusion GPU picks
* Reset per-model GPU knobs before applying saved settings
* Address review comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard manual tensor splits and keep remembered context on auto-load
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Snapshot compare knobs, seed splits from free VRAM, flag zero-offload loads
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Exempt CPU-only loads from the guard floor and harden compare and reseed paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reach full offload from the layers slider and charge extras drafters in the guard
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Warm the GPU device cache before pick reconciles and disable staged GPU controls
* Align the training guard with inherited extras, spec mode, and compare targets
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Hide GPUs from companion-less zero-offload loads
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Size diffusion picks per device, own manual offload flags, reject XPU picks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop tensor flags at zero layers and exempt CPU-pinned drafters
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Allowlist the zero-layer tensor parallel drop site
* Keep validate and load guards on the same extras and refresh stale baselines
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop mismatched manual tensor splits before launch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate XPU picks on the real backend field and harden split and hydration paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Weight full GPUs as zero, clamp split shares, and refine the zero-layer mask gate
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Carry fit context across mode changes and align drafter and picker gates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Catch variant switches, uncached diffusion repos, and text-only mmproj skips
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Check companions on the first device and size native and remote zero-layer loads
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Replace the training guard's precise VRAM modeling with a conservative bound
* Baseline context pins on non-GGUF hydration and reprobe list-seeded staged GGUFs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Size manual splits by their largest share and preserve resolved context from Default
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Default-deny unsized required companions and price KV at the effective cache dtype
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reserve MTP draft KV and MLA target-copy in the training guard
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Size tensor-parallel loads per device and show GPU controls for native GGUFs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reserve MTP overhead for uncached remote GGUFs and the mmproj runtime factor
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop the training-coexistence VRAM estimation this PR added
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate remembered load settings to GGUF picks
* Lock the remaining load-time controls during a staged load
* Clear the stale native-path token on compare loads
* Drop a stale guard reference from the zero-offload masking comment
* Seed GPU baselines from the rollback response and drop never-emitted offload flags
* Match validate's training guard to load and keep the native reload token
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim verbose GPU-memory comments
* Thread the variants header walk off the event loop, honor device pins on zero-offload, and hold staged GPU edits
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor manual placement and classify pinned zero-offload loads
* Close diffusion admission and status hydration gaps
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Check the actual diffusion GPU during training
* Align staged baselines and manual reload dedupe
* Fix GGUF placement and rollback state
* Harden manual GGUF placement boundaries
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove unused resolve_tensor_parallel import in llama_cpp.py
The name is used only in llama_server_args.py, routes/inference.py, and tests,
not in llama_cpp.py; the unused hoisted import trips the import-hoist verifier
in the source-lint CI job.
* Fix diffusion GPU dedup and training guard for non-numeric device tokens
The diffusion runner drives only its single lowest device and the backend
records that one device (self._gpu_ids = [sorted(gpu_ids)[0]]), but the reload
dedupe compared it against the full requested list, so a multi-GPU pick that
resolves to the same device forced a needless reload. Normalize the request the
same way for a loaded diffusion model in both _already_in_target_state and the
route _request_matches_loaded_settings.
The chat-during-training coexistence guard called int() on the single-device
token and hard-rejected when it could not parse. A non-numeric token (a CUDA
UUID / MIG handle) now sizes against the whole visible pool like the GGUF guard
instead of falsely blocking the load, and an empty token (a CPU-only runner such
as a CPU diffusion GGUF) is allowed outright since it uses no GPU VRAM.
* Tighten comments added by the GPU memory config changes
* Harden GGUF placement from independent review: VRAM sizing, diffusion TP reset, tensor_split validation
- Training coexistence guard: a single-device runner pinned through an
unresolvable UUID/MIG token was sized against the aggregate visible-VRAM pool,
so a load could pass on capacity it cannot use and then OOM active training.
Size against the worst-case visible device (min free) instead, keeping the
guard's documented default-deny contract. The empty-token (CPU-only runner)
allow path is unchanged.
- Diffusion startup: _start_diffusion_server now resets self._tensor_parallel to
False alongside the other placement resets. A prior tensor-parallel chat load
(process killed but not fully unload-reset) otherwise left /status misreporting
tensor parallelism and made an identical diffusion re-Apply reload against the
stale state.
- tensor_split: reject negative / non-finite / all-zero splits up front. They
were dropped at launch but still compared raw in the reload dedupe, so an
identical Apply reloaded indefinitely.
- Tests: the shared httpx stub was incomplete and, installed via setdefault
before real httpx loaded, broke a combined pytest run (collection errors on
httpx.Response). Import the real installed httpx instead.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <unslothshared@gmail.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Replace standalone Studio wording with Unsloth
Replace the single word Studio with Unsloth wherever it is used as
shorthand for Unsloth Studio in docs, CLI output, UI strings, i18n
locales, workflow display names, comments and docstrings.
Kept unchanged: the full name Unsloth Studio, third party product
names (LM Studio, Visual Studio, Mac Studio), feature names
(Recipe Studio, Fine-tuning Studio and its translations), and all
identifiers such as env vars, commands, paths and filenames.
* Address review feedback on the Studio wording rename
Use "an" before Unsloth where the rename left the article as "a".
Restore the split brand where Unsloth and Studio render as two halves
of the full product name: the onboarding sidebar subtitle and the
IPv6 localhost warning. Scope two messages to the full name Unsloth
Studio where plain Unsloth was misleading: the AMD README bullet and
the CLI studio setup error.
* Studio: permission levels for chat tool calls (Ask, Approve for me, Off, Full access)
Replace the Bypass permissions on/off toggle with a four level permission
selector, available in Settings > General (new Permissions section above
Notifications), the chat settings panel, the composer plus menu, and a new
always visible composer pill.
Levels:
- Ask for approval: every local tool call pauses for allow/deny.
- Approve for me: only calls detected as potentially unsafe pause; the
python/terminal sandbox stays on.
- Off: never pauses; sandbox stays on (previous default behavior).
- Full access: never pauses and the sandbox is disabled. Still requires
the danger confirmation and is never restored across reloads.
Backend adds permission_mode to the OpenAI compatible and Anthropic
passthrough payloads and threads it through both tool loops. Auto mode
uses a fail closed classifier in tools.py: terminal commands must be on
a read only allowlist with no redirection or substitution, python code
is AST scanned for writes, exec, process and network use, MCP tools
auto run only with read only style names. Unknown tools always ask.
Legacy bypass_permissions and confirm_tool_calls keep their exact
behavior for existing API callers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio permissions: Off is a plain toggle below Full access
Off moves to the bottom of the level menu with a short description and
acts as the feature-off state: the composer pill is hidden entirely
while Off, and reselecting the active level toggles back to Off.
* Studio permissions: higher contrast composer pill text
The permission pill uses a foreground based grey instead of the shared
muted pill color, so it reads darker in light mode and lighter in dark
mode. Full access keeps the danger yellow.
* Studio permissions: panel dropdown layout and shorter tooltip
Chat settings panel: the Bypass permissions label sits on one line with
a full width dropdown underneath, styled like the other panel selects.
Tooltip shortened and wording uses Unsloth instead of Studio.
* Studio permissions: harden auto-mode unsafe detection
Extend the Approve for me classifier to catch write and exec paths that
slipped through:
- terminal: sort -o, tree -o, xxd -r, find -exec/-execdir/-ok/-delete
and find -fprint/-fprintf/-fls now ask; plain read-only forms still
auto-run. awk is no longer allowlisted since its program can write and
call system().
- python: from-imports of mutating names (from os import remove [as rm])
and star imports now ask.
Found by a fuzz and edge-case simulation matrix; pinned in
test_permission_mode.py.
* Studio permissions: split multi-line terminal commands in auto detection
A shell runs each line as its own command, but shlex reads newlines as
whitespace, so "ls\nrm -rf x" demoted rm to argument position and
auto-ran. Normalize newlines and CR to separators, and treat any all
separator token as a command boundary so runs of blank lines still
split. Found by the simulation matrix; pinned in tests.
* Studio permissions: address review feedback on auto-mode detection
Auto-mode (Approve for me) safety classifier hardening:
- Python: flag any reference to a mutating attribute, not only direct
calls, so indirect refs (f = os.remove; f(x)) and aliases ask. Detect
Path.open(mode) write modes and wrap the AST walk to fail closed.
- Terminal: match attached short output flags (sort -o/tmp/out) and keep
find context across grouping parens so find ( -delete ) asks.
- Both: ask before reads that escape the sandbox workdir via parent
traversal or hit credential paths (.ssh, .aws, id_rsa, .pem, etc.).
permission_mode plumbing:
- Fold permission_mode=full into bypass_permissions at the request model
so route-level confirm-gate guards see it as bypass.
- Reject ask/auto on the Anthropic Messages server-tools path, which has
no confirmation channel (mirrors the confirm_tool_calls rejection).
- Keep forced RAG autoinject in auto mode: the safe search_knowledge_base
retrieval never gates, so derive the skip from the real confirm need.
- Reset all local preferences now also clears the legacy confirm key so a
reset restores the fresh default instead of the old level.
Regression tests added for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio permissions: close auto-mode classifier gaps from review round 2
Auto mode ("Approve for me") let a few mutating calls through as safe:
- os.open(...) always creates/writes a descriptor, so treat it as unsafe
even though builtin open in read mode stays safe.
- fd -x/--exec/-X/--exec-batch runs a command per match; scan for these
alongside find's -exec/-delete.
- tempfile writes artefacts and hands back writable handles, so importing
it now asks.
- Calling the result of a call (getattr(os, "remove")("x"), partials) is a
dynamic target the AST can't vet, so fail closed.
- An MCP tool whose name pairs a read verb with a mutating one
(get_or_create_issue, read_and_delete_file) no longer auto-runs on the
read prefix alone.
Also fold permission_mode="off" into confirm_tool_calls=False on both
request models so the non-stream route guard sees the disabled gate, and
drive the Confirm tool calls toggle off permission_mode="ask" so auto no
longer shows it on.
* Harden auto-mode classifier and normalize bypass to full for PR #7079
Approve for me now asks for a few cases it previously auto-ran:
- os.open via an os alias (import os as o; o.open(path, O_CREAT))
- pathlib symlink_to / hardlink_to / link_to
- importlib.import_module dynamic imports
- os.mkfifo / os.mknod / os.utime
Also fold bypass_permissions into full when a stale ask/auto permission_mode
is sent alongside it, so the Anthropic route guard no longer 400s those legacy
callers. Adds classifier and request-model regression tests.
* Close more auto-mode classifier gaps for PR #7079
Approve for me now asks for cases the review surfaced:
- builtin open aliased to a name (f = open; from builtins import open as w)
or looked up dynamically (globals()['open'])
- pickle / marshal / shelve / dill deserialization
- io.FileIO write handles
- sort --compress-program (runs an external program)
- MCP names carrying save/archive/submit/commit/push/sync/register verbs
Also refine the attribute open() write check so an explicit read mode
(ZipFile.open(name, "r")) stays auto while os.open flags still ask. Adds
test coverage for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close three more auto-mode gaps for PR #7079
- rg runs an arbitrary program per file via --pre / --hostname-bin, so
"Approve for me" now asks for those flags (rg is on the read-only
allowlist).
- A path-qualified command token (./ls, /tmp/cat) is an arbitrary
executable, not the trusted utility its basename matches, so it asks
before running.
- A direct /chat/completions caller that sets permission_mode ask/auto
but omits the legacy confirm_tool_calls flag now self-enables the
confirmation gate, so tools can no longer run ungated on that path.
Adds classifier and request-model tests for each case.
* Close auto-mode classifier gaps from review round 3 for PR #7079
Approve for me now asks for cases the latest pass surfaced:
- short-option clusters bundling a write flag (sort -uo out => -u -o)
- procfs reads that leak a process env/args/memory
(cat /proc/self/environ, /proc/PID/cmdline, maps)
- env-assignment prefixes that change command lookup/loading
(LD_PRELOAD=x ls, PATH=. ls, IFS=x ls); benign FOO=1 cmd stays auto
- os.open imported as a bare callable (from os import open as o)
Also drops ps from the safe terminal allowlist: its BSD environment
flags (ps auxe, ps eww) dump a parent process's unscrubbed env and
cannot be flag-parsed reliably, so ps always asks now. Adds classifier
tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 4 for PR #7079
Terminal (Approve for me now asks for these):
- cd dropped from the safe allowlist: cd /; cat etc/passwd moves the
shell out of the session workdir so a later relative read escapes it
- env -C/--chdir (workdir escape) and -S/--split-string (builds a fresh
command line); wrapper flags are now checked
- /etc//passwd and /etc/./passwd normalize to /etc/passwd before the
sensitive-path scan
- a sensitive path split across an assignment and an argument
(p=/etc; cat $p/passwd) via best-effort NAME=value expansion
Python:
- builtins.exec / builtins.eval attribute calls (dynamic code execution)
- destructured open aliases (f, _ = (open, print); f('out', 'w'))
- a sensitive path composed from literals (os.path.join('/etc','passwd'),
'/etc' + '/passwd')
- ZipFile/TarFile write modes (ZipFile(name, 'w')); the reader stays auto
Adds classifier tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 5 for PR #7079
Terminal (Approve for me now asks for these):
- procfs reads hidden by shell quotes (cat /proc/$PPID/enviro''n) or
quoted/nested-variable assignments (p="/proc/$PPID"; cat $p/environ):
quotes are stripped and NAME=value prefixes expanded before the scan
- LESSOPEN/LESSCLOSE, which make less run an input preprocessor command
Python:
- os.chdir / os.fchdir, which move the cwd so a later relative read
escapes the sandbox workdir
- sensitive paths composed via a pathlib / chain (Path('/etc') / 'passwd')
or an f-string of literals (f'/proc/{pid}/environ')
- runpy (import) and runpy.run_path / run_module, which run arbitrary code
Adds classifier tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 6 for PR #7079
Approve for me now asks for these:
- a mutating callable reached through a getattr alias
(rm = getattr(os, "remove"); rm("f")): calls through a getattr-bound
name fail closed
- compound MCP tool names carrying clone/checkout/comment/fork/tag/
invite/share, which start with a read verb but still mutate
- a sensitive path hidden behind a glob (cat /e??/passwd,
cat /e[t]c/passwd): a ? / * / [..] token is matched against the
sensitive-file set and bracket classes are de-obfuscated; benign
globs (ls *.py) stay auto
Also run first-pass RAG retrieval in off mode: like auto, off never
prompts, so a direct caller passing a stale confirm flag should not lose
document retrieval (both tool loops).
Adds classifier tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 7 for PR #7079
Approve for me now asks for these:
- __builtins__.exec / __builtins__.eval (dynamic code via the dunder)
- terminal reads that hide a credential path behind a backslash escape
(cat /et\c/passwd)
- read-named MCP filesystem calls pointed at a credential path
(mcp__fs__read_file {"path": "/etc/passwd"})
- compound MCP names carrying append / prepend
- open aliased through a subscript or builtins attribute
(f = globals()["open"]; f = builtins.open) then called to write
- open(..., **{"mode": "w"}) where a kwargs splat hides the write mode
- a sensitive path with a dynamic segment (open(f"/etc/{name}"),
os.path.join("/etc", name)); /tmp/{name} stays auto
- urllib3 networking
Also stop folding permission_mode ask/auto into confirm_tool_calls for
external-provider requests: that branch rejects confirm_tool_calls with
tools, and the mode only governs local tool calls. Local requests still
self-gate. Adds tests for each case.
* Close auto-mode classifier gaps from review round 8 for PR #7079
Approve for me now asks for these:
- dbm on the unsafe-module list: dbm.open(file, "c"/"n") creates files,
and importing the family signals a persistence writer
- reads of ~/.azure and ~/.config/gh credential stores (Azure/GitHub
tokens), in terminal, MCP arguments, and Python literals
- compound MCP names carrying upsert / assign
Adds classifier tests for each case.
* Gate secret mounts and fix the composer pill count for PR #7079
- Add Docker/Kubernetes secret mount dirs (/run/secrets,
/var/run/secrets) to the sensitive-path checks, so Approve for me asks
before reading injected credentials (terminal, MCP args, Python).
- Count the always-visible permission pill in the composer's compact
threshold so labels collapse at the intended width instead of
overflowing by one pill.
Adds classifier tests for the secret mount paths.
* Close auto-mode classifier gaps from review round 10 for PR #7079
Approve for me now asks for these:
- qualified pathlib constructors (pathlib.Path('/etc') / name), folded
the same as bare Path(...), so a dynamic sensitive path is detected
- open aliased through an annotated assignment (f: object = open;
f('out', 'w')), tracked like a plain assignment
- recursive searches rooted at an absolute path (grep -R TOKEN /home,
rg TOKEN /, fd pattern /etc), which read host files outside the
sandbox tree; sandbox-relative searches stay auto
Adds classifier tests for each case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 11 for PR #7079
Approve for me now asks for these terminal reads, which bash would
expand into a sensitive path only after the classifier had approved:
- a glob that resolves into a secret mount or credential dir
(cat /r?n/secrets/hf_token, cat /root/.s??/id_rsa)
- a recursive search rooted at a tilde home (grep -R TOKEN ~root,
grep -R TOKEN ~/logs)
- a brace expansion that builds a credential path (cat /etc/pass{w,}d)
- a default/alternate parameter expansion that builds one
(cat /etc/pass${x:-wd})
- an input redirection that hides a glob (cat </e??/passwd)
And these python calls:
- a str.format-built sensitive path (open('/etc/{}'.format('passwd')))
- writer methods that persist to disk without open() (numpy.save,
Image.save, plt.savefig, DataFrame.to_csv, json.dump)
Segment-wise directory matching keeps benign globs (ls /home/*/projects)
auto. Adds regression tests for each case and its safe counterpart.
* Close auto-mode classifier gaps from review round 12 for PR #7079
Approve for me now asks for these too:
- a terminal read whose parent traversal hides behind a redirection with
no following space (cat <../../notes)
- a python read whose path is built with str.join
(open(''.join(['/etc', '/passwd']))), told apart from os.path.join
- a dynamic-code builtin reached through an alias
(from builtins import eval as e; e(...); x = builtins.exec; x(...))
Adds regression tests for each case and its safe counterpart.
* Close auto-mode classifier gaps from review round 13 for PR #7079
Approve for me now asks for these too:
- a recursive search whose root is hidden behind an assignment
(p=/; grep -R TOKEN $p): the recursive-root test now runs on the
assignment-expanded tokens as well
- a python read whose sensitive path is split through a literal variable
(base = '/etc'; open(base + '/passwd')), including via an f-string
- numpy ndarray.tofile, which persists without open()
- a sequence brace read (cat /etc/pass{w..w}d), expanded alongside the
comma brace form before the sensitive-path scan
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 14 for PR #7079
Approve for me now asks for these python reads that assemble a sensitive
path in a form the fold did not yet recognize:
- a pathlib object reused through a name (p = Path('/etc'); p / 'passwd')
- old-style percent formatting ('%s/%s' % ('/etc', 'passwd'))
- Path.joinpath ('/etc'.joinpath('passwd'))
- a bytes path literal (open(b'/etc/passwd'))
And these terminal reads, which bash expands into a sensitive path only
after the classifier had approved:
- a substring parameter expansion off an assignment
(p=passwd; cat /etc/${p:0:6})
- an ANSI-C quoted path (cat $'/etc/pass\x77d')
- a glob into an Azure or GitHub CLI config dir
(cat /home/*/.az?re/..., cat /home/*/.config/g?/...)
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 15 for PR #7079
Approve for me now asks for these terminal reads, which bash expands into
a sensitive path only after the classifier had approved:
- a per-thread procfs env alias (cat /proc/$PPID/task/$PPID/environ)
- a recursive root behind a default parameter (grep -R TOKEN ${root:-/home})
- a path built by pattern replacement (p=passXd; cat /etc/${p/X/w})
And these python reads:
- a pathlib .parent/.parents chain that escapes the session workdir
((Path.cwd().parent / 'other' / 'notes').read_text())
- a sensitive path resolved through glob (glob.glob('/e??/passwd')[0])
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 16 for PR #7079
Approve for me now asks for these terminal reads, which bash expands into
a sensitive path only after the classifier had approved:
- a case-modifying parameter expansion (p=PASSWD; cat /etc/${p,,})
- a mutating find action hidden behind an assignment (f=-delete; find . $f)
- a glob assembled through an assignment (g=e??; cat /$g/passwd)
- a POSIX bracket class glob (cat /etc/pass[[:lower:]]d)
And these python reads/writes:
- a glob pattern folded from a literal variable
(base='/e??'; glob.glob(base + '/passwd'))
- a directly imported os.path.join (from os.path import join; join('/etc', 'passwd'))
- a directly imported writer (from numpy import save; save(...))
- an aliased pathlib constructor (from pathlib import Path as P; P('/etc') / 'passwd')
The find/fd and glob scans now run on the assignment/parameter-expanded
command, and pathlib/join/writer import aliases are tracked. Adds
regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode gaps from review round 17 for PR #7079
Two fixes:
- Gate sqlite3 in auto mode. sqlite3.connect(path) creates or mutates a
database file (and runs DDL/DML) with no open()/writer attribute for
the AST checks to catch, so treat the module like dbm and ask.
- Only self-enable confirm_tool_calls for Studio's own tool loop. The
ask/auto fold previously set confirm on every non-provider request,
including a plain client-tool passthrough (client-supplied tools that
Studio does not execute), which then tripped the local-tool
streaming-confirm route guard and rejected the passthrough. Restrict
the fold to requests that actually ask Studio to run tools
(enable_tools / enabled_tools / mcp_enabled).
Adds regression tests for the sqlite3 write and for the passthrough vs
tool-loop confirm behavior.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode gaps from review round 18 for PR #7079
Classifier (auto mode asks for these):
- os.open through a module alias (import os as o; o.open(...)); os/posix
aliases are tracked like the literal module name.
- less/more pagers, whose escapes (+cmd, !shell, -o/--log-file, LESSOPEN)
can run a command or write a file the command-name allowlist cannot
see, so they are no longer auto-approved.
- a read-named MCP tool carrying a mutating query
(query_database {"query": "DELETE FROM runs"}); DML/DDL statements are
matched as whole statements so a natural-language query that merely
contains "delete" stays safe.
- ML persistence helpers (save_pretrained / save_file / save_model /
save_weights / save_lora / save_checkpoint) that export weights to disk.
Route:
- Honor CLI-forced tools when deriving the confirm gate. When a process
policy (unsloth run --enable-tools) opens the local tool loop without a
request-level tool signal, a permission_mode ask/auto request now
derives confirm at the route (GGUF and safetensors paths) so the mode
still gates the call, and a non-streaming ask/auto request is rejected
rather than running unprompted. A plain client-tool passthrough (no
local loop) is unaffected.
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 19 for PR #7079
Approve for me now asks for these too:
- a terminal read whose path is built by indirect parameter expansion
(x=passwd; p=x; cat /etc/${!p})
- a bash /dev/tcp or /dev/udp redirection, which opens a network socket
(cat </dev/tcp/host/port)
- a python read via pathlib's receiver-plus-pattern glob
(Path('/etc').glob('passw?'))
- a python read whose sensitive root passes through a normalizer
(os.path.abspath('/etc'), Path('/etc').resolve())
- a pickle-backed loader that can execute code on load
(torch.load, joblib.load, pandas.read_pickle), tracked through module
import aliases
- compiled code wrapped into a callable (compile(...) + types.FunctionType)
Adds regression tests for each case and its safe counterpart.
* Honor unset permission_mode as ask across the local tool loop for PR #7079
Three gaps where an omitted permission_mode did not behave as the
documented default ("ask"):
- The frontend only sent permission_mode / confirm_tool_calls /
bypass_permissions when a tool pill was on. A process policy
(unsloth run --enable-tools) can open the tool loop with no pill, so
the backend never saw the selected gate. Send the three permission
fields at the top level of every local chat payload instead.
- The backend read payload.confirm_tool_calls directly at the
pre-switch guard and both late per-backend derivations, so an unset
mode fell through as no-gate even for an explicit ask/auto. Add
_permission_mode_confirm(payload): explicit confirm_tool_calls wins,
explicit ask/auto engage the gate, off/full never prompt, and an
unset mode defaults to ask only where realizable (streaming), keeping
the legacy no-gate run for non-streaming unset requests.
- A forced ask/auto tool loop (CLI --enable-tools) with no stream now
400s at the pre-switch guard before evicting the resident model,
matching the existing confirm-without-stream rejection.
Adds test_permission_mode_confirm_derivation covering the derivation
truth table.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Declare permission_mode and bypass_permissions on the local chat request type
The previous change moved permission_mode, confirm_tool_calls and
bypass_permissions to the top level of the local chat payload. They had
lived inside a conditional spread, which is not subject to excess
property checking, so the fields were never declared on
OpenAIChatCompletionsRequest. At the top level tsc flagged
permission_mode as unknown (TS2322), failing the frontend build and
every job whose Studio install builds the frontend.
Add permission_mode and bypass_permissions to the request interface
(confirm_tool_calls was already present).
* Close auto-mode classifier gaps from review round 21 for PR #7079
Auto mode ("Approve for me") now asks for these too:
- a pathlib read built from a concrete constructor (PosixPath, WindowsPath
and their Pure* forms), which the folder previously ignored so
PosixPath('/etc') / 'passwd' lost its /etc root and ran unprompted
- a terminal or python read of the ssh host keys under /etc/ssh, which
the sensitive-path regex only covered for passwd/shadow/sudoers
- a read whose path variable is reassigned: the whole-tree pre-scan kept
the last binding, so base = '/etc'; open(base + '/passwd'); base = 'data'
folded to data/passwd and ran even though execution reads /etc/passwd;
any multiply-bound name now folds to the escape sentinel and asks
Also stop the pre-switch guard from rejecting a plain client-tool
passthrough. permission_mode only implies the confirm gate for Studio's
own local tool loop (enable_tools / enabled_tools / mcp_enabled); a
non-streaming client-tool passthrough that carries permission_mode
ask/auto (confirm_tool_calls left unset by the validator) must forward to
the provider branch. Only an explicit confirm_tool_calls=True still forces
the local-confirm rejection there.
Adds regression tests for each case and its safe counterpart.
* Fix permission-pill compaction count and Full-access confirm sync for PR #7079
Two frontend consistency issues in the permission-level UI:
- The composer collapses tool pills to icons above four, but the count
left out the permission pill, which renders in every mode except off.
With one optional pill also shown the row reached five pills without
collapsing and could overflow. Count the pill when it is visible
(permission_mode != off).
- Entering Full access via setPermissionMode('full') or
setBypassPermissions(true) left confirmToolCalls at its previous value,
so a Full-access run (which sends confirm_tool_calls=false) could still
report confirmations as enabled in response metadata. Set
confirmToolCalls false at both entry points.
* Close auto-mode classifier gaps from review round 23 for PR #7079
Auto mode ("Approve for me") now asks for these too:
- a command using an abbreviated GNU long option that reaches a
write/exec action (sort --out= for --output, env --ch= for --chdir,
fd --base-dir= for --base-directory); a prefix of an unsafe long flag
now fails closed
- printf -v NAME, which assigns to a shell variable, so
printf -v PATH %s .; ls can rewrite PATH and run ./ls unprompted
- fd --base-directory / --search-path, which move the search root
outside the session workdir without any positional slash token
- an MCP tool whose compound read name carries a copy-style mutator
(read_and_copy_file, get_and_snapshot_volume): copy, duplicate,
import, export, download, backup, restore, snapshot, mirror
Also treat an omitted permission_mode as its documented default ("ask")
on the Anthropic Messages server-tool path. That branch has no
confirmation channel and already rejects explicit ask/auto, so an
omitted mode now falls into the same rejection instead of silently
running server tools unprompted, unless the caller opted out with
confirm_tool_calls=false (the legacy equivalent of "off"). off/full and
that opt-out still run; the two routing tests that relied on the old
implicit run now set permission_mode="off".
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Refine permission gating from review round 24 for PR #7079
Four fixes from the latest review:
- Anthropic Messages server tools: an omitted permission_mode no longer
rejects a request that only runs safe server tools (web_search), so
existing Anthropic callers keep working. It still rejects an omitted
mode when a local tool (terminal/python) is selected, and an explicit
ask/auto is still rejected outright. off/full and a
confirm_tool_calls=false opt-out always run.
- Pre-switch confirm-without-stream guard: use
_explicit_studio_tool_loop_requested (the same predicate the
passthrough router uses) instead of the policy-inclusive
_effective_enable_tools, so a process --enable-tools policy no longer
turns a client-tool passthrough into a local-loop rejection.
- Auto mode now asks for `uniq INPUT OUTPUT`: uniq writes its second
file positional, so a second positional (numeric flag values skipped)
is treated like `sort -o`. A lone `uniq file` or piped `... | uniq`
stays safe.
- MCP mutation check now strips SQL comments before matching, so
DELETE/**/FROM and UPDATE/**/users (comment-as-whitespace) no longer
slip past the DML/DDL denylist.
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode gaps from review round 25 for PR #7079
Auto mode ("Approve for me") now asks for these Python cases too:
- a bare archive constructor with a write mode (from zipfile import
ZipFile; ZipFile('out.zip', 'w')), tracked through import aliases like
the zipfile.ZipFile attribute call already was
- a dynamic lookup aliased through getattr (g = getattr;
rm = g(os, 'remove'); rm('file')), not just direct getattr(...) calls
- a callable that wraps open or a writer via functools.partial
(w = partial(open, mode='w'); w('out.txt')), which hides the write mode
Also:
- Always-safe tools (render_html) stream their early provisional canvas
card in auto mode again. The provisional-card guard mirrored the raw
confirm flag, which suppressed the early card under Approve-for-me; it
now reuses the auto-mode safety decision (is_always_safe_tool).
- The assistant-ui composer no longer counts the permission pill toward
its collapse threshold when the level is Off (the pill renders null
there), matching the other composer.
Adds regression tests for each case and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Align permission-mode confirm guards with the router (review round 26)
Three pre-switch confirm-gate checks disagreed with how the tool
loop actually enters, so a valid request could 400 (or an invalid
one could evict the resident model) at the wrong point:
- The /chat/completions pre-switch guard only looked at explicit
request fields, so a process --enable-tools policy that forces the
loop on (request omits enable_tools, no client tools) slipped past
it and only 400ed after _maybe_auto_switch_model had swapped the
model. It now mirrors the router's own loop-entry gate
(_effective_enable_tools or mcp, tool_choice="none" disabling it
unless explicitly asked) while still deferring to client-tool
passthrough, so the policy-forced case is caught before the switch.
- The ChatCompletionRequest full/off fold treated enabled_tools by
itself as a local-loop request and set confirm_tool_calls=True.
The router never starts the loop on enabled_tools alone (it only
filters which tools run), so a non-streaming passthrough carrying
client tools plus enabled_tools 400ed instead of routing verbatim.
The fold now keys off the same enable_tools / mcp_enabled signals.
- The Anthropic /v1/messages unsupported-mode rejection (ask/auto,
or an omitted mode selecting terminal/python) ran inside the
post-switch server-tools block, so an invalid request evicted the
resident model before the 400. It now runs before the auto-switch,
determined from the requested server tools, like the neighboring
malformed- and mixed-tool guards.
Adds regressions for each: a policy-forced non-streaming ask/auto
guard rejection that never reaches the switch, an enabled_tools-only
passthrough that keeps confirm unset, and an Anthropic rejection that
precedes _maybe_auto_switch_model.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close auto-mode classifier gaps from review round 27 for PR #7079
Auto mode ("Approve for me") now asks for these host-mutating or
host-reading cases it previously ran unprompted (the sandbox does not
jail filesystem reads, and terminal commands can change host state):
- Destructured string literals fold into the scanned path now, so
base, leaf = ('/etc', 'passwd'); open(base + '/' + leaf).read()
resolves to /etc/passwd and asks, like the single-assignment form
already did. The tuple/list unpacking branch tracked only aliases to
open; it now also binds literal and folded-path elements.
- pathlib name rewrites fold to the rewritten path:
Path('/etc/x').with_name('passwd').read_text() (and with_stem /
with_suffix) spell no literal /etc/passwd but resolve to it, so they
are folded and caught. Benign in-sandbox rewrites stay safe.
- hostname NAME (or -F/--file, -b/--boot) sets the hostname, so a
positional or a set flag asks; bare hostname and the display flags
(-f/-i/-I/...) stay read-only.
- date -s/--set STRING and the bare MMDDhhmm... positional set the
system clock and now ask; the display forms stay read-only (+FORMAT,
-u/-R, and -d/-r/-f whose following value is skipped so date -d
tomorrow is not mistaken for a clock-setting positional).
Adds regression rows for each gap and its safe counterpart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close more auto-mode classifier gaps from review round 28 for PR #7079
Auto mode ("Approve for me") now asks for these cases too:
- Mapping-style %-formatted paths. '/etc/%(f)s' % {'f': 'passwd'} folds
to /etc/passwd and asks; a dynamic value or a non-literal mapping
leaves the NUL marker so /etc/<dynamic> still fails closed. The path
folder previously handled only tuple/scalar % right-hand sides and
returned None for a dict, hiding the sensitive segment.
- A read-named MCP database tool carrying PostgreSQL COPY. COPY ... FROM
bulk-loads a table and COPY ... TO writes a server-side file, so both
are matched as mutating queries like DELETE/UPDATE already were. A
'copy' substring in a column name stays safe (word boundary).
- logging file handlers. logging.FileHandler('out.log', mode='w') (and
the default append mode, RotatingFileHandler/TimedRotatingFileHandler/
WatchedFileHandler, and the bare from-import form) create or truncate
a file like open(..., 'w'), so they are classified as writer calls.
StreamHandler / NullHandler and logging reads stay safe.
Adds regression rows for each gap and its safe counterpart.
* Fix writer aliases, GraphQL mutations, and auto server tools (review round 29)
- Auto-mode Python: an aliased writer or archive constructor is tracked
like the existing open alias, so from numpy import save; s = save;
s('out.npy', arr) (and z = ZipFile; z('a.zip', 'w'), incl. the
destructured forms) ask instead of running the write unprompted. A
benign builtin alias (x = len) stays safe.
- Auto-mode MCP: a read-named tool carrying a GraphQL mutation now asks.
query_graphql {"query": "mutation { deleteIssue(id: 1) }"} matches a
leading mutation keyword (GraphQL uses # comments, so it scans the raw
payload); GraphQL read queries stay safe.
- Anthropic /v1/messages: permission_mode "auto" no longer 400s a
safe-only server-tool selection. auto only needs a confirmation
channel for an unsafe call, so like the omitted default it runs for
web_search / RAG / render and rejects only when a gate-needing local
terminal/python tool is selected. ask still always rejects (it asks
per call, which this passthrough cannot honor). The rejection stays
ahead of the model auto-switch.
Adds regression rows/cases for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate asyncio spawn, net clients, default-captured open; allow safe-only auto (round 30)
Auto-mode Python now asks for more process/network/write vectors:
- asyncio process spawners (asyncio.create_subprocess_exec/shell and a
loop's subprocess_exec/shell) run an arbitrary program without the
terminal blocklist, so they gate like os.system/subprocess.
- stdlib network clients imaplib / poplib / nntplib / xmlrpc(.client) /
webbrowser open outbound connections the sandbox does not namespace
off, so their import asks like the other network modules.
- a callable captured as a function or lambda parameter default
(def f(o=open): o('out', 'w')) now binds that parameter into the same
alias set, so the later write through it is gated. A benign default
(o=len) stays safe.
Also, permission_mode "auto" no longer 400s a non-streaming local tool
request whose selection is always-safe-only (web_search / RAG / render).
auto only prompts for a classifier-flagged call, so a safe-only auto
request needs no stream, while ask, an explicit confirm_tool_calls=true,
MCP, and an unrestricted or unsafe selection still require it. Applied
via a shared _confirm_gate_needs_stream helper at the pre-switch, GGUF,
and safetensors confirm-stream guards; the loop's per-call confirm flag
is unchanged.
Adds regression rows/cases for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Catch brace-glob paths and attribute writer aliases; unfold auto (round 31)
- Terminal auto mode now runs the glob-sensitive scan over every
expansion candidate, so a brace-expanded glob (cat /e{t,}c/pass?d,
which bash expands to /etc/pass?d and then globs to /etc/passwd) asks.
Brace expansion alone spells no literal /etc/passwd and the glob only
resolves once the brace group is expanded, so scanning both together
is required. A benign brace + glob stays safe.
- Python auto mode now tracks a mutating attribute captured as a plain
name: s = np.save; s('out.npy', arr) binds a writer alias, a captured
.open bound method (p = Path('out').open; p('w')) fails closed on any
call since its mode position varies, and z = zipfile.ZipFile is gated
like the bare import. A benign attribute alias (x = np.mean) stays safe.
- permission_mode "auto" is no longer folded to confirm_tool_calls=true
on the request model. Folding it defeated the safe-only-selection
exception in _confirm_gate_needs_stream (an explicit confirm forces
stream=true), so a non-streaming safe-only auto request was rejected.
Leaving it unset lets the route apply the exception; the mode still
drives the loop's per-call gate. "ask" still folds (it gates every
call).
Adds regression rows/cases for each.
* Harden SQL/GraphQL/writer classification and passthrough guards (round 32)
MCP argument mutation detection (read-named query tools):
- CREATE DDL now matches modifiers and the broader object set, so
CREATE OR REPLACE VIEW, CREATE UNIQUE INDEX, CREATE TEMP TABLE,
CREATE MATERIALIZED VIEW and CREATE FUNCTION ask.
- Stored-procedure invocation (CALL proc(...), EXEC/EXECUTE) and VACUUM
ask; a natural-language "call me back" stays safe via the trailing
"(" / ";" / end lookahead.
- GraphQL # comments are stripped before the mutation match, so
mutation # note\n { deleteIssue(id: 1) } no longer hides the mutation.
Python auto-mode classification:
- numpy.memmap / open_memmap and pandas ExcelWriter / HDFStore create or
truncate a file on construction, so they gate like open(..., "w").
- asyncio networking (asyncio.open_connection, loop.create_connection /
create_server and unix variants) opens outbound connections/listeners
the sandbox does not isolate, so it gates like socket.connect.
Terminal auto-mode: file -C / --compile writes a compiled magic database.
Routing:
- A JSON-schema response_format is guided-decoding passthrough, not a
local tool loop, so a --enable-tools policy no longer 400s a
non-streaming ask/auto structured-output request at the confirm guard.
- An explicit confirm_tool_calls=False opts out of the Anthropic Messages
server-tool gate entirely (it wins over the mode, mirroring
_permission_mode_confirm and the GGUF path), so it runs even under ask.
Adds regression rows/cases for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Track path-ctor aliases, exempt empty selection and safe safetensors card (round 33)
- Python auto mode now propagates path constructor / join aliases, so
assigning Path or os.path.join to another local name is still folded:
P = Path; (P('/etc') / 'passwd').read_text() and j = os.path.join;
open(j('/etc', 'passwd')) ask, while a benign /tmp alias stays safe.
- _confirm_gate_needs_stream now distinguishes an omitted enabled_tools
(None, all tools) from an explicit empty list ([], no tools). An empty
selection runs no built-in tool and cannot prompt, so a non-streaming
auto request with enable_tools=true, enabled_tools=[] is no longer
400ed under a --enable-tools policy.
- The safetensors provisional render_html card now uses permission_mode:
render_html is always safe and never prompts, so its early canvas card
streams under auto (which ships confirm_tool_calls=true) instead of
being suppressed, matching the GGUF path's is_always_safe_tool exemption.
Adds regression rows/cases for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Extend auto-mode classifier: SQLite mutations, more net/xattr/compressed writers
Additional fail-closed gaps found by a fresh adversarial pass, each with a
reproduction and a benign control:
- MCP read-named tools now ask on SQLite-flavored writes the base DML/DDL regex
missed: ATTACH / DETACH DATABASE, a write-form PRAGMA (PRAGMA journal_mode=WAL
/ user_version=42 / foreign_keys(0), while the read-form PRAGMA journal_mode
stays safe), and load_extension() which loads and runs an arbitrary shared
library.
- Python auto mode now gates the remaining asyncio network entry points
(start_server, open_unix_connection, loop.create_datagram_endpoint,
sock_connect), os.setxattr / os.removexattr metadata writes, the gzip / bz2 /
lzma single-stream writers (GzipFile / BZ2File / LZMAFile, mode-gated like
ZipFile so a read stays safe), pandas to_xml, and the websockets client.
Benign controls (SELECT 1, read-form PRAGMA, asyncio.sleep, gzip read, numpy
read, natural-language "attach"/"analyze") stay safe. Regression rows added to
test_permission_mode.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close follow-up auto-mode gaps: SQLite/GraphQL variants, more writers and net
A fresh adversarial pass on the previous round found consistent extensions of
the same fail-closed rules, each reproduced with a benign control:
- MCP read-named tools: DROP / ALTER now cover the same broad object set as
CREATE (DROP FUNCTION, ALTER INDEX, DROP MATERIALIZED VIEW); ATTACH is caught
without the optional DATABASE keyword via its quoted-path form; a
schema-qualified write PRAGMA (PRAGMA main.user_version=1) is matched; and a
GraphQL mutation carrying directives (mutation M @audit { ... }) is treated as
a mutation.
- Python auto mode: os.startfile (Windows program launch), asyncio
start_unix_server, and the socketserver framework now ask; a gzip/bz2/lzma
open imported under an alias (from gzip import open as gopen) is gated like
builtin open; and a dynamic path prefix that can form a sensitive absolute
root (open(chr(47) + "etc/passwd"), open(os.sep + "etc/passwd")) is treated as
sensitive, while a dynamic prefix with a benign suffix stays safe.
Benign controls (read-form PRAGMA, natural-language "attach ... as", "drop the
idea", SELECT dropped_at, query @cached, gzip read alias, dynamic prefix +
data/file suffix) stay safe. Regression rows added to test_permission_mode.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate GNU time -o, basicConfig/methodcaller/fileinput, and more SQL mutations
Another adversarial pass surfaced further consistent fail-closed gaps, each
reproduced with a benign control:
- Terminal: GNU time -o/--output/-a/--append truncate or append to a file with
timing output; time is a wrapper, so the flag is checked before the wrapped
command like env -C.
- Python auto mode: logging.basicConfig(filename=...) opens a log file for
write; operator.methodcaller("write_text"/...) hides a writer method behind a
string and is now treated as dynamic dispatch (like getattr/partial);
fileinput.input(..., inplace=True) rewrites a file in place (the default read
form stays safe).
- MCP read-named tools: UPDATE now matches quoted, bracketed, and
schema-qualified targets (UPDATE "users" / public.users / ONLY public.users /
[users] / `users` SET); SELECT ... INTO OUTFILE/DUMPFILE writes a server file;
and state-changing SQL functions inside a SELECT (pg_terminate_backend,
setval, pg_write_file, lo_export, ...) ask.
Benign controls (time ls / time -p, basicConfig(level=), methodcaller("upper"),
fileinput read, NL "update ... set", setval_col column, PL/pgSQL SELECT INTO
var) stay safe. Regression rows added to test_permission_mode.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten auto-mode classifier comments
Collapse the multi-line rationale blocks in the permission classifier to one or
two lines each without dropping the exploit each branch closes. Comments and
whitespace only (no code change); the classifier tests are unchanged and pass.
* Retry transient SSE stalls in the tool-calling smoke probes
The tool-calling job flaked with a bare "TimeoutError: timed out": the
server-side python/bash probes stream over post_sse(), which (unlike
post()) had no transport-level retry, so a single stalled stream on a
shared CI runner hard-failed the whole step even though function calling
had already passed.
post_sse() now mirrors post(): a transport-level stall (stream open or a
mid-stream read timing out) is retried once with a fresh request capped
at 300s, while HTTP status errors still surface immediately. The
Linux _run_tool_probe caps each attempt at 360s and treats a stall that
outlives the retry as a failed attempt (rotate to the next seed) instead
of raising, and the web_search probe uses the same 360s cap. A genuine
server wedge still fails (the retry also times out), so real regressions
are not masked. Applied to the Linux, macOS, and Windows inference-smoke
workflows, which share the probe.
* Close five more auto-mode classifier gaps from review
Each reproduces with a benign control:
- Path constructor aliased through an attribute (P = pathlib.Path) now folds
like the bare-name alias, so (P('/etc') / 'passwd').read_text() asks while a
/tmp alias stays safe.
- Callable defaults that are not plain names now bind the parameter: an
attribute writer (def f(s=np.save)), an archive constructor, a captured .open,
and partial(open, mode='w') fold like the equivalent assignment; a benign
default (np.mean) does not.
- A dynamic piece inside a sensitive name (open('/et' + chr(99) + '/passwd'),
which folds to '/et\x00/passwd') now asks: the literals around each dynamic
segment are matched against a credential target with the segment as any run of
non-separator chars, so an all-dynamic ('1 + 1') or segment-spanning
(a + '/' + b) path stays safe.
- MCP read-named tools now ask on REFRESH MATERIALIZED VIEW and REINDEX; a
'refresh' column or natural-language 'refresh' stays safe.
- A writer/open alias handed to a higher-order invoker (map(open, names, modes),
starmap(np.save, ...)) is gated even without a direct call site; a benign
map(len, ...) is unaffected.
Regression rows added to test_permission_mode.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Default tool pills off on model load so tool execution is opt-in
resolveToolsEnabledOnLoad turned the web-search and code pills on for
any tool-capable model when the user had expressed no preference. Default
them off instead, so tool execution is enabled only when the person
clicks the pill to turn it on; a saved preference (on or off) is still
honoured, so a user who already enabled tools keeps them on.
* Gate mark/subscribe MCP verbs and qualified higher-order writer invokers
- A read-prefixed MCP tool name carrying mark / subscribe / unsubscribe
(get_and_mark_read, get_and_subscribe) now asks; a 'mark' substring inside
one token (list_bookmarks) stays safe.
- The higher-order writer check now also fires for a qualified invoker
(itertools.starmap(open, ...), functools.reduce(open, ...)), matching the
bare-name map/filter form; the writer-check on the first arg keeps a benign
itertools.starmap(len, ...) or itertools.chain(...) safe.
Regression rows added to test_permission_mode.py.
* Close more auto-mode gaps and align the ask confirm fold across paths
Each classifier change reproduces with a benign control:
- MCP read-named tools now ask on reply / notify verbs (get_and_reply_email,
list_and_notify_users), on catalog writes COMMENT ON / SECURITY LABEL / LOCK
TABLE and CREATE|DROP|ALTER POLICY, and on state-changing PostgreSQL functions
inside a read-shaped SELECT (nextval, set_config, pg_notify, the advisory-lock
family). A 'comment' column, a 'locks' table, and a 'nextval' column prefix
stay safe; the natural-language NOTIFY/SET ROLE statement forms are left out
because SET/NOTIFY overlap ordinary prose.
- Python auto mode now gates loader.exec_module (runs a module's code), archive
extractall (zip-slip file writes), the ensurepip / venv modules (install pip /
build an environment), and pydoc.writedoc. The Hugging Face login token
(~/.cache/huggingface/token and stored_tokens) is now a sensitive path, while
the rest of that cache (model data) stays readable.
- ChatCompletionRequest no longer overwrites an explicit confirm_tool_calls=false
when permission_mode='ask': the fold only self-enables the gate when the flag
is unset, so an explicit opt-out wins on the chat path exactly as it already
does via _permission_mode_confirm and the Anthropic pre-switch guard.
Regression rows added to test_permission_mode.py.
* Gate sort -T, xxd outfile positional, and the legacy HF token path
- sort -T / --temporary-directory writes spill files to a caller-chosen dir,
so it joins -o / --output in sort's unsafe-flag set.
- xxd [infile [outfile]] writes its second positional, like uniq; xxd now uses
the same second-positional-write handling (xxd in.bin out.hex asks, xxd
in.bin and xxd -c 16 in.bin stay read-only).
- The sensitive-path regex now also covers the legacy ~/.huggingface/token
location (optional leading dot), not just ~/.cache/huggingface/token; an
unrelated dir like myhuggingface/token stays safe.
Regression rows added to test_permission_mode.py.
* Catch multi-char SQL mutation targets, globbed credential names, digit outfiles
Three fail-open gaps in the auto-mode classifier, each with a benign control:
- SQL: the trailing word boundary on the MCP mutation regex meant a bare \w
stopped at the first character, so TRUNCATE users, GRANT SELECT ON t, and
REVOKE ALL ON t (multi-character names) slipped through while single-letter
targets matched. Match the whole identifier instead, and accept an explicit
AS alias on UPDATE (UPDATE users AS u SET). The implicit-alias form is left
out because it is indistinguishable from the prose "update <noun> <noun> set".
A truncate_log column and a grants table stay safe.
- A glob that resolves to a credential basename anywhere (cat ~/.huggingface/tok?n
-> token, cat proj/.netr? -> .netrc, cat repo/.aws/cred*) now asks; the fixed
target list only covered a handful of home paths. notes/dra?t.txt and
token_counts.tx? stay safe.
- uniq / xxd counted file positionals but skipped every numeric token to ignore
a flag value, so a file literally named with digits (uniq 123 out) hid the
output positional. Track each command's value-taking flags and consume only
the value, so uniq -f 2 in stays safe while uniq 123 out asks.
Regression rows added to test_permission_mode.py.
* Isolate the permission-mode loop tests from process-global state
The loop-driving tests (auto/off/full/bypass) drove run_safetensors_tool_loop
against a process-global approval registry (state.tool_approvals._pending)
keyed by a single shared session id, and read os.environ. Other backend test
modules mutate both, some at import time, so in the full-suite ordering a stale
pending approval or a leaked env var could make the loop deny or skip a call
these tests expect to run. It passed when the file ran alone but failed only in
the complete tests/ run on CI.
Add an autouse fixture that snapshots and restores os.environ and the approval
registry around each test, and give every _drive call a unique session id so a
leaked approval can never collide. Attach a compact event-stream dump to the
loop assertions so any residual full-suite-only failure reports what the loop
actually did instead of a bare diff.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden auto-mode classifier for recursive listers, sort file lists, aliased invokers, single-member extract
Close four fail-open gaps in is_potentially_unsafe_tool_call:
- terminal: tree/du (always recursive) and ls -R rooted at an absolute or
tilde path now ask, matching the existing grep/rg/find recursive-read gate;
relative walks stay safe.
- terminal: sort --files0-from=F reads the file list named in F, so it can
read arbitrary host files indirectly; added to sort's unsafe flags.
- python: track aliases of the higher-order invokers (m = map;
from itertools import starmap as sm) so an aliased invoker handed open/a
writer is still gated; a benign callable (map(len, ...)) stays safe.
- python: single-member archive extract (ZipFile/TarFile.extract) writes to
disk like extractall and is vulnerable to a crafted member path, so gate it.
Also update the stale _FakeExecuteTool in test_permission_mode.py to accept
the thread_id keyword that run_safetensors_tool_loop now forwards to
execute_tool after the main merge, which had broken the five tool-loop tests.
Adds regression rows covering each gap plus benign controls.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: normalize unknown permission_mode to 'ask' instead of a 422
The request models validated permission_mode with Literal[ask, auto, off,
full], so an unrecognized value from a newer UI/client was rejected with a 422
before the tool loops could apply their unknown -> ask fallback
(safetensors_agentic.py:464, llama_cpp.py:9001). That made the intended
forward-compat degradation unreachable at the API boundary for both Chat
Completions and the analogous Anthropic field.
Accept a plain string on both ChatCompletionRequest and AnthropicMessagesRequest
and normalize in a before-validator: None stays unset, the four known modes pass
through, and any other value degrades to the safest gate ('ask'), matching the
loops. Adds a regression test covering unknown/None/known across both models.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: close five more auto-mode classifier gaps
- terminal: xargs is no longer a safe wrapper. It appends arguments read from
stdin that the scan never sees, so `echo -o out /etc/passwd | xargs sort`
forwards to `sort -o out /etc/passwd` (a write + sensitive read) while only
the allow-listed literals are visible. Any xargs command now asks.
- terminal: ionice -p/-P/-u change the I/O priority of an already running
process / group / user instead of forwarding to a wrapped read-only command,
so `ionice -c 3 -p <pid>` now asks. ionice -c 3 <cmd> stays safe.
- MCP: gate ALTER SYSTEM, which persists PostgreSQL server configuration and was
not one of the DDL objects the mutation detector matched.
- MCP: a credential noun in a read-named tool (read_secret, list_tokens,
get_credentials, fetch_api_key) is a sensitive disclosure, so it asks even
without a mutating verb or a path/SQL argument. Scoped *_key nouns keep a
primary_key / keyboard lookup safe.
- render_html: no longer unconditionally safe. A static canvas still auto-runs,
but one whose HTML/JS reaches the network (fetch/WebSocket/remote script) asks,
since it can egress under the canvas CSP when artifact network access is on.
Its early provisional card is suppressed under the auto confirm gate, and the
confirm-without-stream guard now requires a stream when render_html is
selectable.
Adds regression rows and benign controls for each, and updates the render_html
provisional-card and confirm-gate tests to the new behavior.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: extend auto-mode gates for indirect file lists, dynamic lookups, HTML network loads, and Anthropic render_html
Follow-ups on the previous classifier round:
- terminal: wc/du/find --files0-from (and find's -files0-from primary) read a
NUL-separated list of input paths from a file, the same indirect mechanism as
sort --files0-from, so a crafted list reads arbitrary host files past the
literal path/root checks. Gate them like sort.
- python: a namespace lookup through a dict-style call (f =
__builtins__.__dict__.get('open'), globals().get('open'), vars(x).get(...))
can return open/eval/a mutator, so poison the bound name like getattr/subscript
lookups already are. An ordinary dict .get or os.environ.get stays safe.
- render_html: broaden the network detector so a canvas that loads a resource
via CSS url()/@import, srcset, or a root-relative (/path) or protocol-relative
(//host) src/href is treated as networked, not just fetch/WebSocket/remote
script. Relative ./x and url(#id)/data: refs stay static/safe.
- Anthropic /v1/messages: drop render_html from the unprompted-safe server-tool
set. Since it can prompt (networked canvas) and this channel invokes the loop
without confirm, selecting it under ask/auto/omitted now rejects like
terminal/python; off/full (or an explicit confirm opt-out) run it.
Adds regression rows and benign controls for each, plus an Anthropic route test.
* Studio: close six more auto-mode classifier gaps
- terminal: a glob that expands to a project .env (cat .e?v) now asks; .env
joins the sensitive glob-basename set, matching the literal-path gate.
- python: an open bound onto an attribute (box.f = open; box.f('out','w'))
is tracked by attribute name, and open invoked via .__call__
(open.__call__('out','w'), unwrapped to the underlying callable) is gated,
so neither slips past the name-based open-alias checks. Benign attribute
callables and .__call__ on non-writers stay safe.
- python: a namespace lookup via .get/.pop/.setdefault already covered the
builtins case; unchanged here.
- MCP: a mutating HTTP verb in a method/verb argument (get_url
{"method": "DELETE"|"POST"|"PUT"|"PATCH"}) now asks, so a generic HTTP
tool cannot mutate an external service unprompted; GET/HEAD stay safe.
- MCP: a credential/secret environment-variable value (get_env
{"name": "OPENAI_API_KEY"}) is treated as a sensitive read via the same
credential-noun match used for tool names; PATH/HOME stay safe.
- render_html: self-navigation sinks (location.assign/replace, window.open,
assigning a URL to (window.)location(.href)) join the network detector, so a
canvas that navigates itself to an external URL asks; location.reload() /
history.back() stay static.
Adds regression rows and benign controls for each.
* Studio: gate obfuscated canvas egress, sensitive-dir iteration, and MCP metadata-host reads
- render_html: strip block comments before the network scan so fetch/*x*/(...)
cannot hide egress, and match bracket-access forms (window['fetch'](...),
self['open'](...)). Line // comments are left alone so the // in an https URL
is not eaten. A comment-only canvas stays static.
- python: enumerating a directory outside the sandbox (Path('/etc').iterdir(),
os.scandir('/etc'), os.listdir('/home'), os.walk('/')) reads host filenames
the direct /etc/passwd checks would prompt for, so gate it when the target dir
folds to an absolute/tilde/sensitive path; a relative dir stays safe and an
unresolved dynamic dir is left to other checks.
- MCP: a read-named HTTP tool pointed at a cloud-metadata / link-local host
(fetch_url {"url": "http://169.254.169.254/..."}, metadata.google.internal)
reads instance credentials, so classify those URL arguments as sensitive,
mirroring the sandbox SSRF blocklist; ordinary and localhost URLs stay safe.
Adds regression rows and benign controls for each.
* Studio: gate meta-refresh navigation, pandas HTML/markdown exporters, absolute glob roots, and checksum verify mode
* Studio: gate starred open writes, builtins.__import__, computed render_html sinks, and procfs fd reads in auto mode
* Studio: gate remote worker canvases, huggingface_hub downloads, and write callables passed to user helpers in auto mode
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: offer the latest transformers release for brand-new architectures
When a model's config.json model_type is absent from every installed
transformers overlay (base 4.57.x and the .venv_t5_530/550/510 sidecars),
Studio now checks, unauthenticated and cached, whether the newest
transformers ships it:
- utils/transformers_latest.py fetches the latest release version from
https://pypi.org/pypi/transformers/json and the CONFIG_MAPPING_NAMES
sources for that tag and for main from raw.githubusercontent.com
(never api.github.com), parsing them with the same AST extractor the
static router uses (no code execution, no trust_remote_code). Results
are cached in memory and in a JSON snapshot under studio_root()/cache
with a one day ttl; fetches are bounded to 5s with one retry and a
failure backoff, and offline mode or the new kill switch
UNSLOTH_STUDIO_NO_LATEST_TRANSFORMERS=1 short-circuits to None.
- POST /api/inference/validate gains requires_transformers_upgrade plus
a transformers_upgrade payload (model_type, pypi_version,
supported_in_pypi, supported_in_main) so the frontend can raise the
install consent dialog before /load, mirroring the existing
remote-code consent flow. The check fires only when the model_type is
unknown to all installed overlays and the hardcoded tier tables.
- POST /api/inference/install-latest-transformers provisions a new
persistent .venv_t5_latest sidecar after user consent, pinned to the
exact PyPI version (re-verified server-side) with the same
--target/--no-deps recipe as the fixed sidecars. A JSON pin marker
inside the dir records the installed package set, so restarts
revalidate it and routing resolves the new highest-ranked tier
automatically. A dependency preflight (compat_plan) compares the
release's requires_dist against the running env: unsatisfied
tokenizers/safetensors floors are shadow-installed as exact pins into
the sidecar, anything else unsatisfied blocks the install with a
clear message.
Routing for every already-supported model_type is unchanged: the
hardcoded lists and the 530/550/510 static resolver run first, the new
tier only participates once its venv exists, and the probe order gains
the latest sidecar only when provisioned. Verified against live PyPI
and GitHub (transformers 5.13.0: 674 model_types, 26 absent from all
installed overlays, e.g. cosmos3_omni; 4 dev-only on main) and with a
real sidecar install plus restart persistence. 64 new tests; the
existing 200-test transformers_version suite passes unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Latest-transformers check: fetch outside the lock, serialize installs
Release the module lock during the network refresh so a slow fetch cannot
stall other threads in the ASGI pool; concurrent callers during a fetch get
None (the graceful fallthrough) via an in-flight flag instead of stacking
fetches. Serialize install_latest_transformers with an in-progress flag so
concurrent consents cannot race the sidecar delete and recreate; the loser
gets a structured already-in-progress refusal.
* Latest-transformers check: LoRA bases, pin-gated mapping, live reverify
Run the upgrade check over the [adapter, base] target set so a LoRA whose
base model is a brand-new architecture surfaces the prompt (the worker
activates transformers for the base, not the adapter).
Gate the latest overlay's mapping lookup on a valid pin marker, matching
activation and the probe order, so a partial or manual .venv_t5_latest dir
cannot be routed to and then refused at activation.
Re-verify the requested version against a live PyPI snapshot at install
time, falling back to the cached one on fetch failure, so a release
published inside the cache TTL is not silently missed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Latest-transformers check: nested config types and latest-tier vision probe
Collect every model_type in the config (top level plus each nested
sub-config) and signal on the first one missing from all installed
overlays, so a supported wrapper carrying a brand-new backbone still
surfaces the upgrade prompt; wrappers instantiate sub-configs through
CONFIG_MAPPING and would fail on the nested type.
Route the vision capability subprocess through the pinned latest sidecar
when the model resolves to the latest tier, so latest-only VLMs are not
misclassified as text-only; every other tier keeps the 5.5 sidecar used
today.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Latest tier: nested routing, vision probe after raw miss, safe upgrades
Route by every model_type in the config: a nested sub-config type can raise
the tier (wrappers instantiate sub-configs through CONFIG_MAPPING), so a
supported wrapper with a latest-only backbone routes to latest once
installed instead of staying on default. An unknown nested type never
vetoes; the primary type keeps its previous semantics. The collector is
shared with the upgrade checker.
Vision detection: when the raw heuristics say False for a model that routes
to the latest tier, run the AutoConfig subprocess under the pinned latest
sidecar instead of trusting heuristics built from older transformers.
Provisioning: stage-and-swap. Build the new sidecar in .venv_t5_latest.staging
and swap it in only when the install and pin marker are complete, so a failed
upgrade never destroys a previously working sidecar; restore the old dir if
the final swap fails.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Latest-transformers checker, vision subprocess, and cache fixes
Require the latest release to support every missing model_type (the
primary included) before prompting; a nested-only match cannot make the
model loadable, so no install is offered for it.
The vision-check subprocess now unions the active sidecar's own
registry mappings into the inlined parent-process detection sets, so
architectures only the sidecar knows classify correctly.
A successful sidecar install clears the tier probe cache, the latest
tier's model_type mapping, and the vision-detection cache so the new
venv takes effect without a restart. Tests for all three.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Aggregate upgrade support flags and keep install off /v1
The upgrade signal now reports supported_in_pypi only when the latest
release covers every missing model_type; a mix with a main-only nested
type surfaces as dev-only so no PyPI install is offered that would
still fail at load. The consented install endpoint moves to
studio_router so it is not reachable through the OpenAI-compatible /v1
mount. Tests for both.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor the latest-transformers kill switch in routing
With UNSLOTH_STUDIO_NO_LATEST_TRANSFORMERS set after the sidecar was
provisioned, the latest tier still joined mapping and probe routing
because only the pin was checked. Both admission points now also check
the kill switch, so operators can roll back a problematic sidecar
without deleting files.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Repair the latest sidecar through stage-and-swap
The lazy repair path installed into the live .venv_t5_latest, which
_ensure_venv_dir wipes first, so a failed repair deleted the pinned
sidecar and its marker. Both the consented install and the repair now
share one stage-and-swap helper: the incomplete-but-pinned dir survives
any failure and a later attempt can still repair it.
* Tighten comments
* Remove the staging dir when a latest-sidecar install fails
A pip failure inside _ensure_venv_dir returns False without raising, so
the except cleanup never ran and the partial .venv_t5_latest.staging
leaked until a later attempt. Also note on the validate response fields
that frontend consumption ships in the follow-up PR.
* Add the transformers-upgrade consent dialog to the frontend
When /validate reports requires_transformers_upgrade, every explicit load
path (chat runtime and the compare composer) now pauses on a consent
dialog modeled on the remote-code one: it names the model_type and the
latest PyPI transformers version, and on Accept calls
/api/inference/install-latest-transformers itself, shows an installing
state, and resumes the original load automatically on success. Errors
surface in the dialog with a retry; Cancel aborts the load like the
trust dialog's deny path. Architectures shipped only on transformers
main get a dev-only notice with no install button. Background auto-load
skips upgrade-requiring candidates instead of prompting, mirroring the
trust_remote_code rule. The dialog mounts once in the root layout and
runs before the security dialogs, since no load can proceed without the
runtime.
* Route a non-installable new architecture to the custom-code consent as a last resort
When the upgrade dialog has no installable PyPI release (the architecture
is only on transformers main, which Studio never installs), the dialog now
says so explicitly, and when the model also declares custom (auto_map)
code it offers Continue with custom code: resolving the paused load into
the existing trust_remote_code consent gate instead of hard-aborting.
Models with no custom code keep the Cancel-only notice. The backend
returns no upgrade signal at all for architectures unknown to both PyPI
and main, so those still route straight to the unchanged security gate.
* Force a 16-bit load for models on the latest-transformers sidecar
Live validation with Zyphra/ZAYA1-8B (model_type zaya, shipped by
transformers 5.13.1 but unknown to every installed tier) surfaced a
generation crash when the consented sidecar load kept the default bnb
4-bit quantization: transformers' grouped-MoE kernels feed the packed
uint8 expert weights straight into torch._grouped_mm, and generation
dies (plain 16-bit works). New latest_tier_active_for() mirrors the
sidecar activation's tier resolution and never raises; the inference
worker flips load_in_4bit off when it reports true, and the load route
applies the same flip so the pre-load VRAM guard and the worker command
agree. Fixed tiers are untouched. With the guard, ZAYA1-8B loads and
generates correctly in Studio chat.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Offer the custom-code fallback when a latest-sidecar install fails
* Fail remote mapping fetches wholesale and mirror the 16-bit flip in validate
A transient fetch or parse failure of one auto-mapping file no longer caches
a partial latest-release map for the TTL (a real 404 on pre-5.10 tags is
still tolerated), and validate_model now applies the same latest-sidecar
16-bit sizing flip as /load before the training guard so the two agree.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments in the latest-transformers changes
* Resolve remote LoRA bases, fold nested tiers, and guard the sidecar swap
latest_tier_active_for now resolves a remote adapter's base model the same
way worker pre-activation does (and returns early without a sidecar pin), a
hardcoded fast-path tier is raised when a nested sub-config's model_type
needs a higher sidecar, and the install route refuses to swap .venv_t5_latest
while training runs on it and unloads a latest-tier chat model first.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate the sidecar install on worker liveness and size installable upgrades 16-bit
The install route now refuses while any training or export runs (tier
re-resolution without the load token is unreliable for gated repos), holds
the inference lifecycle gate across the unload and the swap so no load can
interleave, and passes the model name to unload_model. validate_model runs
the upgrade check before the training guard and sizes an installable
upgrade as 16-bit, matching what /load and the worker will force after the
consented install.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close the sidecar install races and honor the kill switch over cached mappings
Training starts and mutating export routes now refuse while a transformers
install is in progress (shared is_install_in_progress flag), the chat unload
and idle export-worker teardown moved into a before_swap hook that runs only
once the staged install succeeded, and _config_model_types checks the kill
switch before returning a cached latest mapping.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reserve the sidecar swap before the gate wait and abort it on failed teardown
The install-in-progress flag moved into a shared sidecar swap reservation in
transformers_version, taken by the install route before awaiting the
inference lifecycle gate (so training and export starts see it for the whole
window) and by the lazy .venv_t5_latest repair path. The before_swap hook
now raises when the chat unload or export teardown reports failure, leaving
the previous sidecar untouched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Back the sidecar swap reservation with a cross-process lock file
The lazy repair runs inside worker subprocesses, where a module-level flag
is invisible to the parent's route checks. The reservation now also creates
a lock file next to .venv_t5_latest (O_EXCL, owner-only removal, stale after
two hours for crashed owners), so is_install_in_progress sees a repair from
any Studio process.
* Hand the swap reservation to the installer thread and harden pre-swap teardown
A cancelled install request no longer releases the reservation while the
installer thread is still staging (the thread owns and releases it, shielded
from cancellation). The route refuses while another inference request is
generating, export teardown runs before the chat unload and is judged by
worker liveness rather than the cleanup return value, and a live inference
worker with no active model (failed load residue) is shut down before the
swap.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the lifecycle gate with the installer and recheck the swap at spawn time
The gate moved into the shielded install task so a cancelled POST cannot
release the guard /load honors while the installer still runs, cached latest
probe results are ignored while the kill switch is set, and the training and
export subprocess spawns recheck the sidecar swap reservation right before
spawning (the route-level guards are one-shot and validation can outlast an
install's start).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Close the spawn-registration windows against the sidecar install
Training marks the spawn in progress before its reservation recheck and
is_training_active honors the flag, so the install route sees a start that
has passed proc.start() but not yet recorded _proc. Export load-checkpoint
rechecks the reservation after setting _export_active and before tearing
down the old worker, so losing the race keeps the loaded checkpoint instead
of surfacing a 500.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Refine the install-window interleavings around worker teardown
The inference busy count is rechecked under the lifecycle gate (streams
start by taking that gate, so nothing slips past a held gate), the training
handshake moved ahead of the VRAM-freeing before_spawn hook so a lost race
leaves chat/export intact, the export spawn-time check is op-aware (inside
an active op the install is the side that aborts), and the Xet-stall respawn
waits out a transient reservation instead of stranding the run.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Track the install's server-side unload and guard export ops against the swap
The upgrade dialog store records when its install actually ran (the server
unloads the active chat model before swapping), and the load flow then marks
the previous model as unloaded so a later cancelled gate still triggers
rollback; the custom-code fallback leaves the flag unset. _run_export gained
the same reservation handshake as load_checkpoint so an install cannot block
behind an hours-long export op instead of returning 409.
* Tighten comments in the install-guard and upgrade-consent changes
* Surface install-race refusals cleanly and roll back after a failed swap unload
/load refuses while the sidecar swap is reserved so a load cannot succeed
and immediately be unloaded by the pre-swap teardown, worker starts that
lose the install race raise a typed SidecarSwapInProgress mapped to 409
instead of a 500, the install response reports model_unloaded even on a
structured failure so the client can restore its state, and the compare
flow tracks the server-side unload like the primary load path and clears a
stale checkpoint on abort.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Type the export install races, scope the lock release, and keep the unload signal
Export load-checkpoint and export ops raise SidecarSwapInProgress (mapped to
409 in every export route) instead of a 400-shaped failure, the export spawn
check distinguishes repair reservations (always refused) from install ones
(op-aware), the swap lock release only unlinks a lock this process wrote so
a stale-superseded owner cannot drop the new owner's live lock, and the
frontend unload signal survives a superseding consent via read-and-clear
consumption instead of a reset.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Finalize a stalled run when the respawn loses the install race and latch the unload signal
The Xet-stall respawn timeout now finalizes the run as a failure instead of
raising into the pump's broad finalization catch (which stranded it in a
training state with no worker), and a successful install retry ORs the
model_unloaded signal with the latched value so a failed-after-unload first
attempt still triggers rollback.
* Recheck the swap under the load gate and latch the unload before resolver checks
/load rechecks the sidecar reservation after acquiring the lifecycle gate
(an install can reserve while the load queues on it), and the dialog store
latches model_unloaded as soon as the install response arrives, before any
resolver-identity guard, so a superseded consent's unload still reaches
whichever load consumes the signal next.
* Report cleared-state unload failures, guard queued installs, and fold name tiers
A failed chat unload that still cleared the orchestrator's model state now
reports model_unloaded so the client rolls back, the installer aborts with
a 409 when a model load completed while it waited on the lifecycle gate,
and the fixed-tier name fast path consults the config mapping when a latest
sidecar is pinned so an accepted upgrade routes to the sidecar it installed
(no I/O added to the unpinned path).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Report cleared-state unload failures and harden the spawn handshake flag
The failed-unload branch in before_swap now detects that the orchestrator
cleared its model state and reports model_unloaded before aborting (the
earlier commit claimed this fix but a scripting error dropped the edit),
the installer's queued-load check compares a load generation counter so a
same-model reload is caught, and both training spawn sites wrap everything
after the handshake in a guard that resets _spawn_in_progress on any
exception so a failed start cannot wedge is_training_active.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Bump the load generation when the load is published, not at load start
A start-time bump is already visible when the installer snapshots mid-load,
so a same-model reload completing after the snapshot looked unchanged and
could be unloaded by the swap. The counter now increments alongside the
active_model_name publish.
* Self-heal a broken pinned sidecar, guard lazy repairs, and refresh stale retries
A valid pin whose transformers source dir vanished now triggers the repair
from the routing path (with a five minute backoff after failures) instead of
silently routing latest-only models to older tiers, the lazy repair refuses
while parent-visible chat/training/export workers are active since it has no
teardown of its own, and a version-mismatch install failure carries the
superseding release so the dialog's Retry re-requests a version that can
succeed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Flip latest-tier loads to 16-bit outside chat and protect export state
Training and export workers now apply the same latest-sidecar 16-bit flip
as the chat worker so a brand-new grouped-MoE architecture cannot reach bnb
4-bit through those paths, the latest-tier vision override returns None on
an inconclusive probe so a transient failure is not cached as not-vision,
and the install route refuses while an idle export checkpoint is loaded
rather than discard it with no rollback signal on a failed swap.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address parallel-review findings on the sidecar guards and install checks
The training route sizes latest-tier jobs 16-bit before GPU selection, the
inference subprocess spawn rechecks the swap reservation like training and
export (covering the OpenAI auto-switch path) with the typed error mapped
to a retryable 409, compat_plan blocks the install when dependency metadata
cannot be fetched instead of proceeding unverified, snapshot model-type
lists must contain only strings, and pin-marker package specs are validated
against the sidecar's own package set before ever reaching pip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Parent-only repairs, live-owner locks, remote-base activation, pre-teardown recheck
Lazy sidecar repairs now refuse inside worker children (whose empty backend
singletons cannot see live siblings) and run only in the parent where the
active-worker guard is real, swap-lock staleness requires the owner pid to
be dead so a slow live install is never superseded, both activation entry
points resolve a remote adapter's base model like the inference worker and
latest_tier_active_for already do, and load_model rechecks the reservation
before tearing down the old worker so losing the race keeps the current
model loaded.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Check workers under the repair reservation and keep state on refused swaps
The lazy repair now reserves first and checks workers under the reservation
(worker starts set their active markers before rechecking, so every
interleaving aborts one side), with export ops and in-flight inference loads
counted as active. The inference pre-teardown and spawn guards refuse only
repair reservations since an install shares the load's lifecycle gate and
aborts via its queued-load snapshot, a SidecarSwapInProgress raised before
teardown no longer clears the live model mirrors, and an export spawn abort
after teardown clears current_checkpoint so the page cannot claim a loaded
checkpoint with no worker.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Repair a present-but-incomplete latest sidecar from routing
The routing self-heal only fired when the pinned sidecar's transformers/
dir was missing. A sidecar that kept transformers/ but lost another pinned
package still routed models to the latest tier, and workers refuse
parent-only repairs, so every load failed until a manual reinstall. Routing
now validates the full pin (via _venv_dir_is_valid) and repairs any
incomplete sidecar under the same swap reservation and 5-minute backoff.
* Treat an unrepaired latest sidecar as unavailable in routing
When the pinned sidecar is incomplete and the lazy repair fails (offline,
pip failure, workers active) or is inside the backoff window, routing
returned the source dir anyway, sending models to a tier whose worker
activation is known to fail. Return None instead so models an older tier
supports keep loading there until a repair succeeds, matching the behavior
when the sidecar dir is missing entirely.
* Harden sidecar swap and repair against crash, survivor, and 16-bit paths
Reclaim a swap lock as soon as its recorded owner PID is dead instead of
waiting out the two-hour cutoff, so a crash mid-install no longer wedges
/load, training, export, and repair for hours. A lock whose PID cannot be
read yet still uses the long cutoff so the create-before-write window is
never mistaken for dead.
Probe process liveness with OpenProcess on Windows: os.kill(pid, 0) there
is CTRL_C_EVENT (a real Ctrl+C via GenerateConsoleCtrlEvent), not a
harmless check, and psutil is not always present.
Return whether _shutdown_subprocess actually killed the worker and keep the
live handle when it survives terminate/kill (an uninterruptible CUDA
syscall can outlive SIGKILL). The pre-swap liveness guard now trusts that
result, so the destructive .venv_t5_latest rename cannot proceed while a
live worker still holds sidecar modules.
Recover a sidecar stranded at .old when a swap's activation rename and its
rollback both fail: reading the pin restores it when no swap holds the
reservation, so latest-tier models are not permanently broken.
Resolve the latest tier in the parent for export loads and for explicitly
16-bit training runs, not only 4-bit ones: tier resolution self-heals an
incomplete sidecar, and repairs are parent-only, so those paths could not
recover before. Sidecar integrity and quantization are independent.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Revert the parent-side latest-tier repair probe on training and export loads
The probe ran before the route freed VRAM, so a resident chat or export worker
made _workers_active_for_repair() refuse the parent-only repair; the route then
tore that worker down and spawned a child that also cannot repair, so an
incomplete sidecar still failed to load. Repairing correctly requires running the
repair between the worker teardown and the child spawn, decoupled from VRAM
sizing, which is a larger change tracked separately. Restore the prior behavior
so these paths match the reviewed form and do not partially attempt a repair that
cannot complete while workers are resident.
* Honor failed worker shutdowns on load and revalidate the cached latest mapping
The fresh-load paths spawned a new worker straight after _shutdown_subprocess
without checking its result, so a worker that outlived terminate/kill (a wedged
CUDA syscall) had its handle overwritten by the replacement while it still held
GPU memory, and is_worker_alive/the pre-swap guard could no longer see it. Both
the inference load and the export checkpoint load now abort when the old worker
did not exit, so the load can be retried once it does.
_config_model_types returned a cached latest mapping without re-checking the
sidecar, so a sidecar deleted or broken in-process after its first parse was
never re-validated: routing kept sending latest-only models to the stale latest
tier while activation failed. The cached latest mapping is now dropped and
re-resolved (self-healing) when the sidecar is no longer intact.
* Drop cached latest mapping when the pin is gone; keep 4-bit for custom-code fallback
_latest_sidecar_intact now returns False when the pin marker itself is gone, not
just when a pinned package is missing. Otherwise a cached latest mapping outlived
a deleted pin: _config_model_types kept returning it, so routing sent latest-only
models to a tier whose worker activation then failed (no pinned version) until
restart. It now drops the cache and re-resolves to no latest tier. The
_overlay_transformers_dir caller already gates on a present pin, so it is
unaffected.
validate_model forced 16-bit sizing whenever a PyPI upgrade was merely offered,
even for a model that can fall back to its own auto_map code. /load loads such a
model 4-bit without the install, and the install route refuses while training is
active, so 16-bit sizing here returned a VRAM 409 for the only viable 4-bit path.
The offered-upgrade flip is now gated on the absence of a custom-code fallback;
an already-active latest sidecar still always sizes 16-bit.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: persistent stdio MCP sessions so server state survives across tool calls
call_tool_sync spawned a fresh stdio subprocess per tool call
(keep_alive=False) and tore it down when the call returned, so any stateful
MCP server lost its state between calls: with @playwright/mcp,
browser_navigate opened the page in one subprocess and
browser_take_screenshot ran in a brand-new one, screenshotting about:blank.
Keep one connected client per (command, env) on a dedicated event-loop
thread and reuse it across calls:
- idle sessions are reaped after 5 minutes (in-flight calls excluded) and
everything closes at exit, preserving the old design's no-orphans property
- a dead subprocess is detected via is_connected() and retried once on a
fresh session; tool-level errors leave the session alone
- cancel and timeout semantics are unchanged, and a timed-out call does not
tear the session down
- updating a server's endpoint/env/enabled state or deleting it closes its
live session
- HTTP/SSE servers stay one-shot per call
* address review feedback
* fix stdio session cleanup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* address review: per-thread MCP scope, close-during-connect and abort races
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* address review: unblock no-limit calls on close, drain borrowers before close, scope closes to url+env
* don't retry sessions closed by config changes, re-verify server row before caching, keep env secrets out of generation keys
* fail fast on connect errors and make the stdio key-lock wait cancellable
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* quote MCP scope parts so IDs with colons can't collide
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* serialize per-session stdio calls, span one timeout budget across connect and call, hash urls in generation keys
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden persistent stdio MCP sessions: crash recovery, concurrency, scoping
- Evict a stdio session on any transport-level (non-ToolError) call failure and
do not replay it, so a mid-call subprocess crash can no longer poison the scope.
Never gate liveness on Client.is_connected() (it only reports that a session
object exists, not that the subprocess is alive); add a version-adaptive
dead-transport probe that works on fastmcp 3.0.2 and newer.
- Re-check closed/defunct/config and transport liveness after acquiring the call
lock, and retire a session before releasing the lock, so a queued same-scope
caller never reuses a session that another caller's timeout already retired.
- Force a ProactorEventLoop on Windows so the stdio transport can always spawn
subprocesses regardless of the active event-loop policy.
- Scope stdio sessions per conversation: require thread_id to persist, and tag
the fields so a session_id and a thread_id with the same value cannot collide.
A session_id alone is project-wide, so it now falls back to a safe one-shot
session instead of sharing browser/DB/REPL state across conversations.
- Forward thread_id on the Anthropic Messages path.
- Treat timeout=None as unlimited on connect and the key lock (was capped at 60s).
- Bound the session cache (default 32, override via
UNSLOTH_STUDIO_MAX_STDIO_MCP_SESSIONS) with LRU eviction of idle sessions.
- Run config_check on cache hits, and log a redacted exe#digest label instead of
the raw command so credentials in argv never reach the logs.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim the stdio MCP session cache on release and skip close-generation for HTTP servers
Two fixes from review of the persistent stdio session lifecycle:
- Re-enforce the session cap when a session goes idle. A concurrent burst of
distinct-scope calls can overshoot the cap while every cached session is busy
(insert-time eviction only reclaims idle sessions), and the overshoot used to
persist until the 5-minute idle reaper. _release_stdio_session now trims the
idle overshoot back within the cap, without ever evicting an in-flight call.
- close_stdio_sessions() now no-ops for a specific non-stdio (HTTP/SSE) url.
Those transports are never cached as stdio sessions, so calling it on every
HTTP server update or delete used to accrue an unbounded close-generation entry.
Both are covered by regression tests that fail before the change and pass after.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the live stdio MCP session across a display-name rename
The edit dialog resends url, headers, and use_oauth unchanged whenever a
server is saved, so gating the tool-cache invalidation and stdio session
close on field presence dropped the persistent process on a plain rename
or any no-op edit. Gate on a real value change against the stored row so
only a genuine endpoint, auth, or enable change closes the session.
Regression tests: a rename that resends unchanged url/headers/oauth keeps
the session; a real command change still closes it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments in the stdio MCP session lifecycle
Collapse a few verbose comments to fewer lines with the wording preserved,
and drop one that restated the clear_oauth_tokens_async docstring. Comments
only; no code change.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Studio /v1/messages: accept thinking and unknown content blocks
The Anthropic-compatible /v1/messages endpoint modeled a message's content as
Union[str, list[{text|image|tool_use|tool_result}]], so any other block type
made Pydantic reject the whole request with
`messages.N.content.str: Input should be a valid string`. Resuming a Claude
session commonly replays assistant turns that carry `thinking` (extended
thinking) blocks, and sometimes a null content for a tool-only turn, both of
which tripped this and returned a 400.
Accept them:
- Add a permissive AnthropicUnknownBlock fallback (any block whose type is not
one of the four known ones), so thinking/redacted_thinking/provider-specific/
future blocks validate. A validator keeps known types on their typed models,
so a malformed known block (e.g. a tool_use without id) still fails cleanly.
- Coerce a null message (and tool_result) content to "" so the converter's
`for block in content` stays safe.
The converter already drops block types it does not translate, so a thinking
block is not forwarded to the model.
* Studio /v1/messages: keep user content validation strict
Make the thinking/null leniency role-aware so it never silently drops real
user input. Assistant turns (replayed history) still accept unknown/thinking
blocks and coerce a null tool-only turn to empty. User turns keep the strict
boundary: a null user content is rejected, and a content block the converter
cannot translate is rejected instead of being dropped into an empty prompt.
Also remove an empty file committed by accident.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio /v1/messages: coalesce resumed user turns and tighten content checks
- The /v1/messages count and generation paths now coalesce the adjacent user
turns that dropping an empty or null assistant turn can leave behind, so a
strict GGUF chat template no longer 400s on non-alternating roles.
- A user content block with a non-string type (list / dict) is rejected as a
clean 400 instead of raising TypeError and escaping as a 500.
- The assistant null-to-empty coercion only applies to an explicit null; an
assistant turn that omits content entirely still fails required-field
validation instead of being silently coerced to an empty string.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio /v1/messages: tighten comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: apply presence_penalty on the safetensors and MLX inference paths
The safetensors and MLX generate paths resolved the inference config and
then dropped presence_penalty before generation, so the same model applied
the configured value under GGUF and 0 under safetensors/MLX. Thread the
already-resolved presence_penalty through the orchestrator command, worker
gen_kwargs, and the safetensors/MLX generate calls, and apply it with a
small logits processor (subtract once per distinct completion token,
prompt excluded, presence not frequency, zero is a no-op, negatives raise).
Backwards compatible: presence_penalty defaults to 0.0 (byte-identical
output when unset) and the GGUF path is unchanged. Also forward min_p on
the legacy /generate/stream route and add the missing min_p field to
GenerateRequest.
* Studio: bound presence_penalty generated ids to valid vocab range on both paths
The presence-penalty logits processors index by generated token ids. The
torch path filtered only the upper bound (seen < vocab_size), so a negative
id would silently wrap to the wrong row; the MLX path had no bound at all,
and MLX out-of-bounds indexing is documented undefined behavior (crash or
memory corruption on Apple Silicon), unlike torch's harmless negative wrap.
Bound generated ids to [0, vocab) consistently on both paths:
- torch: seen[(seen >= 0) & (seen < vocab_size)] (zero-regression safety net;
real completion tokens are always in range).
- MLX: route out-of-range/negative ids to a discarded scratch slot via
mx.where and a (vocab + 1)-wide scatter-assign mask, then subtract. MLX has
no boolean-mask filtering (data-dependent output shape), so this keeps a
fixed shape, stays on-device, and preserves once-per-distinct-token
semantics without any torch/numpy dependency.
Add torch tests for out-of-range and negative ids (only in-range distinct
ids penalized, stray ids ignored, no wrong-index wrap) and a bound-documenting
MLX test that runs on the arm64 macOS CI.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: client-tool passthrough healing for safetensors and MLX
PR 6801 made response-side tool-call healing default-on for the client-tool
passthrough, but only on the GGUF path: the passthrough branch in
/v1/chat/completions is gated on using_gguf, and the safetensors section never
reads payload.tools, so a client-tools request against a safetensors or MLX
model silently dropped the tool schemas and returned prose with no tool_calls.
Add the missing leg. When a non-GGUF model is loaded, the request declares
client tools (or carries tool-role history), server-side tools are off, and the
template supports tools, the route now:
- renders the tools into the chat template for a single turn via the existing
backend.generate_chat_response(..., tools=...) seam (worker templating
already accepts role=tool and assistant.tool_calls messages, normalized with
_openai_messages_for_passthrough);
- non-streaming: promotes text-form calls with heal_openai_message, honors the
opt-in nudge single retry (nudge_should_retry / nudge_messages), caps healed
calls when parallel_tool_calls=false (covers the nudge retry too), and sets
finish_reason=tool_calls with content null on a pure tool-call turn;
- streaming: derives deltas from the worker's cumulative snapshots and feeds
StreamToolCallHealer, emitting healed tool-call deltas and the correct
finish chunk, guarded against repeated or shrinking snapshots.
heal_gate semantics are identical to the GGUF passthrough: default on,
auto_heal_tool_calls=false or UNSLOTH_DISABLE_TOOL_CALL_HEALING=1 relays
verbatim, tool_choice narrows promotion, undeclared names stay text. MLX rides
the same orchestrator seam, so both local backends gain the behavior.
CompletionMessage.content becomes Optional so a promoted pure tool-call turn
matches the OpenAI contract (content null when only tool_calls return).
Adds tests/test_sf_client_tools_passthrough.py (22 cases: healing, gating,
opt-outs, streaming deltas, tool-role history, dict-arguments history, forced
tool_choice, parallel cap, usage, nudge on/off/double-failure, generator error
hygiene, disconnect reset, empty output, MLX path).
* Address review: tool_choice none, developer folding, retry fallback, monitor reply
Four review follow-ups on the safetensors/MLX client-tool passthrough leg:
- tool_choice="none" keeps the tool-history templating but no longer
advertises the tools, so a forced final-answer turn is not prompted into
emitting markup that the (correctly disabled) healer would relay as prose.
Mirrors the GGUF passthrough where llama-server honors tool_choice itself.
- OpenAI "developer" messages fold into a single leading system message via
_set_or_prepend_system_message before templating; local templates reject the
role and the fallback formatter drops it.
- A nudge retry that fails or is cancelled after the original answer exists
falls back to the first response instead of surfacing a 500, matching the
GGUF nudge path.
- The API monitor records the healed tool call summary instead of the raw
markup on a promoted turn.
Adds four regression tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: forced tool_choice templating, content-part flattening, stream monitor parity
- A forced tool_choice function is now the only schema rendered into the
local template, so the advertised tools and the healer allowlist can no
longer disagree (llama-server enforces tool_choice itself on the GGUF path).
- Content-part lists are flattened to their text parts before templating.
Remote image URLs are not decodable locally, so such requests reached this
path with part lists that raise inside apply_chat_template on text-only
templates; the plain non-GGUF path has always flattened them.
- The streaming monitor entry is now fed from the healed events the client
actually receives, recording promoted calls as the [tool_calls] summary
the non-streaming path records.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: gate passthrough on the engaged server path, deserialize templated arguments
- The client-tools gate now keys on _sf_use_tools (whether the server-side
tool path actually claimed the request) instead of the raw mcp_enabled
flag: with an empty MCP registry or a CLI --disable-tools policy, a client
that sets mcp_enabled while declaring its own tools fell through to plain
generation with the tools silently dropped. The GGUF passthrough gate has
no mcp_enabled clause either.
- New _structured_tool_history_for_local_template deserializes assistant
tool_calls[].function.arguments JSON strings into mappings for the
templated copy only: spec-compliant clients send strings, but local chat
templates iterate arguments as a mapping or raise on strings, which
crashed or misrendered multi-turn tool history. The HTTP response and the
GGUF wire shape keep strings.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments and docstrings in the client-tools passthrough
* Report first-attempt usage when a nudge retry is discarded
When nudge_should_retry fires but the retry produces no healable tool call
(or raises), the first response is still delivered to the client. The retry's
generate() had already overwritten stats_holder, so _monitor_usage recorded
the unseen retry's token counts against the request instead of the first
attempt that was actually returned. Capture the first attempt's stats before
the retry and restore them on both the no-heal and exception paths so the
monitor reports the usage of the response the caller received.
* Do not promote buffered tool markup when a stream is cancelled
The streaming client-tool heal path breaks out of the token loop when
cancel_event is set (the registry "Stop" path), but then still fell through to
healer.finalize(), which heals incomplete tool markup at EOF (allow_incomplete)
and emits a tool_calls delta plus finish_reason=tool_calls. Because the Stop
request only sets the event and leaves the SSE socket open, the client received
that promoted call and executed a tool the user had just cancelled. The disconnect
path already returns before finalize; guard finalize and the finish_reason on
cancel_event too, so a cancelled stream ends with finish_reason=stop and no tool
call. Adds a regression test driving a Stop mid-emission with buffered markup.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments in the client-tools passthrough
* Trim client-tools passthrough comments further
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* inference: add passthrough tool-call healing core (heal_gate, heal_openai_message, StreamToolCallHealer, nudge helpers)
Small GGUF models often emit tool calls as text (<tool_call>{...}</tool_call>,
Gemma <|tool_call>, <function=> XML) instead of structured tool_calls. Studio's
enable-tools loop already heals these, but the client-tool passthrough
(unsloth run --disable-tools, unsloth start agents) relays them verbatim, so
the agent sees prose and the turn dies.
This module is the shared response-side repair layer the passthrough routes
will call: promote parsed text-form calls to structured calls, but only for
function names the client actually declared; coerce arguments through the same
canonical-key healing as the tool loop; never touch the upstream request body
(llama-server KV/slot reuse stays byte-identical). StreamToolCallHealer is the
streaming buffer-and-repair state machine: prose forwards immediately, only a
partial-signal tail or a suspected tool block is held, false alarms flush
verbatim, and a 64 KiB bound caps memory. nudge_should_retry/nudge_messages
support an opt-in single-retry nudge for non-streaming routes (wired later).
Kill-switch: UNSLOTH_DISABLE_TOOL_CALL_HEALING=1. Reuses
core/tool_healing.parse_tool_calls_from_text, strip_tool_call_markup, and
tool_loop_controller.coerce_tool_arguments unchanged.
* inference: heal text-form tool calls on the OpenAI and Responses passthrough
Wire the passthrough healing core into /v1/chat/completions and /v1/responses,
default ON whenever the request declares client tools:
Non-streaming: heal_openai_message runs inside the existing response-mutation
loop; a promoted call flips finish_reason to tool_calls and nulls the content,
and the verbatim-bytes fast path still applies when nothing was healed.
/v1/responses non-streaming inherits this through openai_chat_completions.
Streaming: a StreamToolCallHealer per stream. Ordinary prose relays
byte-for-byte (a fast path keeps upstream bytes when the healer passes a chunk
through whole); once a tool signal appears, content is held, and at the
finish/[DONE] boundary either synthetic delta.tool_calls chunks replace the
markup (finish_reason rewritten to tool_calls, including the synthetic-finish
path) or a false alarm flushes the held text verbatim. Structured upstream
deltas put the healer to sleep after flushing anything held, so grammar-mode
responses stay byte-identical. The Responses stream feeds healed calls through
the same per-call state machinery as structured deltas (indexes live in a
disjoint range so a healed call can never merge into a structured call's
state), and the visible/reasoning split runs first so reasoning text is never
promoted. parallel_tool_calls=false caps healed calls on every path.
The upstream request body is never touched and healing issues no extra
generation, so llama-server slot/KV-cache reuse is unchanged. Opt-out per
request with auto_heal_tool_calls=false (Responses reads it from the
extra-body); requests without tools relay verbatim.
* inference: heal text-form tool calls on the Anthropic /v1/messages passthrough
Streaming: AnthropicPassthroughEmitter.enable_healing(allowed_tools) routes
content deltas through the shared StreamToolCallHealer. A promoted call closes
any open text block (only the safe prose prefix ever streamed into it), opens a
synthetic tool_use block with a fresh toolu_* id, carries one input_json_delta,
and closes; finish() then forces stop_reason to tool_use unless a truncation
(max_tokens) wins. Structured upstream deltas flush anything held and put the
healer to sleep, so grammar-mode responses are untouched, as is every stream
where enable_healing is never called (Studio's own loop, no-tools requests).
disable_parallel_tool_use caps healed calls too.
Non-streaming: the OpenAI message dict is healed BEFORE block building, so the
existing tool_use promotion loop and stop_reason line treat promoted calls
exactly like native ones (finish_reason length still maps to max_tokens). The
legacy tool-XML strip still runs on remaining text, so opted-out requests keep
today's cleanup behavior byte-for-byte.
auto_heal_tool_calls is now a typed field on AnthropicMessagesRequest
(default True, mirroring Chat Completions) and threads into both passthrough
calls. Healing never touches the upstream request body.
* inference: opt-in single-retry tool-call nudge on the non-streaming passthrough
When the model clearly tried to call a tool (a tool signal in the text) but
healing produced nothing usable, re-ask once: the retry body is the original
body plus an assistant turn (the model's own failed text) and a short user
nudge naming the declared tools. The prompt prefix stays byte-identical, so
llama-server reuses the slot's KV cache and only the two-message suffix is
prefilled. The retry replaces the original response only when it actually
yields a promotable or structured call; on any error or still-garbage output
the original response is returned unchanged. Exactly one retry, non-streaming
OpenAI and Anthropic passthroughs only (a stream has already emitted bytes).
OPT-IN per user decision: nudge_tool_calls=true per request (typed on both
ChatCompletionRequest and AnthropicMessagesRequest, lifted from the Responses
extra-body), or UNSLOTH_TOOL_CALL_NUDGE=1 to flip the process default.
auto_heal_tool_calls=false disables healing AND the nudge.
Also align the non-streaming heal on allow_incomplete=True: the response is
final, so a trailing unclosed tool block is a model failure worth repairing,
matching the enable-tools loop's drain semantics.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: never assume the upstream response shape in the nudge helpers
llama-server error bodies can carry message: null (or no choices at all), and
_last_assistant_text / response_has_promotable_calls / nudge_should_retry
called .get() on the message without a dict check, so a malformed upstream
response raised an AttributeError the surrounding except tuples did not catch,
failing the request instead of degrading to 'nothing to heal'. Route the shape
probing through one _first_choice_message helper that returns None for any
non-dict message, and add a parametrized test over the malformed shapes.
* inference: constrain healing by tool_choice, preserve length finish_reason, keep healed event order in Responses streams
Three review findings on the passthrough healer:
- heal_gate now honors the request's tool_choice: "none" disables healing
outright and a forced function narrows the promotion allowlist to that
one function, so healing can never contradict the request's tool-choice
constraint. Wired through the OpenAI chat (stream and non-stream),
Responses, and Anthropic (converted shape) passthroughs.
- The OpenAI non-streaming heal only upgrades finish_reason "stop" to
"tool_calls"; a truncated generation keeps "length" (the healed call
stays attached) matching the streaming and Anthropic paths.
- The Responses stream emits healer events in order instead of collapsing
all text ahead of the healed calls, so text after a healed call no longer
jumps ahead of the function_call item and output indexes are claimed in
the order the model produced them.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: all-or-nothing promotion when a response mixes declared and undeclared text-form calls
Promoting a subset used to strip ALL tool markup from the content, which
silently deleted the text of any call naming an undeclared tool. The heal
now declines entirely when any parsed call is unpromotable, so the whole
message relays verbatim (pre-PR behavior) and no bytes are ever lost. In
streaming, a declared call that completed before an undeclared one arrived
is already emitted; the late undeclared markup still flushes as raw text.
The nudge helpers mirror the same contract via a shared predicate.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests: wrap long lines in the Responses healing tests to the project style
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: span-exact healing, disjoint healed stream indexes, per-call Responses message items, allowlisted nudge acceptance
Four review findings on the passthrough healer:
- parse_tool_calls_from_text gains an optional with_spans return so healing
removes EXACTLY the promoted calls' markup. This supersedes the previous
all-or-nothing rule: declared calls promote and every unpromoted byte
(undeclared calls, unparseable closed blocks, suppressed alternate
formats such as a <function=...> block after a JSON call) relays as text.
The stream healer also processes one block per pass, so text between two
healed calls keeps its document position instead of trailing them.
- The OpenAI chat stream shifts native tool-call delta indexes past any
already-emitted healed calls; clients merge deltas by index, so a healed
call and a later native call can no longer merge into one.
- A healed call in the Responses stream closes the open message item and
trailing text opens a fresh one with a later output index, matching the
native stream shape; response.completed snapshots every message item
with its own text.
- The nudge retry only replaces the original response when the retry's
structured call names a DECLARED tool; a hallucinated undeclared call is
not an improvement.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stop the heal path folding trailing prose into a closed function call
parse_tool_calls_from_text(allow_incomplete=True) cut a <function=...> body only
at an end-anchored </function>, so a fully closed call followed by trailing prose
(<function=..>..</parameter></function> words) folded </parameter></function> and
the prose into the tool argument and deleted the prose from visible content. The
strict path (allow_incomplete=False) already cut at the real </function> via rfind.
Do the same in both modes: trim the body at the real </function> when present and
end the removal span there, falling back to the end-anchored strip and body_end
only when the call is genuinely truncated. Add a regression test.
* inference: one shared single-call budget for healed and native calls
Codex round 5: the parallel-call caps counted healed and native calls
separately, so a healed text-form call followed by a native structured
delta double-emitted on all three streaming surfaces when the client
disabled parallel calls.
- OpenAI SSE: once a healed call went out with parallel_tool_calls
false, native tool_call deltas are dropped instead of index-shifted.
- Anthropic emitter: native deltas skip block allocation when the
healed-plus-native count already filled the single slot, and healed
emission counts open native states too.
- Responses stream: native deltas that survived the chunk-level cap are
skipped once a healed call claimed the slot.
Also adds a span assertion for the closed-</function> trailing-prose
parse fixed in the previous commit.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: relay undeclared text-form calls as text on Anthropic non-streaming
heal_openai_message promotes only declared text-form tool calls and
span-trims just their markup, deliberately leaving every unpromoted byte
(undeclared text-form calls included) in the content to relay as text.
The Anthropic non-streaming builder then ran a blanket _TOOL_XML_RE strip
over that content unconditionally, deleting the undeclared block before
building the text part, so Anthropic clients silently lost a call the
OpenAI non-streaming path preserves. The strip was harmless when healing
was all-or-nothing but became data loss once healing turned span-exact.
Gate the legacy strip on whether healing promoted a call, matching the
OpenAI passthrough and the intent already stated in the comment above.
Add a route-level regression test for the mixed declared+undeclared case.
* inference: require fully declared nudge retries; keep unpromoted Anthropic text
Codex round 6, two findings:
- response_has_promotable_calls accepted a nudge retry when any one
structured call named a declared tool, so a mixed retry (hallucinated
undeclared call plus a declared one) replaced the original and the
caller forwarded the undeclared call, or with parallel_tool_calls
false could keep only it. All structured retry calls must be declared.
- The Anthropic non-streaming builder still ran the legacy _TOOL_XML_RE
strip after span-exact healing, deleting undeclared or malformed call
text that healing deliberately preserved. The legacy strip now runs
only when healing is off (no declared tools, or opted out), matching
the OpenAI passthrough.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: keep unpromoted Anthropic text whenever healing is active
The previous commit skipped the legacy strip only when a call was
actually promoted, so an undeclared-only (or malformed-only) response
was still silently emptied: exactly the dead-turn shape this path
exists to fix, and inconsistent with the OpenAI passthrough, which
relays those bytes verbatim. Gate the strip on healing being active
instead; opt-out and no-tools requests keep the legacy strip.
* Fix schema-aware tool healing for PR #6801
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix passthrough healing ordering for PR #6801
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix stream finish ordering for PR #6801
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: wasimysaid <112766706+wasimysaid@users.noreply.github.com>
* Fix Gemma 4 GGUF OpenAI API streams
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Avoid duplicate Responses stream disconnect watcher
* Keep reasoning-only Responses output hidden
* Address Gemma stream review comments
* Avoid Responses stream task-group cleanup
* Harden OpenAI chat completion streams
* Address OpenAI stream review issues
* Clean up Studio OpenAI stream helpers
* Fix Studio passthrough cold stream timeout
* Fix tool parser compatibility exports lint
* Preserve audio stream disconnect cancellation
* Avoid synthetic finish after passthrough errors
* Address stream cleanup and Gemma parser reviews
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gemma 4: parse bare-string tool args and keep safetensors tools for native <|tool_call>
- Quote bare unquoted string values in Gemma native tool-call args (e.g.
{location:Tokyo,unit:celsius}) so they parse; JSON scalars stay typed.
- Stop _detect_safetensors_features from suppressing supports_tools for
templates that emit Gemma native <|tool_call>, which the shared parser
now reads.
- Add tests for both.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden Gemma tool-call parsing and stream-error detection
Address three issues in the Gemma-native tool-call path:
- _quote_gemma_object_keys stopped a bare (unquoted) string value at the
first comma, so an argument like `location:New York, NY` was split
mid-value and the synthesized JSON failed to parse, dropping the whole
tool call. A bare value now ends only at `}` or a comma that begins the
next `key:` pair.
- parse_tool_calls_from_text scanned the entire response for Gemma markers
even inside a tool call already parsed from a `<tool_call>{...}` JSON
block, so a marker-like string inside an argument (data) was promoted to
a second, unintended tool call. Matches inside an already-consumed call
span are now skipped.
- _openai_passthrough_stream relied on _monitor_openai_sse_line to flag a
stream error, which returns early when monitor_id is None
(skip_api_monitor), so an upstream error chunk left saw_stream_error
unset and the synthetic-finish guard emitted a successful finish_reason
after a failed stream. Error chunks are now detected independently of API
monitoring.
Adds tests/test_gemma_tool_parse_edge_cases.py covering the comma and
marker-injection cases.
* Emit the terminal finish_reason chunk in GGUF streams
The OpenAI chat-completions GGUF tool stream and plain stream both built a
final ChatCompletionChunk carrying finish_reason but never yielded it, so
clients received the optional usage chunk and [DONE] with no chunk carrying
finish_reason. OpenAI-compatible consumers rely on that terminal choice to
distinguish stop/length/tool_calls. Yield it before the usage chunk and
[DONE], matching the other streaming paths.
* Parse tool calls in document order and skip nested markers both ways
Unify the JSON- and Gemma-format tool-call passes into a single
position-ordered scan:
- Calls are now emitted in byte order across both formats, so a mixed
output like `<|tool_call>call:create{...}<tool_call|> ... <tool_call>
{"name":"read",...}</tool_call>` executes create before read, matching
the order they appear in (tools run in returned order).
- A candidate that starts inside an already-accepted call's span is
skipped, in both directions: a JSON marker inside a Gemma argument and a
Gemma marker inside a JSON argument are treated as data, not promoted to
a second executable tool call.
Extends tests/test_gemma_tool_parse_edge_cases.py with the ordering and
JSON-in-Gemma nesting cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Quote bare Gemma array elements; order finish before trailing usage
- _quote_gemma_object_keys skipped array values, so a Gemma call with a
bare-string array argument like labels:[bug,ui] produced invalid JSON and
the whole tool call was dropped. Array values are now scanned and bare
string elements quoted, while numbers, quoted strings, and JSON literals
are preserved.
- In the OpenAI passthrough stream, a trailing usage-only chunk
(stream_options.include_usage) that arrived before any finish chunk was
relayed before the synthetic finish, producing usage -> finish -> [DONE].
Emit the synthetic finish before that usage chunk so the order matches the
other streams (finish -> usage -> [DONE]).
Extends tests/test_gemma_tool_parse_edge_cases.py with the bare-array cases.
* Harden Gemma array parsing, XML-parameter guard, and stream teardown
Address five review findings on the Gemma tool-call and OpenAI passthrough
streaming paths:
- parse_tool_calls_from_text collected JSON and Gemma markers without the
_inside_open_parameter guard, so a marker embedded in an existing
<function=...><parameter=...> value was promoted to a separate tool call.
Candidates that start inside an open XML parameter are now skipped, matching
the guard the XML-style parser already applies.
- _quote_gemma_array_elements preserved array elements starting with { or [
verbatim, so an array of objects (items:[{path:a}]) or a nested array failed
json.loads and the whole call was dropped. Object and nested-array elements
are now normalised recursively.
- _openai_passthrough_stream synthesized a finish chunk before a trailing
usage-only chunk and set saw_finish_reason, which made the EOF guard skip the
[DONE] sentinel. The EOF path now emits [DONE] whenever the upstream omitted
it, even after a finish chunk was already synthesized.
- /generate/stream drove generation through asyncio.to_thread with no
disconnect watcher, so a client disconnect during a long generation went
unnoticed until the next send. It now runs _await_disconnect_then_cancel
against the request, matching the other local streaming endpoints.
- _SameTaskStreamingResponse closed the body iterator with aclose() on a
send-side disconnect, raising GeneratorExit so the generators' cancellation
handlers (which finish the api_monitor entry) never ran. It now throws
CancelledError, falling back to aclose() when athrow is unavailable.
Extends tests/test_gemma_tool_parse_edge_cases.py with array-of-objects,
nested-array, and marker-inside-XML-parameter cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Watch disconnects on Anthropic streams; keep timestamps in Gemma values
Two follow-ups on the streaming and tool-parse paths:
- _anthropic_tool_stream and _anthropic_plain_stream drove generation through
asyncio.to_thread(next, gen, ...) and only polled is_disconnected() between
events, so a client disconnect during prefill or a long generation/tool step
held the decode slot until the next event or a failed send. Both now run the
_await_disconnect_then_cancel watcher used by the other local streams, stop it
in finally, and break promptly when cancel_event is set.
- _GEMMA_NEXT_KEY_RE treated any comma followed by word-chars-then-colon as the
next key, so a bare value such as "meet at 10:00, 11:00 tomorrow" was split
into bogus keys. The next-key token must now be identifier-shaped (start with
a letter or underscore), so a comma before a timestamp, ratio, or other
numeric-then-colon text stays part of the value.
Adds a timestamp-in-bare-value regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard nested markers, reset on disconnect, clean unstarted streams
Three follow-ups on the tool-parse and streaming paths:
- parse_tool_calls_from_text only skipped markers that fell inside a span it
had already parsed successfully, so when an unquoted Gemma argument contained
a literal marker (code:<|tool_call>call:terminal{...}<tool_call|>) the outer
object failed to normalize, its span was never recorded, and the inner marker
was promoted to a standalone terminal call. Candidates nested inside any other
candidate's brace span are now skipped regardless of whether the enclosing
candidate parsed, so a marker in malformed outer data is never executed.
- /generate/stream skipped backend.reset_generation_state() when the disconnect
watcher set cancel_event between chunks: the loop broke and the finally's reset
is guarded on cancel_event being unset. A subprocess backend kept decoding
after the client left. The cancel-break path now resets the backend.
- _SameTaskStreamingResponse threw CancelledError / called aclose() on the body
iterator on a send-side disconnect, but neither runs the try/finally of a
generator that never started (early disconnect on http.response.start), so the
passthrough's eagerly-opened upstream httpx stream and cancel-registry entry
leaked. It now tracks whether the body started and, when it did not, runs an
optional unstarted_cleanup hook; the OpenAI passthrough wires it to close the
upstream resp/client and exit the cancel tracker.
Adds a nested-unquoted-marker regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Recognise the Gemma 4 separate-drafter MTP family, auto-download the drafter with retry, fall back to n-gram with a clear reason when it cannot be resolved, and retry the download on reload. Gemma 3n (ships no drafter) and embedded-MTP models (Qwen) are unaffected.
Fixes#6406
* Studio: Auto disables MTP for MLA models (GLM-5.2 et al.); UNSLOTH_MLA_MTP_ENABLED to re-enable
Studio's Auto speculative mode promotes any embedded-MTP model >=3B to
--spec-type draft-mtp. For MLA models (GLM-5.2/DeepSeek/Kimi) that is a
regression: llama.cpp's MLA/DSA MTP path keeps a duplicated full target-KV
context and recomputes the sparse-attention indexer every draft step, so it
runs ~2x slower than no speculation (GLM-5.2 UD-IQ1_S bench: 27 vs 45 tok/s,
flat across draft depth 1..6 and 96-100% acceptance, on both prose and code).
vLLM/SGLang get a speedup from the same model, so this is a llama.cpp
implementation gap, not a model property.
Auto now drops embedded MTP for MLA models and falls back to ngram-mod (or
spec-off when the binary lacks ngram-mod), mirroring the existing sub-3B
fallback. The metadata separator is kv_lora_rank: it is present on MLA models
and absent on non-MLA embedded-MTP models (Qwen3.x-MTP), whose MTP module is
structurally identical but fast, so a "full layer" heuristic cannot tell them
apart. Qwen MTP, separate drafters (Gemma, --model-draft), and non-MTP models
are unchanged.
Explicit overrides still engage the slower MTP route: choosing MTP / MTP+Ngram
in Settings, or passing --spec-type in extra args. UNSLOTH_MLA_MTP_ENABLED=1
re-enables Auto promotion for MLA once the upstream path is optimized.
A new spec_fallback_reason value "mla_mtp_disabled" surfaces this as an
Auto-mode policy downgrade (not a binary/update problem), with a settings
banner that points users at the MTP override. It is deliberately kept out of
the "Update llama.cpp" affordance since updating does not help.
Tests: resolver-matrix rows for MLA->ngram-mod / MLA-no-ngram->off /
non-MLA-Qwen->draft-mtp / MLA-separate-drafter->draft-mtp /
non-MTP-MLA->default / forced mtp|mtp+ngram on MLA->draft-mtp / env flag;
kv_lora_rank metadata fixtures; and reload-skip coverage (Auto ngram-mod is
idempotent, forced mtp bounces a reload).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: fix Bypass Permissions menu freeze and show decimal GB for model sizes
Bypass Permissions freeze: the warning dialog lived inside the composer
"+"/More dropdown and kept the menu mounted via onSelect preventDefault,
so confirming or cancelling the dialog left both popovers frozen open.
Lift the dialog out of the menu into a store-driven
BypassPermissionsConfirmDialog mounted at a stable spot in the composer.
The menu item now closes normally on select and just toggles a new
bypassConfirmOpen store flag, so the popovers dismiss as expected.
Model search sizes: formatBytes divided bytes by 1024 but labelled the
result "GB", so unsloth/GLM-5.2-GGUF:UD-IQ1_S showed 201.8 GB where
Hugging Face reports 217 GB. Switch the search display to decimal
(base-1000) units to match what Hugging Face reports. The GPU-fit math
stays base-1024 since VRAM capacity is binary.
* Studio: address review feedback and add GLM-5.2 high/max/disabled thinking
Review feedback on the Bypass Permissions and size-format changes:
- Mount the Bypass Permissions warning dialog once at the chat-page root
instead of inside each Composer. It is driven by global store state, so
the per-composer mount meant Compare mode (multiple composers) rendered
duplicate dialogs and the shared-composer menu had none. A single root
mount fixes both.
- Defer opening the dialog past Radix's menu-close focus restoration with
setTimeout(0), so the dropdown does not steal focus back and break the
dialog's focus trap.
- Clamp the unit index in formatBytes so units[i] cannot go out of bounds
past TB (and to absorb log() float error at exact powers of 1000).
GLM-5.2 reasoning levels:
GLM-5.2's template gates thinking with enable_thinking and also reads a
reasoning_effort level ('high' or 'max'), so it needs high / max /
disabled rather than the binary toggle it got before (its style was
detected as enable_thinking, which made 'high' unreachable). Add a new
reasoning style 'enable_thinking_effort' that reuses the effort dropdown
but, unlike gpt-oss, can be fully disabled:
- detect_reasoning_flags classifies a template that has both
enable_thinking and reasoning_effort, extracting the discrete levels
from the quoted effort literals it branches on. Templates with only one
of the two (gpt-oss, Qwen3, DeepSeek, GLM-4.6) are unchanged.
- _request_reasoning_kwargs maps the new style to enable_thinking plus an
in-range reasoning_effort; disabling sends enable_thinking=false. The
gpt-oss reasoning_effort path is left untouched.
- The backend reports reasoning_effort_levels on the load/status response;
the frontend carries them through to the effort dropdown and sends
enable_thinking + reasoning_effort for this style.
Verified: backend reasoning kwargs render the real GLM-5.2 template to
"Reasoning Effort: High/Max" (thinking) and an empty <think></think>
(disabled); tsc, eslint, i18n parity and the production build all pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address review feedback on reasoning effort and formatBytes
- chat-adapter localReasoningEffort: accept 'minimal' so a template that
branches on it (extracted into reasoning_effort_levels) is sent through
instead of being coerced to 'low' and then dropped by the backend.
- formatBytes: return '0 B' for non-finite / non-positive sizes (missing
metadata -> NaN, Infinity, negatives) and clamp the unit index lower
bound to 0, so sub-1-byte values can't produce a negative index.
* Studio: hybrid reasoning none gate and decimal GB in load progress
- _request_reasoning_kwargs: for enable_thinking_effort models, treat a
raw reasoning_effort='none' (OpenAI 'no reasoning' sentinel) as the
enable_thinking=false off gate, so a direct API caller can disable
thinking even without passing enable_thinking. The frontend already
sends enable_thinking=false; this only affects raw API callers.
- use-chat-model-runtime: the download / 'X of Y GB in memory' load
progress divided bytes by 1024**3 but labelled GB, so it disagreed with
the model picker and Hugging Face. Use decimal GB (1e9) to match.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: carry hybrid reasoning levels on all load paths and harden formatBytes
Review follow-ups on the enable_thinking_effort work:
- Every model-load path now copies reasoning_effort_levels and derives
supportsReasoningOff, via a shared reasoningCapsFromLoad() helper. The
shared/Compare composer load and the three chat-adapter auto-load paths
previously set only reasoningStyle, so a GLM-style hybrid model loaded
through Compare or first-chat auto-load fell back to the default
low|medium|high and lost its Max / Off controls.
- The local send path clamps the effort to the loaded model's advertised
levels (clampReasoningEffortToLevels) instead of a hard-coded list. A
stale "max" carried over from an external provider no longer reaches a
pure reasoning_effort (gpt-oss) model that only accepts none|low|medium|
high, where the backend would have dropped it.
- formatBytes divides iteratively instead of via Math.log, which has float
error at exact powers of 1000 (log(1e12)/log(1000) = 3.9999... would
label 1 TB as "1000 GB"). Keeps the non-finite/non-positive guard.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: free chat model VRAM at training start only when the GPU is tight
The training start route unconditionally tore down the transformers/MLX
inference subprocess before training, and never stopped the llama.cpp GGUF
server at all, so a loaded GGUF chat model kept holding VRAM for the whole
run. Conversely the HF model was always unloaded even when there was plenty
of room to keep it.
Make the unload VRAM aware and cover every inference backend:
- Add routes/training_vram.py with summarize_resident_chat(),
can_keep_chat_during_training() and free_chat_models_for_training(). The
keep/unload decision reuses the same estimator and live per device free
VRAM reader the training GPU selection already uses (auto_select_gpu_ids,
estimate_required_model_memory_gb, get_visible_gpu_utilization), so the
probe agrees with the placement computed later in start_training.
- When a chat model is resident and training fits alongside it with a
conservative margin (required_gb * 1.15 + 4 GB), keep it loaded so the
user can train and chat at the same time; on a multi GPU box training
lands on a different GPU and both coexist. Otherwise unload the HF/MLX
orchestrator and the llama.cpp GGUF server before training starts.
- The export subprocess shutdown stays unconditional and now runs first so
its freed VRAM is reflected in the decision.
Default deny: non CUDA backends, unestimable models, or any probe error
fall back to the previous always unload behavior.
Adds tests/test_training_vram_coexistence.py and updates two existing route
tests in test_gpu_selection.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: per-GPU floor for explicit GPU lists + don't unload chat on invalid gpu_ids
Address review feedback on the chat coexistence probe:
- Explicit gpu_ids mode now enforces a per-GPU floor in addition to the
aggregate free-VRAM check, mirroring auto_select_gpu_ids' min_per_gpu_N.
Without it, an uneven split such as free [45, 10] for a 40 GB job passed
the aggregate threshold and kept chat loaded even though the 10 GB GPU
could not hold its training shard, risking an OOM.
- Invalid explicit gpu_ids (ids outside the visible set, or a UUID/MIG
mask) make resolve_requested_gpu_ids raise. That request is rejected with
a 400 before training starts, so leave the resident chat model untouched
instead of unloading it.
- Tighten the target_modules / gpu_ids type hints to List[str] / List[int].
Adds tests for the per-GPU floor (uneven split unloads, even split keeps)
and for invalid gpu_ids keeping the chat model loaded.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: only free chat VRAM once training will start; handle in-flight and CPU-only chat
Address the second review pass on the chat-coexistence path:
- Run the chat/export VRAM teardown as a before_spawn hook inside
TrainingBackend.start_training, fired only after the start guards pass.
Previously the route freed chat VRAM before calling start_training, so a
refused start (e.g. a lingering pump thread) would tear down the resident
chat model even though no training job began.
- Treat an in-flight HF chat load (loading_models set, no active model yet)
as not safely sizeable: free it rather than risk both OOMing as the load
keeps allocating after training starts.
- Do not count or tear down a GGUF llama-server confirmed to run entirely on
CPU (_gpu_offload_active is False): it holds no VRAM, so killing it cannot
help training fit.
Adds tests for the before_spawn hook (runs on start, skipped when a
subprocess is alive or a pump thread will not die, survives a hook error),
the in-flight load flag, and the CPU-only GGUF exclusion in both the resident
summary and the unload path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: treat any in-flight chat load (HF swap / mid-start GGUF) as unsafe to keep
Tighten the in-flight detection in summarize_resident_chat so the keep check
never sizes a load that is still allocating:
- Flag loading on ANY non-empty loading_models, not only when active_model_name
is empty. load_model adds the new model to loading_models before clearing the
old active_model_name, so a replacement load during a swap was previously
sized as a normal resident and could OOM as the new model finishes loading.
- Flag a GGUF server that is active but not yet healthy (is_loaded False) as
in-flight: it is still mmaping/offloading layers, so its final VRAM footprint
is unknown.
Consolidates the signal into a single resident["loading"] flag; the route frees
the chat model whenever it is set. Adds tests for the replacement HF load and
the mid-start GGUF cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten comments in chat/training VRAM coexistence (comments only)
* Studio: run before_spawn VRAM hook only after GPU-selection validation
Reviewers found the before_spawn hook fired before prepare_gpu_selection
validated gpu_ids (and before config build), so a refused start (invalid
gpu_ids -> 400, or a bad grad-clip value) could still tear down chat/export
VRAM. Move the hook to immediately before proc.start(), once all synchronous
validation and process construction have passed. This also fixes the route's
in-flight-chat loading branch, since that teardown runs inside the same hook.
Add test_hook_skipped_when_gpu_selection_rejects.
* Studio: recompute GPU auto-selection after the before_spawn VRAM hook
Codex P2: with before_spawn moved after prepare_gpu_selection, placement was
frozen against the pre-teardown VRAM state while the hook freed export/chat
afterward. Auto-selection could pin training onto a GPU the hook then cleared
(or onto a kept chat model). Split validation from placement: explicit gpu_ids
are still validated before the hook (raise -> 400, no teardown; explicit
placement is VRAM-independent), but VRAM-dependent auto-selection now runs
after the hook so it sees the freed memory.
Add test_auto_placement_runs_after_hook and test_explicit_placement_validated_before_hook.
* Studio: allow chatting during training (lift sidebar gate + VRAM-aware load guard) (#6335)
* Studio: allow chatting during training (lift sidebar gate + VRAM-aware load guard)
The sidebar disabled New Chat, project, and home navigation while a training
run was active, so users could not chat during training even though the backend
serves inference fine alongside a run. This removes that gate and adds a backend
guard so the one genuinely risky operation, loading a new local chat model
mid-training, is refused with a clear 409 when it would not fit beside the run.
Frontend (app-sidebar.tsx): drop the chatDisabled = isTrainingRunning gate and
its consumers. Navigation triggers no model load on its own, so chat stays
usable during training.
Backend (routes/training_vram.py, routes/inference.py): add
can_load_chat_during_training plus a load/validate guard that sizes the same
effective load the loader performs (LoRA 4-bit to 16-bit resolved first, HF auto
placement via auto_select_gpu_ids, explicit multi-GPU per-GPU floor, GGUF sized
from on-disk shards and companions or the selected remote variant). It is a
no-op when training is inactive, never blocks external providers or
already-resident models, and default-denies only on a CUDA sizing failure so a
load can never OOM the run. Validate refuses early with the real settings so the
frontend does not unload the resident chat model for a load that would be
rejected.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address review feedback for chat-during-training load guard
- Run the load/validate VRAM guard via asyncio.to_thread so the sync
nvidia-smi + HF metadata work never blocks the event loop.
- Size the GGUF KV cache at the requested context (_estimate_gguf_kv_gb)
and add it to the local GGUF estimate so large-context picks are not
under-counted.
- Keep the requested quantization when adapter_config.json is malformed
(not a JSON object) instead of raising in _effective_load_in_4bit.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: size the training load guard at the launcher's effective GGUF context
The GGUF KV-cache estimate used max_seq_length only, but the llama.cpp
launcher honors a user --ctx-size/-c in llama_extra_args. A load such as
max_seq_length=4096 with --ctx-size 131072 was sized against a 4k cache
while the server allocates 131k, so the guard could approve a long-context
GGUF load that then OOMs training. Size the guard's KV at the larger of
max_seq_length and the parsed --ctx-size (reusing the launcher's own
parse_ctx_override), keeping the conservative f16 cache so the estimate is
never smaller than what the server allocates.
The chat model picker also validated with the raw max_seq_length while
/load sizes with resolveLoadMaxSeqLength, so validate could pass, unload
the current model, then have /load reject the native-context load. Validate
now uses the same effective context; the load path is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: size the GGUF training guard at the server parallel-slot count
The KV-cache estimate assumed a single slot, but llama-server allocates the
cache across --parallel slots (app.state.llama_parallel_slots). On a Studio
launched with --parallel N>1 the guard under-sized the cache N-fold and could
approve a GGUF chat load that then OOMs training. Thread the same slot count
the loader uses into the guard's KV estimate; default 1 leaves single-slot
setups unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments for chat-during-training guard
* Studio: keep chat generation alive across navigation; Train spinner + Return to Chat
Hoist the base chat runtime above the routed outlet so navigating to Train (or any tab) no longer aborts an in-flight generation; only an explicit Stop cancels. Add a Train sidebar spinner and swap New Chat to Return to Chat while a run is active, with a lightweight completion watch so the spinner clears from any tab. Also respawn a chat llama-server killed mid-session and guard unreadable HF cache dirs that 500'd the hub model list.
* Studio: show Return to Chat on the Train tab whenever a chat is live
Previously the top sidebar item only swapped to Return to Chat while training was running; on the Train tab with an idle/just-finished run it stayed New Chat, which started a fresh thread and cancelled an in-flight generation. Show Return to Chat (and navigate back, preserving the run) whenever a generation is running or its thread is still active, or training is in progress.
* Studio: keep a running chat alive when starting a New Chat
Starting a New Chat (or switching threads) while a generation was in flight
remounted the single-chat runtime provider, which detached the in-flight run
and cut the previous chat off (it showed up frozen / empty when reopened).
Key the single-chat view by project instead of by thread or new-chat nonce so
the provider stays mounted and assistant-ui switches to a fresh thread in place.
The previous generation keeps streaming in the background and autosaves on
completion, and returning to that thread reattaches the live run instead of
reloading a half-saved one.
Also:
- "Return to Chat" now lands on the thread that is still generating rather than
the empty new chat that became active after New Chat.
- Skip the explicit /inference/cancel POST when an abort comes from a runtime
detach (navigation / background switch) rather than an explicit Stop, so a
backgrounded generation is never cancelled behind the scenes.
* Studio: make model export non-blocking and inline
The Export tab opened a full-screen modal that trapped focus, could not be
closed or cancelled while running, and showed no progress. It also stopped
training and unloaded the chat model before loading, so export could not run
alongside them.
Export now mirrors the training runtime pattern:
- Inline panel embedded where the Export Model button was, with no modal or
backdrop, so the rest of the UI stays usable during an export.
- Global export runtime store plus an app-root lifecycle hook, so a run keeps
going and streaming across navigation and is reflected on the Export nav item
from any tab.
- The worker log stream now stays connected across the load to export phase
boundary instead of stranding on "Waiting for worker output".
- Progress bar driven by phase and quant index (quant N of M for GGUF), with
elapsed time and a working Cancel.
- load-checkpoint no longer stops training or unloads inference; export loads in
its own subprocess in parallel and surfaces out-of-memory as a clear error.
- Add POST /api/export/cancel and is_export_active on /api/export/status.
* Studio: show Return to Chat on the Export tab too
Extend the New Chat to Return to Chat swap to the Export route so leaving a
running chat for Export offers a way back to the live generation, matching the
Train tab.
* Studio: smooth out Export animations and polish the panel
- Drop the height-based reveal animations (source switch, run panel, quant
picker, hub fields) that caused flashing and reflow; use instant swaps and
quick opacity fades instead.
- Method and quant cards now transition colors only, with no transition-all or
hover lift, so selecting a method or quant is crisp instead of jumpy.
- Auto-scroll the export panel into view when it opens and add a scroll-to-bottom
button when its output is below the fold, like Chat.
- Show Return to Chat on the Export tab while an export is running, matching how
training drives it on the Train tab.
- Surface the current phase or stage in the live output before the first worker
line arrives so the panel never looks stuck while progress is advancing.
* Studio: show Return to Chat on every non-chat tab
Generalize the Return to Chat swap from just Train/Export to any non-chat route
(Recipes, Projects, Hub, ...) so a running or active chat is always one click
away, instead of showing New Chat there.
* Studio: stream export logs over the Cloudflare tunnel; drop janky export animations
Exporting over a --secure Cloudflare quick tunnel showed "connecting..." with no
logs while the progress bar advanced. Cloudflare buffers text/event-stream and
only flushes when the stream closes, so the SSE log stream never reached the
browser during the run (direct localhost is unaffected, which is why this only
showed up over the tunnel).
Add a tunnel-safe JSON poll fallback (GET /api/export/logs?since=) that the
runtime lifecycle hook polls while a run is active. Short JSON responses are not
buffered by the proxy, so logs show up in near real time over the tunnel. It
shares the orchestrator's monotonic seq cursor with the SSE stream and the store
de-dupes by seq, so the two transports run together (SSE on localhost, poll over
the tunnel) without double-printing. A successful poll marks the panel
"streaming" instead of leaving it stuck on "connecting...".
Also remove the framer-motion AnimatePresence reveals from the export config and
run panel (quant picker, hub fields, the inline run panel, and the live log
section). The expand/slide animations flashed and felt clunky; the sections now
render in place.
* Studio: recover export over the Cloudflare tunnel when the blocking POST times out (524)
A model export over a --secure Cloudflare quick tunnel showed "Request failed
(524)" even though the export succeeded on the backend (the GGUF was written).
Cloudflare returns 524 when a single request takes longer than ~100s to respond,
and a GGUF conversion routinely runs for minutes, so the blocking per-method
export POST is cut off while the backend keeps going.
Confirm completion via short status polls instead of relying on the long POST
response (the same approach that fixed log streaming):
- The orchestrator records each finished op's outcome (status / output_path /
error) with a monotonic seq, exposed on GET /api/export/status.
- parseJson now preserves the HTTP status; a 524/520/522/523/502/503 or a
status-less network drop is classified as a recoverable transport error.
- runExport wraps each phase (load, every export method, each GGUF quant): on a
recoverable failure it keeps the run alive (logs keep streaming, the panel
shows "reconnecting...") and polls status until the still-running op finishes,
then settles from the recorded result, recovering the output path for the
success banner. A real 4xx still fails immediately; localhost still uses the
fast POST response. applyBackendStatus also settles a reloaded run from the
last-op record.
Verified over the tunnel: a 3m14s gemma-4-E4B-it GGUF export now ends on the
success banner with the output path instead of 524.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep the export method + logs visible after navigating away mid-export
While an export was running, navigating to another tab and back to Export
remounted the page and reset the local form state (exportMethod, quant levels),
so the method card showed unselected and the run panel's log area was hidden
until the card was re-clicked. The run itself lives in the global store and was
unaffected.
Seed exportMethod / quantLevels from the active run's summary via lazy useState
initializers on (re)mount, and gate the panel's log area on the live run
(isExporting / logLines / the run's method) rather than only the local form
selection. The card stays selected and the logs/progress stay visible across
navigation; nothing changes when no run is active.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: address export/training review findings
- Export: guard Start against an empty GGUF quant selection so an inline-panel
run with no quant can't settle as success with no file produced.
- Export: thread the source HF token into the background load so gated/private
HF source exports (and gated bases) authenticate, matching the consent path.
- Export: only settle a recovered (non-owned) run as a finished export when the
last backend op was an export, not a standalone load_checkpoint.
- Training: free the export subprocess whenever an export is active, not only
once a checkpoint is loaded, so an in-flight export load can't race training
for VRAM (current_checkpoint is unset during the load phase).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Harden model fetching: consent gate for trust_remote_code
Add a load-path consent gate that scans a model's auto_map repository code
before it executes and blocks CRITICAL/HIGH findings unless the user pins
approval of that exact code version. Capability detection stays code-free,
reading raw config.json instead of AutoConfig.
- Scan config.json and tokenizer_config.json auto_map, nested local helpers,
and external owner/name--module repos; fail closed on partial downloads.
- Gate inference, training, and export workers, including the MLX path and a
LoRA's base model, and report requires_trust_remote_code from the raw config
so chat and auto-load surface the dialog.
- Verify trusted-org auto-enable against the Hub with the request token and key
the verdict cache by token; reject local-path and spoofed names.
- Add a consent dialog showing the flagged file, line, and surrounding code.
- Thread hf_token through the scan and load paths for gated repos.
* Address review: token handling, tokenizer/LoRA scan coverage, rollback
- Send the HF token for remote-code scans in the POST body, not the URL, so it
never lands in a log or browser history.
- Collect tokenizer_config.json auto_map files directly instead of relying only
on the repo file listing.
- Resolve a LoRA's base model for the validate flag and the scan endpoint so the
dialog scans the code the workers actually gate.
- Pass the request token to the training YAML trusted-org auto-enable.
- Resend a previously approved fingerprint when rolling back to a custom-code
model after a failed switch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Consent UX: drop legacy chat toggle, fix decline copy, purge declined downloads
The per-model consent dialog is now the single approval path for custom
(auto_map) code in chat, so three leftovers from before it existed are removed:
- Remove the "Enable custom code" switch from Chat Settings and stop persisting
trust_remote_code, so a previously saved blanket-on cannot linger and load a
model without going through per-version review. The flag stays as an internal
YAML/preset default (e.g. first-party auto-enable); the load path still gates
every custom-code load on a fingerprint only the dialog produces.
- Reword the decline message and the auto-load toast to describe approving the
model's code from the dialog, not a missing settings toggle.
- On decline, purge the repo the scan downloaded so untrusted code is not left
on disk. A new /api/models/discard-remote-code endpoint deletes only a
metadata-only cache entry the scan created; it refuses local paths, loaded
models, and any repo with weight files cached, so a model the user already had
or pre-downloaded is always left untouched. The frontend only calls it when
the scan reported created_by_scan.
Adds discard-endpoint tests (delete metadata-only, refuse on weights/gguf,
refuse local, no-op when not cached) and a created_by_scan payload assertion.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Export: remove the user-facing trust remote code toggle
The Export page kept a "Trust remote code" switch (default on) next to the HF
token field. Like chat, custom (auto_map) code should be approved per model
through the load-time review dialog, not a persistent blanket switch, so the
toggle is removed. The export load path already routes through the same consent
dialog: an HF source now starts with trust_remote_code off and only enables it
when the user approves the scanned code in the dialog (a local checkpoint the
user exported stays trusted by default). With the dialog unreachable and no
approval, an HF source loads with trust_remote_code off, which fails closed
rather than running unreviewed code.
* Block loads of repos with unsafe files using Hugging Face's security scan
The trust_remote_code consent gate covers one load-time RCE vector (a repo's
auto_map Python). It does not cover the other: a malicious pickle inside a weight
file (pytorch_model.bin, *.pkl, *.dat) deserializes during from_pretrained even
with trust_remote_code False, so a repo with a normal config plus a poisoned
pickle slips past the existing gate.
Add a metadata-only malware gate that uses Hugging Face's own scan (picklescan +
ClamAV), read via model_info(securityStatus=True).security_repo_status. It never
downloads, opens, or unpickles the flagged files; it only reads the Hub's verdict
and surfaces the flagged file names. New evaluate_file_security runs
unconditionally (independent of trust_remote_code) in every load path (inference,
training SFT/MLX, export), blocking the load when a file is flagged
unsafe/suspicious/malicious. The /remote-code-scan preflight and the validate
endpoint also report the result so the consent dialog opens as a hard block (no
override) listing the flagged files, even for a repo with no custom code.
Policy: hard block with no user override; fail open when the scan is unavailable
(offline/unscanned) so legitimate loads are not broken; no first-party exemption
(a poisoned pickle in a compromised trusted repo still blocks); local paths and
GGUF are skipped (no Hub scan, non-pickle format). Blocking does not gate on
scansDone, since that is often false for clean repos and a file already flagged
unsafe is unsafe regardless.
Adds test_file_security.py covering the block/allow/fail-open/skip matrix.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: scan list-form tokenizer auto_map, gate unsafe files on all load paths
Fixes from a 10-reviewer pass on the model-fetching hardening:
- The remote-code scanner skipped tokenizer auto_map encoded as a [slow, fast]
list (transformers' standard tokenizer shape, e.g.
{"AutoTokenizer": ["owner/repo--tokenization_x.Slow", null]}). External
tokenizer code in that form was never fetched, scanned, or fingerprinted, so an
AutoTokenizer(trust_remote_code=True) load could run it. _auto_map_refs now
flattens string, list, and nested values. Adds a regression test.
- Compare-mode chat loads and background auto-load only gated on
requires_trust_remote_code, so a repo flagged unsafe by the Hub scan but with no
custom code skipped the hard-block dialog. Both now also gate on
requires_security_review, matching the main chat path.
- The /remote-code-scan and /validate routes collapsed a LoRA adapter to its base
before the malware scan, so unsafe files in the adapter repo itself were missed
in the pre-load review (the workers already scan both). Both routes now run the
file-security scan over the adapter and the base.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Require approval for all HIGH remote code, fail closed when unscannable
Tighten the load-time security gates based on review:
Consent gate
- HIGH-severity auto_map code now requires explicit, per-version approval for
every repo, including first-party unsloth/nvidia. The org is no longer a
blanket bypass: a compromised first-party repo with HIGH code still warrants
review. CRITICAL stays a hard block; clean code still loads after the consent
prompt.
- Fail closed when auto_map code is present but cannot be fully fetched or
listed to scan (gated, offline, transient, or a repo-listing failure that
could hide an imported helper). We cannot fingerprint code we cannot see, so
this is a non-approvable block, retryable once the repo is reachable.
- Scan auto_map from every config that can carry one (model, tokenizer, image
and feature processor, processor, video processor), not just config.json and
tokenizer_config.json, so a custom-processor model is not missed. The file
list is the single source of truth in remote_code_scan and is pinned to the
transformers filename constants by a guard test.
- Distinguish a genuine 404 (config truly absent) from a transient error: only
the latter forces a scan, so a repo with no config is correctly a no-op.
Malware gate
- Scan a remote repo even when its name ends in .gguf; only local paths skip the
Hub scan, so a repo cannot dodge the scan by naming itself "*.gguf".
- Correct the docstring: a file already flagged unsafe blocks regardless of
scansDone; the only fail-open path is an unavailable scan.
Coverage
- Resolve a remote LoRA adapter's base model (not just local directories) so the
base, where the code and weights actually execute, is scanned in validate,
the scan route, and the training and export workers.
- Gate the embedding training path (FastSentenceTransformer) with the malware
and consent checks, matching the other load paths.
Tests updated and added for each change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope malware gate to the load-path vector; stop false-blocking first-party models
Follow-up hardening from a second review pass + a broad live model matrix
(unsloth/* , nvidia/* , third-party, and the eicar malware repo).
Malware / unsafe-file gate
- Scope the block to the actual RCE vector: a root-level file in a code-executing
format. from_pretrained deserializes weight files at the repo ROOT, so a flag is
only a load-path pickle vector there. Two exclusions, because neither is loaded:
inert formats (safetensors is tensor-only, gguf is non-pickle, configs/text/
images) and files in subdirectories. This keeps eicar blocked (its *.pkl/*.dat/
eicar_test_file sit at the repo root) while no longer false-blocking legitimate
first-party repos: nvidia/Nemotron-H-8B-Base-8K ships root safetensors plus NeMo
pickle checkpoints under nemo/ that the loader never touches, and the Hub flags
both; the gate previously hard-blocked it.
- Unknown / future non-"safe" levels now fail closed (block) instead of being
silently allowed, so Hub schema drift cannot introduce a bypass; in-progress
("pending"/"scanning"/"error") levels stay non-blocking to avoid false blocks.
Consent gate
- Ignore a STALE own-repo auto_map target that is absent from the repo listing (an
older config pointing at a file the repo no longer ships) instead of failing the
whole repo closed as unscannable. The present .py are still fully scanned, which
is the stronger coverage, and a file that is not there cannot execute. This
unblocks first-party models like unsloth/PaddleOCR-VL (its tokenizer_config.json
names processing_ppocrvl.py while the repo ships processing_paddleocr_vl.py). A
referenced .py that IS present but cannot be fetched, and a repo-listing failure,
still fail closed.
Remote LoRA base resolution
- Distinguish a genuine 404 (not a LoRA / repo absent -> None) from a transient
error: the transient case is retried once, then logged as a WARNING (a missed
base is scanned by neither gate) rather than silently skipped.
Discard endpoint
- Treat .onnx and .ckpt as weights so a repo whose only heavy artifact is one of
those is never eligible for the declined-download purge.
Tests added for each: load-path scoping (safetensors/subdir/Nemotron-H shapes,
unknown-level fail-closed, pending non-block), stale own-repo auto_map ref, remote
LoRA transient retry, and the empty-config-list (all-404 -> []) semantics.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make LoRA-base transient-warning test robust to logging backend
Assert on the logger object directly instead of capsys, so the test does not
depend on whether the real structlog logger or the module-stub logger is active
(which varies with test collection order).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Allow a repo with auto_map but no executable code (e.g. GGUF) instead of blocking
A config can declare an auto_map yet the repo ship NO executable .py -- most
commonly a GGUF repo whose config.json carries an auto_map copied from the original
model (e.g. unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF references
modeling_decilm.py, which the GGUF-only repo does not contain). A GGUF model loads
through llama.cpp, which never executes auto_map, and transformers cannot run a file
that is not present, so there is nothing to scan and trust_remote_code is a no-op.
The fail-closed change treated this empty result the same as "code is present but we
could not fetch it" and hard-blocked the load. Distinguish the two: repo_remote_code_files
now RAISES RemoteCodeUnscannable when code is present but cannot be fully fetched or
listed (offline / gated / transient / a present .py that 404s / a listing failure),
and returns an empty dict only when the listing succeeded and the repo genuinely ships
no executable .py. The consent gate blocks on the exception (fail closed) and allows the
empty case as a no-op. Real unscannable code still hard-blocks; eicar and CRITICAL/HIGH
custom code are unaffected.
Verified against all 37 unsloth/*Nemotron* models (two GGUF repos were false-blocked,
now load) and the existing matrix (eicar still blocks; DeepSeek-OCR / NVLM-D-72B still
prompt approvable consent). Tests updated to expect the raise for unscannable cases and
added for the no-executable-code no-op.
* Ignore vestigial auto_map in GGUF repos (llama.cpp never runs it)
A GGUF repo's config.json is often copied verbatim from the original
transformers model, auto_map and all, but a GGUF load goes through
llama.cpp which never executes auto_map, so the config is inert. Treat
a direct .gguf reference, and a repo that ships .gguf weights with no
.safetensors, as having no remote code so the consent flow is never
triggered. A mixed repo with both .gguf and .safetensors is still gated,
since the safetensors variant would load through transformers where
auto_map does run. The check sits behind the existing auto_map-present
gate so normal models pay no extra repo listing.
* Add scanner-result copy to the remote-code consent dialog
Make the consent dialog state the scan outcome in plain language for
every model. When the static scan finds nothing, reassure the user with
'Our automatic scanner did not flag any worrying files, but please
double check.' (shown only for the clean, approvable case). When the
scan flags custom code or unsafe files, label the list with 'Our
automatic scanner flagged issues including:'. The Hugging Face
attribution for unsafe files stays in the dialog description.
* Close GGUF-suffix consent bypass for repo ids ending in .gguf
The .gguf short-circuit in _config_has_auto_map skipped the scan for any
model name ending in .gguf, including a bare two-segment repo id like
'evil/model.gguf'. Such a repo can still ship safetensors plus auto_map
Python that transformers would execute, so skipping the scan was an
asymmetric bypass (file_security already scans those repos). Restrict the
short-circuit to genuine direct GGUF file references via
_is_direct_gguf_file_ref: a local .gguf path, or a remote repo_id plus
filename (three or more segments). A two-segment repo id named *.gguf now
falls through to the config scan and _is_gguf_repo file inspection, so it
only skips consent when it actually ships .gguf weights and no safetensors.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Align consent dialog body with the title and fix narrow-width overflow
The scan results (the 'Our automatic scanner...' label, finding/unsafe
cards, and the clean-scan reassurance) sat at the dialog's left padding
while the title and description were indented past the status icon, so
the body did not line up under the description. Move the title,
description and results into one column to the right of the icon so they
share a left edge, and let that column fill its width so the description
no longer wraps early.
Also stop a wide code snippet from pushing the dialog off-screen on
narrow viewports: AlertDialogHeader is a grid with place-items-center,
which sized the content row to its content; give the row w-full so it
fills the track, and add min-w-0 down the results chain so the snippet
scrolls inside its card instead of widening the dialog. Verified aligned
and contained from mobile portrait through ultrawide.
* Treat a repo as GGUF-only only when it ships no transformers weights
_is_gguf_repo excluded only .safetensors, so a repo with a .gguf and a
pytorch_model.bin (or .pt/.pth/.h5/.msgpack/.onnx/.ckpt) and no
safetensors was treated as GGUF-only and skipped the consent scan, even
though transformers can load that weight set and execute the repo's
auto_map code. Require the absence of ANY transformers-loadable weight
before treating the repo as a llama.cpp-only GGUF load. A genuine
GGUF-only repo (only .gguf) is still inert; a mixed repo with any pickle
or safetensors weight is gated. Adds a regression test across all the
non-safetensors weight formats.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Block flagged subdir weight shards referenced by a root index
The malware gate treated every subdirectory file as non-loadable, but
from_pretrained deserializes a subdir shard a root index references
(pytorch_model.bin.index.json -> shards/...-00001-of-00002.bin). Read the
root weight indexes and block a flagged subdir pickle the weight_map
points at; a flagged subdir pickle no index lists (NeMo nemo/*.distcp)
stays non-blocking, and an inconclusive index lookup fails closed.
* Pass hf_token to the export checkpoint load
ExportBackend.load_checkpoint scanned with hf_token in the worker but
loaded the weights unauthenticated, so a gated/private checkpoint passed
preflight then 401'd at from_pretrained. Add hf_token to load_checkpoint
and forward token to every from_pretrained branch; the worker passes the
command's hf_token.
* Scope created_by_scan to every HF cache the discard searches
created_by_scan used get_cache_path (active HF_HUB_CACHE only) while
/discard-remote-code deletes across active, legacy, and default caches. A
repo the user already had in a legacy/default cache was marked
scan-created and deleted on decline. Check all three caches for the repo
dir before declaring the scan created it.
* Scan the full .py closure of external auto_map repos
An auto_map cross-repo ref (owner/name--module.Class) only had its entry
file downloaded, but transformers also fetches that file's relative
imports from the same repo, so a dangerous helper.py was left outside the
scanned fingerprint. List each external repo's .py and scan the whole set
(plus the referenced entry files); fail closed if the repo cannot be
listed or fetched.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fail closed when a weight index cannot be fully read
_indexed_shard_paths treated a partial result as definitive: if one weight
index read cleanly but another failed transiently, it returned the shard
paths it did see. A flagged subdirectory pickle listed only by the index we
could not read would then be classed as "not a load input" and skipped,
re-opening the very fail-open this guard was added to close.
Return None whenever any index read is inconclusive, even if another read
cleanly, so the caller blocks the already-flagged subdir pickle. A repo that
ships no index files raises EntryNotFoundError for each (never inconclusive)
and still returns an empty set.
* Match cached repos case-insensitively in the created_by_scan guard
_repo_in_any_hf_cache resolved casing only against the active cache and then
probed every cache with an exact directory name. A case-variant already
present in a legacy or default cache (models--Unsloth--Foo for a scan of
unsloth/foo) was missed, so the repo was marked created_by_scan and deleted
on decline -- but discard_remote_code_download deletes case-insensitively,
so that delete would hit the user's pre-existing cache entry. Detect
case-insensitively too, mirroring the deletion path.
* Skip remote-code and security review for selected GGUF variants
validate_model ran the trust_remote_code and Hugging Face security-scan
preflight against the repo even when the selected artifact is a .gguf. A
GGUF loads through llama.cpp, which never executes the repo's auto_map
Python and never deserializes root pickle weights, so repo-level Transformers
artifacts (a config.json with auto_map, or an unsafe pytorch_model.bin next
to the .gguf in a mixed repo) are inert for that load. Gating the GGUF on
them is a false positive. Run both preflights only for non-GGUF loads.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope the malware gate to actual load roots and serialized files
Two fixes to evaluate_file_security so it neither misses a load-path pickle nor
false-blocks an inert file:
- Honor subdirectory load roots. Spark-TTS / BiCodec call from_pretrained on the
snapshot's LLM subdirectory, so a flagged pickle directly under it is a
root-level load artifact there. A new load_subdirs parameter (set from the
model's audio type via security_load_subdirs) reclassifies those files relative
to the load root and looks for weight indexes under it, so a flagged shard in
that subdir is no longer skipped as "not root-level".
- Exempt source files. A root .py is never deserialized by from_pretrained;
executable repo code runs only through auto_map, which the remote-code consent
gate scans. Flagging a Python helper here would false-block a repo that merely
ships a build or train script.
* Scan a LoRA adapter and base as one consent unit, and gate MEDIUM code
A LoRA load runs both the adapter's and the base's repo code. The consent gate
scanned them separately and pinned one fingerprint per repo, so an adapter that
shipped its own auto_map code was either never shown in the dialog (which only
saw the base) or impossible to approve with the base's fingerprint.
evaluate_remote_code_consent_for_targets now scans all of a load's repos as a
single combined unit and pins ONE fingerprint over the union of their code, so
approving the load approves every repo's code together. evaluate_remote_code_consent
becomes a thin single-target wrapper, and an unscannable target fails the whole
load closed.
Also gate MEDIUM findings: like HIGH they now block pending pinned approval, so a
direct API caller cannot run flagged code by setting trust_remote_code=True
without consenting. Only a clean scan loads without a fingerprint.
* Preflight a LoRA load's adapter and base as one combined consent scan
scan_model_remote_code rewrote a LoRA adapter to its base and scanned only the
base for remote code, so the dialog never surfaced an adapter's own auto_map
code. Scan the adapter and base together through
preflight_remote_code_consent_for_targets, which pins one combined fingerprint
the worker gate accepts. The malware preflight is also scoped to each target's
load subdirectories.
* Apply combined consent and subdir-aware malware scan in load workers
Each load worker (inference, export, training) evaluated remote-code consent
once per target with a single shared fingerprint, so a LoRA adapter that ships
its own auto_map code could not be approved by the base's fingerprint. They now
scan the adapter and base together via evaluate_remote_code_consent_for_targets,
which pins one combined fingerprint over the union of their code. The malware
scan in each worker is also scoped to the model's load subdirectories so a
flagged pickle under a from_pretrained load subdir is not missed.
* Report a consistent trust_remote_code requirement after a model loads
validate_model reports requires_trust_remote_code from the YAML default OR the
raw auto_map, but the load, already-loaded, and status responses reported only
the YAML default. A custom-code model approved and loaded via auto_map was then
reported as not requiring trust_remote_code, so the frontend stored false and a
later retry or rollback sent trust_remote_code=false and failed.
A shared resolver reports the same requirement for a loaded model (a value
stored at load time, else the trust_remote_code the load used, else the YAML
default, else the raw auto_map check), and the load response persists it so the
status and already-loaded paths stay consistent. The selected-GGUF security
review is also scoped to the model's load subdirectories.
* Run the consent gate on training resume and for YAML-only trust_remote_code
Three frontend gaps left a model loading without the trust_remote_code it needs:
- The shared consent helper returned early when the scan found no auto_map and no
unsafe files, dropping a requirement that comes from a model's Studio YAML
default (e.g. GLM-4.7-Flash). It now grants the caller's requirement with an
empty pin instead of sending trust_remote_code=false.
- Resume-from-history called startTraining directly with no consent gate, so a
resumed run whose model needs custom code (or an old run with no approved
fingerprint) hit the worker block with no dialog. It now runs the same gate as
a fresh start.
- HF export passed requiresTrustRemoteCode=false for every HF source, so a
YAML-only model could not flip the flag before export. It now signals the
requirement for HF sources.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Cover both LoRA repos in validate, report GGUF as inert, purge all declined repos
Three follow-on gaps from the combined adapter+base consent work:
- validate_model resolved requires_trust_remote_code from the base alone, so a
LoRA adapter that ships its OWN auto_map code (with a plain base) was reported
as not needing trust_remote_code and the consent dialog never opened. It now
checks the [adapter, base] target set, matching the scan route and the workers
(which already gate both) and the security review already running over both.
- The already-loaded, loaded, and status responses for a selected GGUF reported
requires_trust_remote_code from the model's YAML default. A GGUF loads through
llama.cpp, which never executes the repo's auto_map Python, so the requirement
is inert for that load. They now report False, matching validate_model (which
already skips both gates for GGUF) so a status refresh cannot flip the flag
back on.
- The remote-code scan downloads both the adapter's and the base's config, but
created_by_scan tracked only the primary, so a base the scan was first to pull
into the cache was left on disk when the user declined. The scan now reports
scan_created_repos (every repo it newly cached) and the decline cleanup purges
each; created_by_scan stays for older clients. The frontend falls back to the
primary flag when the list is absent.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scan the repo the load fetches, purge external code on decline, harden consent pins
Six follow-on hardening fixes from a fresh review pass over the gate:
- The malware gate scanned the literal "Spark-TTS-0.5B/LLM" alias, but the trainer
downloads it as unsloth/Spark-TTS-0.5B and loads LLM/, so the alias 404'd and
failed open, missing a flagged LLM/ pickle. evaluate_file_security now resolves
the alias to the repo the loader fetches and scans LLM/ as a load root.
- security_load_subdirs relied only on tokenizer detection, which fails on an
unresolved alias or offline; it now also honors the Studio YAML audio_type
default, so a BiCodec LLM/ load root is not missed.
- The remote-code scan downloads external auto_map repos (owner/name--module.Class),
but the decline cleanup tracked only the model/adapter/base, leaving the external
untrusted code cached. The scan now enumerates external auto_map repos and reports
the ones it created in scan_created_repos, so a decline purges them too.
- External auto_map refs failed the whole load closed on a stale or mis-derived
dotted ref (sub.mod.py vs the real sub/mod.py) even though the actual file was
present and scanned. They now drop such refs when the repo listing is real, exactly
like the own-repo path; an empty/incomplete listing still fetches and fails closed.
- The combined consent fingerprint keyed code by the raw target string, so the scan
endpoint's canonicalized casing and a worker's raw user input produced different
pins for identical code, rejecting a valid approval. Hub repo ids are now folded to
lowercase in the key (local paths stay case-sensitive), so the pin tracks the code.
- Export threaded hf_token into the weight load but not into detect_audio_type /
is_vision_model, so a gated multimodal base 404'd in detection and fell through to
the text loader. Both probes now use the same token.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Thread the token through check-vision and guard the gate's parallel sites
The /check-vision endpoint classified a model without the hf_token, so a gated or
private vision model 404'd in the probe and was reported as a plain text model --
the same dropped-token shape as the export probes, at a sibling site. It now passes
the token like the neighboring /check-embedding endpoint.
Add deterministic consistency guards (tests/test_security_gate_consistency.py) that
enumerate the gate's parallel sites mechanically instead of relying on a review to
spot a missed sibling: every is_vision_model / is_embedding_model / detect_audio_type
caller under routes/ and core/ must thread the token, every GGUF response must report
trust_remote_code via the resolver or False (never the raw YAML default), and every
load worker that runs the malware or consent gate must resolve the LoRA base. A new
site that drops the token or mis-reports the requirement now fails CI directly.
* Narrow the LLM alias rewrite and make audio detection token-aware
Three fixes from the confirmatory review, one a regression from the previous round:
- _load_scan_target rewrote EVERY remote repo ending in "/LLM" to unsloth/<parent>,
so a real third-party repo named "<owner>/LLM" was scanned as unsloth/<owner>
while the loader still fetched the real repo -- a fail-open hole introduced when
the Spark-TTS alias handling was added. It now rewrites only a registry-known
bicodec alias; every other "/LLM" repo is scanned as itself.
- detect_audio_type cached results under the bare model name, so an unauthenticated
probe of a gated/private repo cached None and poisoned a later authenticated call
with the token. The cache is now keyed by (normalized_name, token_fingerprint),
matching the vision cache.
- The training fallback /check-vision call dropped the hf_token, misclassifying a
gated/private VLM when the config endpoint failed. It now passes the token, like
the getModelConfig call it falls back from; checkEmbeddingModel takes the token too.
Extend the consistency guards: every capability cache must be keyed by a tuple
including the token, so a cache re-declared as Dict[str, ...] fails CI.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Document the broad .py scan as deliberate and enforce it with a test
The remote-code scanner scans every .py in a repo once an auto_map exists, not
just the auto_map entry's static import closure. This is intentional: the entry
module can reach a sibling via an absolute import, importlib, or exec, none of
which a static relative-import closure follows, so closure-only scanning would be
a real bypass of a load-time RCE gate. The broad scan never under-scans; the cost
is that an unrelated benign script can over-block, which is the safe failure
direction (HIGH stays approvable; only CRITICAL hard-blocks).
Spell this out at both the local and remote scan sites so the choice reads as
deliberate, and add a test asserting an unrelated, never-imported .py is still
scanned -- so a future narrowing to the static closure fails CI.
* Purge a declined remote LoRA adapter the scan downloaded
scan_model_remote_code probed the created-by-scan state AFTER resolving the base,
but get_base_model_from_lora_identifier downloads a remote adapter's own
adapter_config.json, so the adapter looked already-cached and was dropped from
scan_created_repos. On decline the adapter -- including the auto_map .py the
preflight fetched -- was left on disk, defeating the "untrusted code is not left
on disk" guarantee for the adapter itself.
Snapshot the primary's cache state BEFORE base resolution and use it when marking
the adapter scan-created; on any probe error treat it as pre-existing so a decline
never deletes it. The base and external repos are unaffected (their configs are not
downloaded before their own probe). Add a test that models the mid-scan download
side effect, which the prior static-stub tests did not.
* Clear remote-code approval when the training model changes
Switching the training model from an approved custom-code model to a clean one
kept the previous model's trust_remote_code=true and approved fingerprint in the
store: setSelectedModel reset visionImageSize on a true switch but not the
remote-code approval. The clean model then trained with trust_remote_code=true,
which bypasses the compiler and disables fused cross-entropy.
Reset trustRemoteCode and approvedRemoteCodeFingerprint on a true model switch.
The new model's own YAML default is re-applied by loadAndApplyModelDefaults, and a
custom-code model still re-opens the consent dialog before training starts, so the
only change is that a clean model no longer inherits a stale approval.
* Trim verbose comments across the model-fetching hardening changes
Condense the explanatory comments and docstrings introduced across the
trust_remote_code consent gate, the malware/unsafe-file gate, the remote-code
scanner, the load workers, the model routes, and the security frontend into
fewer, tighter lines while preserving every security rationale (fail-open vs
fail-closed direction, the deliberate broad-scan anti-bypass note, the
empty-vs-unscannable distinction, stale-ref handling, and the alias-rewrite
spoof guard).
Comments and docstrings only. No code, logic, identifiers, or test behaviour
changed; verified comment-only via the AST/TypeScript checker (40/40), with the
backend test suite and frontend tsc green.
* Do not cache transient audio-detection failures
detect_audio_type cached _detect_audio_from_tokenizer's result
unconditionally, so a transient read failure (network error or 5xx,
returned as None) poisoned the cache and the later successful probe never
ran. Mirror the vision cache: _detect_audio_from_tokenizer now returns
(audio_type, definitive) and the caller caches only definitive results.
A read that succeeds with no audio tokens, or clean 404s for every
tokenizer path, stays a cacheable None; only a genuine transient failure
(connection error, timeout, 5xx, malformed body) skips the cache so the
next call retries.
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: add 'Load on selection' toggle to configure load options before loading
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: seed staged speculative decoding from the standing default
* Studio: address PR review for load-on-selection staging
* Studio: handle direct GGUF staging and stale-stage edge cases from load-on-selection review
* Studio: cancel replaced staged downloads and keep staged pick on load failure
* Studio: centralize staged-download cancel and guard staged-load restore
* fix: address staged GGUF load review
* fix: honor staged GGUF load metadata
* fix: clarify load-on-selection tooltip
Keep the load-on-selection hint visually anchored to the control and make the on/off behavior explicit without changing the broader deferred-load flow.
* Studio: reset orphaned staged knobs on abandon and cap Max Tokens to staged context
* Studio: remove dead code and cancel staged download when loading a different model
* fix: surface staged model in run settings before deferred load
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Studio: Add inline confirmation (Allow/Always allow/Deny) for tool calls
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix race in tool-call confirmation gate
* Studio: gate built-in tool calls and harden the confirmation handshake
The Allow / Always allow / Deny controls only lived in the fallback tool
card, but the built-in tools (web search, python, terminal, code
execution, image generation) render with their own components and so
never showed the buttons. Those calls paused after tool_start with no way
to approve them, hanging until the 1 hour timeout. Only MCP tools, which
use the fallback renderer, actually worked.
Render the controls for every tool card by wrapping each registered tool
component (and the fallback) in thread.tsx with a shared
ToolConfirmationControls, so the gate applies uniformly.
Also make the handshake robust:
- The gate keys on a per-call approval_id minted by the backend and
echoed in tool_start, instead of session_id alone, so a stale or
concurrent confirmation can no longer resolve the wrong call.
- The approval slot is registered before tool_start is yielded, closing
the race where a fast click or an auto "Always allow" could reach the
backend before the waiter existed.
- The frontend resolves with the same session id the request was sent
with (plus the approval_id), fixing the new-thread mismatch where the
confirmation targeted a different session than the blocked stream.
- The confirm endpoint returns {resolved}; the UI keeps the buttons and
shows a retry hint until the backend confirms a match, instead of
hiding them on a failed or mistargeted post.
- The gate runs after the disabled-tool and duplicate-call checks, so a
call that will not execute is not put up for approval. A denied call is
still excluded from duplicate detection, so re-issuing and approving it
works.
- "Always allow" is scoped per session to match the backend gate.
Add backend tests for the approval registry, the SSE no-deadlock
handshake, and the loop integration (allow, deny, disabled, duplicate,
re-issue after deny).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Move "Confirm tool calls" to the Tools section
* Studio: add Bypass Permissions (skip confirmation, disable tool sandbox)
Adds an opt-in Bypass Permissions toggle next to Confirm tool calls. When on,
no tool call shows a confirmation prompt and the python/terminal sandbox is
disabled: safety checks, command blocklist, and resource limits are skipped.
Secret env vars are still stripped and HOME stays repointed at the session
workdir. Default off keeps current behavior, and it takes precedence over
Confirm tool calls. Enabling it requires accepting a warning each time.
* Studio: harden Bypass Permissions secret handling and fix Anthropic tool path
Follow-up to the Bypass Permissions feature. Addresses the review findings:
- Anthropic /v1/messages 500: declare bypass_permissions on
AnthropicMessagesRequest so tool requests that omit the field default to
False instead of raising AttributeError (extra='allow' does not set absent
attributes).
- /proc parent-env leak: stripping the child env did not stop a same-uid
bypassed child from reading /proc/<parent>/environ to recover the
tool-executing process's unfiltered secrets. Clear PR_SET_DUMPABLE on that
process before the first bypass exec so its /proc entries become root-owned.
Hardening is fail-closed: if prctl is denied, bypass execution is refused
rather than run with the parent environ still readable. Mitigation, not a
full boundary; documented in the code.
- Broker/capability vars: strip SSH_AUTH_SOCK, SSH_AGENT_PID, GPG_AGENT_INFO,
GNUPGHOME, KUBECONFIG, DOCKER_HOST so a bypassed tool cannot use the
operator's live agents.
- Credential-bearing URL values: drop any env var whose value embeds URL
userinfo (scheme://user:pass@ and token-only scheme://token@) regardless of
the variable name. Benign proxy/index URLs without credentials are kept, so
proxy-only and internal-index setups still work in bypass mode.
- Windows temp isolation: repoint TEMP and TMP (not just TMPDIR) at the
per-session sandbox dir.
- Frontend: stop persisting bypassPermissions; a reload now starts with the
sandbox/confirmation bypass off and requires re-accepting the warning dialog.
Adds regression tests for each finding in test_bypass_permissions.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strip cred-location env vars (HF_HOME etc.) in Bypass Permissions
Repointing HOME did not stop SDKs auto-reading cached creds via vars that
point at the real home/cache/config: HF_HOME (startup always sets it; token
lives under $HF_HOME/token), HF/XDG cache roots, NETRC/BOTO_CONFIG/
PIP_CONFIG_FILE, and Windows HOMEDRIVE/HOMEPATH. Drop those, and repoint
USERPROFILE/APPDATA/LOCALAPPDATA at the per-session workdir. Adds regression
tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: lock in bypass HF token resolution with an end-to-end test
The drop-based fix relies on the whole HF_HOME/XDG fallback chain being
removed so huggingface_hub resolves under the repointed HOME. Add a test
that sets HF_HOME and XDG_CACHE_HOME at a real cache and asserts the
resolved token path lands under the workdir, not the operator's cache.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strip npm _auth, MYSQL_PWD, and BASH_ENV from bypass env
Three more credential vectors dodged the bypass scrubber: NPM_CONFIG__AUTH
(npm _auth, base64 so no URL userinfo and no AUTH marker), MYSQL_PWD (markers
use PASSWD, not PWD, since PWD is the cwd var), and BASH_ENV (bash -c sources
it for non-interactive shells, so a startup file can re-export stripped
secrets). Add an AUTH marker, the exact MYSQL_PWD name, and drop BASH_ENV plus
PGPASSFILE. Adds regression tests incl. an end-to-end check that a bypass
terminal call does not source BASH_ENV.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: extend bypass env scrubber and enforce confirm precedence in loops
From a parallel review pass over the bypass changes:
- Drop more credential-location vars in _build_bypass_env: npm/yarn/git/cargo/
rclone config pointers (NPM_CONFIG_USERCONFIG, NPM_CONFIG_GLOBALCONFIG,
YARN_RC_FILENAME, GIT_CONFIG_GLOBAL, GIT_CONFIG_SYSTEM, CARGO_HOME,
RCLONE_CONFIG) and the GIT_ASKPASS/SSH_ASKPASS auth helpers.
- Enforce confirm_tool_calls AND NOT bypass_permissions inside the safetensors
and GGUF tool loops, not just at the route, so a direct internal caller
passing both flags never prompts.
- Soften the toggle hint: environment secrets are stripped, but bypassed code
can still read files and credentials on the machine (no overclaim that keys
stay hidden).
Adds regression tests for the new names and the loop-level precedence.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add GGUF loop test for bypass-over-confirm precedence
The safetensors loop precedence is covered behaviorally; the GGUF loop needs a
live llama-server so add an AST guard asserting its _needs_confirm gate
references both confirm_tool_calls and bypass_permissions, matching the other
llama_cpp source-inspection tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: add red Bypass Permissions badge in the composer
When Bypass Permissions is on, show a persistent red pill in the composer
tool-pill row (like the Search/Code pills), matching Claude Code's always-
visible bypass indicator. Clicking it turns bypass off, mirroring the other
composer toggles. Enabling still goes through the settings toggle + warning
dialog. Adds a data-variant=danger style for the destructive-colored pill.
* Studio: show Bypass Permissions badge in the Thread composer too
The empty-state and active Thread render their own composer (thread.tsx),
not shared-composer, so the badge only appeared in the split layout. Mirror
the red dismissible pill in ComposerAction so it shows in every composer.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: keep the Bypass Permissions badge visible when the composer is collapsed
The Thread composer only renders the pill row when expanded, so the active-mode
badge vanished on the default (collapsed) empty state. Render it before the
expand gate (it returns null when bypass is off) so the red indicator always
shows while bypass is on.
* Studio: make the Bypass Permissions confirm button a solid red button
The destructive button variant is a subtle 10% tint that read as bare red text
next to the outlined Cancel. Force the solid destructive fill (the variant's
class loses to the tint through AlertDialogAction's Slot merge, so use the !
override the codebase already uses for this case) and shorten the label to
'I understand' so it fits the small dialog's two-column footer.
* Studio: add Bypass Permissions to the composer + More menu
Adds a 'Bypass Permissions' entry to the composer plus-menu (under More by
default) in both composers, so it can be toggled without opening Run settings.
Enabling routes through the same danger warning dialog; disabling is immediate.
A shared BypassPermissionsMenuItem keeps the two composers in sync.
* Studio: harden bypass env scrubber for IMDS opt-out and connection strings
Two gaps in the Bypass Permissions secret scrubber:
- The broad AWS_ prefix also dropped AWS_EC2_METADATA_DISABLED, a non-secret
opt-out. Removing it re-opens the IMDS instance-role credential path that the
operator explicitly disabled, so a bypassed boto/AWS-CLI call could recover
cloud creds. Keep that flag (and AWS_EC2_METADATA_V1_DISABLED) via a keep-list
while still stripping the real AWS credential vars.
- Azure App Service connection strings (SQLCONNSTR_/CUSTOMCONNSTR_/...,
WEBSITE_CONTENTAZUREFILECONNECTIONSTRING) and values like Password=/AccountKey=
/SharedAccessKey= slipped past the name and URL-only value classifiers. Add
CONNSTR/CONNECTIONSTRING name markers and a connection-string value matcher.
* Studio: let Bypass Permissions suppress the confirm-tool-calls guards
The confirm-vs-bypass precedence (confirm and not bypass) was applied at the
loop call sites but not at the earlier request guards, so a client sending
confirm_tool_calls + bypass_permissions together was rejected (stream=true
required / unsupported for external or Anthropic tools) before the precedence
took effect. Gate all four confirm guards on not bypass_permissions so both
flags together proceed with the gate suppressed, matching the documented rule.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: add Anthropic-compatible thinking parameter
Add `thinking` parameter using Anthropic's format ({type: 'disabled'} /
{type: 'enabled'}) alongside the existing `enable_thinking` boolean for
backward compatibility.
The new parameter is mapped internally to `enable_thinking` at the route
layer so all downstream templates and backends continue to work unchanged.
Changes:
- Add ThinkingConfig model and `thinking` field to ChatCompletionRequest
- Add mapping logic in routes: thinking.type -> enable_thinking
- Add `thinking` field to frontend TypeScript types
- Update frontend request building to send thinking parameter
- Add tests for new thinking parameter
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: move thinking→enable_thinking mapping to model_validator
The Gemini review correctly identified that the route-level mapping
bypasses normalization for external provider requests. Moving the
mapping into a @model_validator on ChatCompletionRequest ensures it
runs during Pydantic validation regardless of routing path.
* Document ThinkingConfig scope and thinking validation behavior
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: serve DiffusionGemma GGUFs with the on-device visual decoder
* Studio: render the DiffusionGemma denoising canvas live in chat with honest stats
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden DiffusionGemma runner resolution (Windows .exe, build/bin lookup, clear stale audio flag, safe PYTHONPATH, Linux-only pdeathsig)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(studio/responses): forward chat_template_kwargs enable_thinking to chat request
The /v1/responses translation in _build_chat_request dropped
chat_template_kwargs (e.g. {"enable_thinking": true}) sent via the
Responses extra-body, so reasoning control was silently ignored.
Lift enable_thinking onto the typed ChatCompletionRequest field,
mirroring openai_chat_completions, so both the non-streaming and
streaming Responses pass-through paths honor it.
Fixes#6198
Signed-off-by: Tai An <antai12232931@outlook.com>
* Fix/adjust Responses reasoning for PR #6202
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust reasoning none for PR #6202
* Fix/adjust structured reasoning for PR #6202
* Fix/adjust responses reasoning review findings for PR #6202
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust responses reasoning follow-ups for PR #6202
* Fix/adjust think parsing gate for PR #6202
---------
Signed-off-by: Tai An <antai12232931@outlook.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: Add Tensor-Parallel llama.cpp support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden Tensor-Parallel fallback and GPU selection
* Studio: reconcile split-mode extras and harden tensor-split planning
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: reconcile split-mode extras in backend duplicate-load guard
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: preserve inherited non-tensor split modes on reload
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: honor cancellation in tensor fallback, preserve tensor mode on rollback, and don't raise an explicit small context
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: reconcile split-mode in reload check and strip it on tensor downgrade
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Strip --tensor-split alongside --split-mode so inherited ratios don't override the tensor planner
An inherited or stale --tensor-split in llama_extra_args was appended after
Studio's computed --tensor-split and won last in llama.cpp, re-introducing the
asymmetric-GPU OOM tensor mode is meant to prevent. Group -ts/--tensor-split
into the split-mode shadow set so it is stripped on inherit and on the layer
fallback; parse_split_mode_override still keys on the mode value only.
* Drop quantized KV for the tensor attempt and report native max context
Tensor mode aborts on a quantized KV cache, so a user with q8_0/q4_1 etc. who
enabled Tensor Parallelism silently fell back to layer split. Clear the cache
type (and strip inherited/explicit --cache-type) for the tensor attempt only;
the layer fallback re-runs with tensor off and keeps the user's choice.
Also report max_available_ctx from the native context, not an explicit small
-c, so the context slider no longer warns too early in tensor mode.
* Reconcile inherited split-mode extras in the already-loaded check
When a same-model load omitted llama_extra_args, the tensor comparison resolved
the raw (None) request and treated an inherited --split-mode tensor server as a
mismatch, forcing a needless reload. Compare using the stored extras stripped
the same way the reload strips them.
* Pass tensor_parallel through compare-mode loads
The generalized compare path loaded each GGUF without tensor_parallel, so
compare ran layer split even with the toggle on and left the settings sheet
stale. Send the toggle and hydrate the loaded state from the response, matching
the main chat and recipe load paths.
* Add --tensor-parallel flag to unsloth studio run
The headless one-liner could only reach tensor mode by passing --split-mode
tensor as a raw llama.cpp extra. Add a first-class --tensor-parallel/
--no-tensor-parallel option that sets the tensor_parallel field on the
/api/inference/load payload, forwarded through the studio-venv re-exec like the
other polarity flags. Matches the web UI toggle and the API field.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* Studio: Add inline confirmation (Allow/Always allow/Deny) for tool calls
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix race in tool-call confirmation gate
* Studio: gate built-in tool calls and harden the confirmation handshake
The Allow / Always allow / Deny controls only lived in the fallback tool
card, but the built-in tools (web search, python, terminal, code
execution, image generation) render with their own components and so
never showed the buttons. Those calls paused after tool_start with no way
to approve them, hanging until the 1 hour timeout. Only MCP tools, which
use the fallback renderer, actually worked.
Render the controls for every tool card by wrapping each registered tool
component (and the fallback) in thread.tsx with a shared
ToolConfirmationControls, so the gate applies uniformly.
Also make the handshake robust:
- The gate keys on a per-call approval_id minted by the backend and
echoed in tool_start, instead of session_id alone, so a stale or
concurrent confirmation can no longer resolve the wrong call.
- The approval slot is registered before tool_start is yielded, closing
the race where a fast click or an auto "Always allow" could reach the
backend before the waiter existed.
- The frontend resolves with the same session id the request was sent
with (plus the approval_id), fixing the new-thread mismatch where the
confirmation targeted a different session than the blocked stream.
- The confirm endpoint returns {resolved}; the UI keeps the buttons and
shows a retry hint until the backend confirms a match, instead of
hiding them on a failed or mistargeted post.
- The gate runs after the disabled-tool and duplicate-call checks, so a
call that will not execute is not put up for approval. A denied call is
still excluded from duplicate detection, so re-issuing and approving it
works.
- "Always allow" is scoped per session to match the backend gate.
Add backend tests for the approval registry, the SSE no-deadlock
handshake, and the loop integration (allow, deny, disabled, duplicate,
re-issue after deny).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Move "Confirm tool calls" to the Tools section
* Studio: Keep tool group open while a tool call awaits confirmation
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix tool confirmation session scope for PR #5869
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix confirmation follow-ups for PR #5869
* Apply pre-commit formatting for PR #5869
* Fix confirmation cleanup for PR #5869
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden confirmation lookups for PR #5869
* Studio: make the tool-call confirmation decision immutable
resolve_tool_decision accepted a second confirmation for the same approval_id
and overwrote slot["decision"] in the window before the waiter reads it and
pops the slot, so a duplicate or out-of-order POST could flip an Allow to Deny
(and returned a misleading resolved:true). Reject once the slot's event is
already set so the first decision wins. Adds a regression test.
* Fix/adjust tool confirmations for PR #5869
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: report the real llama-server context window and add an opt-in overflow policy for OpenAI-compatible serving
A community report showed OpenCode failing tool calls every few minutes
against Studio's OpenAI-compatible API while the same GGUF was stable on
LM Studio. Root cause: Studio advertises the requested context length, but
llama-server can allocate less (memory-fit step on small GPUs, --parallel
slot split), so clients budget against a window that does not exist. Their
generations truncate mid tool call at the real wall (finish_reason=length
with cut JSON arguments) and eventually the prompt itself exceeds the real
window, returning a 400 that agentic clients treat as non-retryable.
Changes:
- After llama-server health, read default_generation_settings.n_ctx from
/props and adopt it whenever it is below Studio's computed context, with
a warning. The load response, status route, UI value, and the passthrough
max_tokens ceiling all become honest automatically.
- Expose context_length and max_context_length on /v1/models so clients can
budget against the enforced window.
- Accept empty role=tool content (commands with no output are routine in
agentic loops; OpenAI and llama-server both accept it) instead of a 400.
- Add context_overflow=truncate_middle (per request, or server-wide via
UNSLOTH_CONTEXT_OVERFLOW=truncate_middle): on exceed_context_size_error
the passthrough drops whole middle turn-groups (system prompt, first turn,
and recent turns kept; tool calls stay paired with their results), clips
oversized contents middle-out when group-dropping is not enough, clamps
max_tokens to the generation headroom, and retries. Default stays 'error'
with code=context_length_exceeded so clients running their own compaction
keep full control.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: allocate the requested context for real (kv-unified, fit-ctx floor)
Two launch-flag gaps caused the advertised vs allocated divergence at the
source:
- llama-server enables --kv-unified only when the slot count is auto; Studio
always passes --parallel N, which silently splits -c into per-slot windows
of -c/N. Pass --kv-unified when N > 1 so a single request can use the full
advertised window (same total KV memory, shared pool).
- with --fit on the fit step may set ctx as low as 4096; pass
--fit-ctx <requested> for explicit requests so fit offloads or fails into
the existing --fit off retry instead of silently shrinking the window.
Both flags are gated on --help capability probing so older builds keep the
current behavior, where the /props readback remains the backstop. Verified
live: -c 98304 --parallel 4 now serves per-slot n_ctx 98304 (was 24576),
48k-token requests pass through the passthrough, and the readback warning no
longer fires.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: surface the llama.cpp update affordance when MTP is disabled
When a model asks for MTP (auto on an MTP model, or forced mtp / mtp+ngram)
but it gets disabled, the load already degrades gracefully and serves without
speculative decoding. Until now the UI gave no hint why, or that an update
would fix it.
Record why MTP was dropped on the backend (spec_fallback_reason): the probe
found no mtp token (binary_no_mtp), the spawn aborted with an outdated-arch /
context-build error such as a prebuilt that predates the Gemma drafter
(binary_outdated), or the current build could not run it, e.g. a CUDA kernel
limit (runtime_error). Expose it in the inference status. In the chat
Speculative Decoding section, show a short note and, for the two update-fixable
reasons, an inline Update llama.cpp button that reuses the existing update flow.
A runtime_error gets the note without an update push, since a newer build may
not fix it.
Backend tests cover the reason being set / cleared. Frontend typechecks.
* Address review: tighten the update hint to genuinely outdated binaries
Reserve binary_outdated (which surfaces the Update llama.cpp affordance) for an
unknown-architecture abort, which proves the prebuilt predates the model;
classify the generic memory/context build failures as runtime_error, where an
update may not help. Frontend: only append the "Update llama.cpp to enable it"
sentence when an update is actually available, so the text never points at an
action the UI is not offering.
* Studio: sync detected model capabilities into models[] after load
The chat composer gates audio upload on activeModel.hasAudioInput, but
/api/models/list omits audio fields for default and active-GGUF entries
and the single chat load path never wrote the load response's
capability flags back into the store. Audio-capable models such as the
Gemma 4 GGUFs therefore never unlocked audio input in the main chat,
while the compare composer (which does sync) worked.
Add syncModelCapabilities and call it after a successful load and after
the status fetch in refresh, so the flags also survive F5 and are not
clobbered by stale catalog data.
* Studio: merge audio upload into the Add photos & files picker
Remove the separate Upload audio row from the composer plus menu and
register an AudioAttachmentAdapter in the shared attachment pipeline,
so the standard picker and drag-drop accept wav, mp3, m4a, ogg, flac
and webm directly. Gating matches images: the picker always lists
audio and models without audio input get a toast at add() time. The
50MB limit is kept and the file shows as a normal attachment chip.
On send the adapter emits an audio content part on the attachment and
findLatestUserAudioBase64 now also scans attachment content, so the
request still carries audio_base64 exactly as before.
* Studio: extract AudioAttachmentAdapter into its own module
Move the adapter out of runtime-provider.tsx so it is importable in
isolation, export the audio send-path and capability-sync helpers for
tests, and guard attachment id generation for non-secure contexts
(crypto.randomUUID is undefined over plain HTTP on a LAN, matching the
existing guard in startCompare).
* Studio: do not claim .webm by extension in the audio adapter
A video/webm file would match the .webm extension entry and route to
the audio adapter. Real audio webm (MediaRecorder output) always
reports the audio/webm MIME, so matching webm by MIME only keeps video
files out while keeping recorded audio working.
* Studio: only send audio from the newest user message
audio_base64 switches the backend onto the audio generation path
(generate_whisper_response ignores chat messages entirely and
generate_audio_input_response bypasses the normal streaming path), so
replaying audio from an older turn hijacked text-only follow-ups:
Whisper would retranscribe the stale clip instead of erroring cleanly,
and audio VLMs lost tools and streaming. Stop the scan at the newest
user message, matching the consumed-on-send semantics of the legacy
pendingAudio path. Regenerating the audio turn itself still resends
its audio since it is the newest user message in that run.
Also guard extractAudioPartBase64 against null parts in deserialized
history content.
* Studio: forward audio input to llama-server for GGUF models (#6096)
* Studio: forward audio input to llama-server for GGUF models
* Studio: harden GGUF audio input handling (multi-format decode, size cap)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: carry GGUF audio in the message list so it works with tools
* Studio: bound decoded audio length and make the soundfile decoder optional
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Handle audio attachment edge cases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate audio file picker by loaded model capability (#6142)
* Gate audio attachments by loaded model
* Use conditional spread for audio attachment adapter
* Preserve audio fallback while filtering picker
---------
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
Co-authored-by: oobabooga <oobabooga4@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: support separate-file MTP GGUF drafters (Gemma 4)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix review findings for separate-file MTP drafters
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: pair local MTP drafters by name and include them in reload dedup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: manage --model-draft in extras and reject MTP/ copies as models
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Adds a self-contained RAG stack to Studio: knowledge bases with chunked indexing, hybrid (dense + lexical) retrieval, and an automatic first-pass context inject into chat. Embeddings run through a local llama-server GGUF backend (default unsloth/bge-small-en-v1.5-GGUF) with a sentence-transformers fallback. The chat tool loop gains a search_knowledge_base tool, a per-turn re-search cap, and source citation, layered on top of the shared ToolLoopController.
* Studio: fix OpenAI- and Anthropic-compatible API spec compliance
* Studio: fix API spec-compliance gaps on passthrough and streaming paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: carry context_length_exceeded through the OpenAI passthrough error path
* Studio: count tool-schema tokens in the Anthropic server-tool stream, and small stream-handling guards
* Studio: guard message_delta usage against None and normalize developer role before proxying
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: honor max_completion_tokens on the external-provider proxy path
* Studio: forward llama-server cached_tokens into OpenAI prompt_tokens_details
* Studio: sanitize messages in count_tokens to match the /v1/messages prompt
* Studio: report max_tokens for truncated tool calls and guard null usage in metadata events
* Studio: drop the request-id middleware (headers aren't declared in either spec)
* Studio: include the required request_id field in Anthropic error bodies
* Studio: honor max_completion_tokens on the audio (TTS / audio-input) paths
* Studio: add the _effective_max_tokens helper and route all max-token sites through it
* Studio: align API compatibility edge cases
* Studio: clarify multi-choice chat support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: clarify logprobs chat support
* Studio: opt the local chat UI into the streaming usage chunk so the context bar and tok/s repopulate
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: forward seed to llama-server, and fix Anthropic server-tool stop_reason, tool_result id correlation, and parallel-tool execution cap
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: align OpenAI chat completion spec edge cases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: align backend API compatibility tests
* Studio: honor tool caps and internal stream usage
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: coerce nullable stream usage counts
* Studio: preserve system prompts with developer messages
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Trim and tighten code comments and docstrings across studio/ Python. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.
Normalize misplaced system-role messages in /v1/messages by hoisting their content into the top-level Anthropic system field, fixing the 422 that newer Claude Code clients trigger. Null and non-text system content is ignored rather than stringified.
Fixes#6001
* added remote MCP server support
* trim
* added tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* increased timeout
* disabling MCP chat toggle
* Fix MCP OpenAI function-name validation + cancel propagation for PR #5750
OpenAI requires function.name to match ^[a-zA-Z0-9_-]{1,64}$ before
streaming starts. The existing 64-char length check is necessary but
not sufficient: MCP servers can return tool names containing '.', '/',
spaces, etc. that would 400 the whole chat request. Validate the
composed mcp__<server_id>__<tool> name against the regex, skip + warn
on miss, and drop duplicate tool names from the same server (which
would also 400 the request as "duplicates").
Also propagate the agentic-loop cancel_event into MCP tool execution
so a /cancel POST during a long-running MCP call (e.g. GitHub MCP
search across a large repo) actually interrupts the in-flight HTTP
call instead of waiting out the 300 s timeout. The watcher polls the
threading.Event at 50 ms cadence inside the asyncio loop (matches
routes/inference.py's existing cancel-watcher cadence) and races
against the call task with asyncio.wait FIRST_COMPLETED.
Tests added:
- test_mcp_specs_skip_invalid_openai_function_names: drops bad chars
- test_mcp_specs_skip_empty_tool_name
- test_mcp_specs_drops_duplicate_names
- test_call_tool_sync_respects_pre_set_cancel_event
Also fix test_desktop_auth.py's router stub that listed every existing
router but missed mcp_servers_router, so importing main.py fails after
this PR adds it to routes/__init__.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* PR #5750 round 2: OAuth cleanup on delete/url-change + mcp_enabled standalone
Round 2 of cross-platform validation surfaced two more P1 findings:
1. OAuth tokens never get cleared. fastmcp keys tokens by MCP URL, not by
server row, and delete / URL change / use_oauth toggle only updated
the SQLite row. Re-registering the same URL would silently reuse the
old account's credentials. Adds clear_oauth_tokens_async() in
mcp_client.py and calls it from the delete + put route handlers when
the row had use_oauth=True and either the URL changes or OAuth is
turned off.
2. mcp_enabled=true was ignored unless the caller also sent
enable_tools=true. The frontend always sends both together so the UI
path was fine, but a direct API caller sending only mcp_enabled would
silently get no MCP tools, which contradicts the field's documented
"append tools from every enabled MCP server" behavior. Loosens the
use_tools gate in both the GGUF and safetensors paths so mcp_enabled
opens the tool loop on its own; when the caller did not also opt
into built-ins, the built-in list starts empty.
Tests added:
- test_clear_oauth_tokens_async_no_op_safe
- test_delete_server_calls_oauth_cleanup_when_oauth_was_on
- test_delete_server_skips_oauth_cleanup_when_oauth_off
- test_update_server_clears_oauth_on_url_change
- test_update_server_clears_oauth_when_oauth_disabled
26 backend MCP tests pass; full studio/backend suite 1710 passed locally.
Cross-platform CI (Linux, macOS, Windows) green on staging fork.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* PR #5750 round 3: reject null bool updates + /test surfaces 400
Round 3 of cross-platform validation:
1. PUT /api/mcp/servers/<id> would 500 with TypeError when the body
explicitly set is_enabled or use_oauth to null. Pydantic accepts
None for an Optional[bool] and _changes_from_payload then passed
None into mcp_servers_db.update_server, which int(None)d. Reject
explicit null at the validation layer with 400 instead.
2. POST /api/mcp/servers/test caught HTTPException under
"except Exception", so an invalid URL came back as HTTP 200 with
{"ok": false, "error": "400: ..."} instead of a real 400. The
create + update paths return 400 for the same input. Move
validation outside the transport try/except so it surfaces 400.
Tests added:
- test_changes_from_payload_rejects_null_is_enabled
- test_changes_from_payload_rejects_null_use_oauth
- test_test_endpoint_surfaces_url_validation_as_400
* PR #5750 round 4: hyphenated MCP tool names + empty-tool-list gate
Round 4 surfaces two more interaction bugs between the new MCP path
and existing safetensors tool plumbing:
1. OpenAI accepts ^[a-zA-Z0-9_-]{1,64}$ for function.name, and round 1
widened the MCP regex to that set, so MCP tools can now be advertised
as `mcp__srv__list-issues`. But the XML tool-call parser in
tool_call_parser.py used `\w+` (no hyphen), so the model could call
the tool but Studio could not parse the call. Same in
routes/inference.py's `_TOOL_XML_RE` stripper, which would leave
hyphenated tool-call XML in the visible content. Both regexes now
use `[\w-]+`.
2. safetensors_agentic treats `tools=[]` as "allow all" (documented
contract, exercised by test_empty_tools_list_does_not_enforce_allowlist).
When a caller sends `enable_tools=true` + `enabled_tools=[]` +
`mcp_enabled=true` and MCP discovery returns 0, the resolved tool
list is genuinely empty and built-in tools (web_search / python /
terminal) could execute via the model's emitted call. Fix at the
route gate instead of breaking the documented contract: set
`use_tools=False` when the resolved list is empty, in both GGUF and
safetensors paths. Existing callers who omit `enabled_tools` still
get ALL_TOOLS and are unaffected.
Tests added (32 total):
- test_tool_xml_parser_handles_hyphenated_function_names
- test_tool_xml_strip_handles_hyphenated_function_names
- test_safetensors_agentic_empty_allowlist_still_means_allow_all
(documents the contract round 4 preserved)
1716 passed locally; cross-platform CI on staging fork still green.
* PR #5750 round 5: GGUF allow-list + CLI policy + hyphenated params + cancel race
Round 5 of parallel-reviewer aggregation surfaced six additional
findings; five are real and fixed here:
1. Hyphenated MCP parameter names (`<parameter=issue-number>`) were
dropped by the XML parser's `\w+` regex. Extended to `[\w-]+` in
both core/inference/tool_call_parser.py and core/tool_healing.py.
The latter is GGUF's own copy of the parser/strip patterns and was
missed by round 4.
2. core/tool_healing.py's `strip_tool_call_markup` still used
`<function=\w+>` so hyphenated MCP tool-call XML leaked into the
GGUF visible content even after round 4 fixed the shared parser.
3+4. `mcp_enabled` re-opened the tool loop even when the operator
passed `unsloth run --disable-tools` (CLI policy False). Round 2's
`(_tools_on or payload.mcp_enabled)` gate ignored the raw process
policy. Now reads `state.tool_policy.get_tool_policy()` and gates
mcp_enabled on `_cli_policy is not False`. Applied to both GGUF
and safetensors paths.
5. GGUF's agentic loop called `execute_tool(tool_name, ...)` without
checking the model-emitted name against the per-request tool list,
while the safetensors loop already enforces this. Added the same
allow-list check so a model that hallucinates a filtered MCP name
or a built-in the caller opted out of returns "not enabled" instead
of executing.
Bonus P2 fixes:
- `call_tool_sync` now checks `cancel_event.is_set()` BEFORE
creating the call task, so a pre-set cancellation does not open
the HTTP transport.
- `clear_oauth_tokens_async` moved the OAuth import + construction
inside the protected try block; a fastmcp.client.auth load error
used to escape and 500 the delete / update route.
NOT fixed (verified false or out of scope):
- finding #10 "structured_content vs structuredContent": fastmcp's
CallToolResult dataclass uses snake_case (verified live against
structured-only tool result; fields are
`dict_keys(['content', 'structured_content', 'meta', 'data', 'is_error'])`).
- finding #11 "asyncio.run from running loop": call_tool_sync is
invoked from `asyncio.to_thread` worker threads which have no
event loop; asyncio.run() is safe there.
Tests added (37 total): hyphenated param names, tool_healing strip,
GGUF allow-list gate, cancel pre-set short-circuit, OAuth cleanup
constructor-error swallowing. 1721 passed locally, no regressions.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Studio: add Gemini provider with web_search, code_execution, prompt caching, and Nano Banana image generation
Wires Google's native Gemini API into Studio's external-provider stack
so users can pick gemini-2.5-pro / gemini-2.5-flash / gemini-2.5-flash-image
(Nano Banana) alongside the existing OpenAI / Anthropic / OpenRouter
providers. Gemini does not speak OpenAI Chat Completions on its primary
endpoint; the new `_stream_gemini` async generator translates between
the two shapes the same way `_stream_anthropic` handles the Messages API.
Backend:
- New `_stream_gemini` translator in external_provider.py. Converts
OpenAI messages -> Gemini `contents` + `systemInstruction`; maps
generationConfig (temperature / topP / topK / maxOutputTokens);
forwards `tools: [{googleSearch: {}}]` for web_search and
`{codeExecution: {}}` for code_execution; passes `cachedContent`
through for prompt caching; sets `responseModalities=[TEXT, IMAGE]`
for Nano Banana image generation.
- Translates streamed `GenerateContentResponse` SSE frames back into
OpenAI chat.completion.chunk frames (text deltas, function_call ->
tool_calls deltas, inlineData -> image_b64 tool_end envelope, usage
chunk before [DONE]).
- Registry entry switched to native base URL
`https://generativelanguage.googleapis.com/v1beta` with
`openai_compatible: False` and the `x-goog-api-key` auth header.
Model lineup curated to current 2.5 / 2.0 family + Nano Banana.
Frontend:
- Provider-capability matrix: Gemini supports temperature, top_p, top_k,
presence_penalty (matches generationConfig); min_p / repetition_penalty
hidden because the API does not accept them.
- `providerSupportsBuiltinWebSearch` / `providerSupportsBuiltinCodeExecution`
/ `providerSupportsBuiltinImageGeneration` extended for Gemini.
- Prompt caching toggle now also lit on Gemini.
Tests:
- 21 new tests in `test_gemini_provider.py` using httpx.MockTransport.
Cover request body shape conversion, URL/header wiring, web_search
forwarded as googleSearch, function-call translation both directions,
prompt caching passthrough, image generation emitting image_b64,
grounded-search citations -> tool_end, finish_reason mapping, and
vision data URL -> inlineData translation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: forward presence_penalty to Gemini and recover function name from tool_call_id
Two follow-up fixes for the Gemini provider:
* Thread presence_penalty into _stream_gemini and set
generationConfig.presencePenalty when non-zero. The OpenAI-side
capability matrix already exposes the slider for Gemini, so the
value was being collected and silently dropped on the way out.
* When an OpenAI role=tool message omits 'name' and only carries
'tool_call_id', recover the function name from the matching
functionCall on the prior assistant turn. Gemini 400s on an empty
functionResponse name.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: surface Gemini code execution parts as code_execution tool events
The Gemini stream parser only handled text/functionCall/inlineData
parts, so when the user toggled the Code pill on a Gemini model the
sandbox output (executableCode + codeExecutionResult parts) was
dropped on the floor while adjacent text reached the UI. Reviewers
flagged this as the headline feature being silently broken.
Translate both parts into the existing code_execution tool envelope
that CodeExecutionToolUI already consumes for OpenAI / Anthropic:
* executableCode -> tool_start with kind=code_execution and the
source code under arguments.code. We mint a tool_call_id and
stash it so the matching result block can pair to it.
* codeExecutionResult -> tool_end on that id with the stdout under
result. Non-OK outcomes (OUTCOME_FAILED / OUTCOME_DEADLINE_EXCEEDED)
are prefixed onto the text so the failure is visible.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: native Gemini model catalog, function-call ids, and honest cache claim
Three follow-ups to the Gemini provider PR after the codex pass:
* list_models() now translates Gemini's native /v1beta/models
payload ({models[{name, baseModelId, displayName,
supportedGenerationMethods}]}) into the OpenAI-compatible shape
Studio expects. Without this the picker stayed empty for Gemini
and fell back to hardcoded defaults. Embedding-only models are
filtered out.
* Forward the OpenAI tool_call id into Gemini's functionCall.id
and mirror it onto functionResponse.id. Two parallel calls to
the same function name can now be paired unambiguously on the
follow-up turn.
* Drop Gemini from the prompt-caching capability set. The wire
flow requires a separate cachedContents POST first and the
boolean Studio emits today is a no-op; the toggle should not
advertise a feature it cannot apply. Leaves a pointer to the
docs for the eventual two-step orchestration.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: distinct tool_calls index per emitted Gemini function call
Codex flagged that the Gemini stream parser hardcoded
tool_calls[0].index to 0 on every emitted functionCall. OpenAI
reassemblers key tool_calls by index when joining deltas, so two
parallel function calls in one assistant turn collapsed onto a
single slot and the second call's arguments overwrote the first.
Track the running count via len(emitted_function_call_ids) - 1
and emit it as the per-call index. The dedupe guard above (skip
when fc_id already in the set) means the index is monotonic and
stable for the lifetime of the stream. Regression test asserts
[0, 1] across two parallel calls in one candidate parts list.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: surface Gemini 3.5/3.1/3 + Nano Banana 2/Pro and plumb thinking budget
`gemini-2.0-flash` / `gemini-2.0-flash-exp` were retired by Google in 2026
(`/v1beta/models/gemini-2.0-flash:streamGenerateContent` returns HTTP 404
"no longer available to new users"), and the picker had nothing past the
2.x family. Verified against the live ListModels catalog: drop the retired
ids from `default_models` + allowlist and surface the chat-capable
3.5 / 3.1 / 3 families plus the Nano Banana image trio.
Also plumb `enable_thinking` / `reasoning_effort` into Gemini's
`generationConfig.thinkingConfig`. Without this, Gemini 3.5 Flash,
gemini-pro-latest, and the 3.x previews silently spend the caller's
`max_tokens` budget on hidden "thoughts" before emitting any visible
answer -- the chat shows a truncated stub like "The capital of" and
streams stop. Mapping:
- enable_thinking=False / reasoning_effort=none -> thinkingBudget=0
(Flash tier; Pro tier coerces to a small positive budget because
the API 400s on 0 with "This model only works in thinking mode")
- minimal/low/medium/high -> 512/2048/8192/24576 budget tokens
- max/xhigh -> -1 (dynamic)
- default (neither knob set) -> thinkingConfig omitted, model decides
Frontend `getExternalReasoningCapabilities` now surfaces a
`reasoning_effort` picker for every Gemini chat id (Pro tier hides the
"none" option; image-tier ids stay knob-less). Adds 6 unit tests
covering Flash/Pro effort mapping, the off-toggle coercion on Pro,
default omission, and the nano-banana-pro-preview alias routing
through the image modalities path. 28 -> 34 tests in
`test_gemini_provider.py`, all green; full backend suite still passes
(1459/1460; the unrelated test_help_output flake is pre-existing and
not in any file this PR touches).
Live verification against generativelanguage.googleapis.com on
2026-05-24 with `_stream_gemini` directly:
text gemini-3.5-flash single PASS multi PASS
text gemini-3.1-pro-preview single PASS multi PASS
text gemini-3.1-flash-lite single PASS multi PASS
text gemini-3-pro-preview single PASS multi PASS
text gemini-3-flash-preview single PASS multi PASS
text gemini-2.5-pro single PASS multi PASS
text gemini-2.5-flash single PASS multi PASS
text gemini-2.5-flash-lite single PASS multi PASS
text gemini-flash-latest single PASS multi PASS
text gemini-flash-lite-latest single PASS multi PASS
text gemini-pro-latest single PASS multi PASS
image gemini-2.5-flash-image PASS (1082 KB png returned)
image gemini-3.1-flash-image-preview PASS (Nano Banana 2)
image gemini-3-pro-image-preview PASS (Nano Banana Pro)
tool web_search PASS
tool code_execution PASS
-> 16/16 e2e through the actual ExternalProviderClient code path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten Gemini provider after review (PR #5720)
Fixes a batch of bugs surfaced by a second-pass review on top of the
3.5/3.1/3 + Nano Banana 2/Pro additions in c6724dbd.
Backend (external_provider.py):
- Constructor normalises legacy /v1beta/openai base URLs to /v1beta so
Gemini providers saved before the native switch keep working without
a manual re-config.
- Skip thinkingConfig, googleSearch, and codeExecution on image-tier
models (-image / nano-banana). The image responseModalities path is
mutually exclusive with text-tool wiring and stale UI state would
otherwise 400 the turn.
- _PRO_THINKING_PREFIXES now includes gemini-3.5-pro and uses anchored
prefix matching (exact id or "<prefix>-...") so the image-tier
gemini-3-pro-image-preview cannot accidentally match the pro guard.
- Gemini 3 functionCall thoughtSignature is round-tripped through the
tool_calls envelope via extra_content.google.thought_signature on
emit, and replayed as a sibling of functionCall on the next request.
- finishReason swaps STOP -> tool_calls when any functionCall was
emitted on the same turn so OAI clients trigger tool execution
(matches the OpenAI Chat Completions contract).
- usageMetadata.thoughtsTokenCount is rolled into output_tokens and
surfaced on output_tokens_details.reasoning_tokens so total_tokens
reflects the full billable spend instead of dropping the hidden
reasoning slice.
Registry (providers.py):
- Drop gemini-3-pro-preview from default_models. Google shut it down
on 2026-03-09 and auto-redirects to gemini-3.1-pro-preview; we
surface the canonical id only.
- Add model_id_deny_exact = ("gemini-3-pro-preview",) so the live
ListModels fetch does not re-surface the redirect alias.
Route schema (models/inference.py):
- enable_prompt_caching widened to Optional[Union[bool, str]] so the
/v1/chat/completions caller can pass a Gemini cachedContent resource
name (e.g. cachedContents/abc123). Without this widening _stream_gemini
s string cachedContent passthrough was unreachable from the public
route (bool_parsing 422). stream_chat_completion signature mirrors.
Frontend (provider-capabilities.ts, chat-page.tsx, chat-adapter.ts):
- providerSupportsBuiltinImageGeneration now also recognises
nano-banana ids (nano-banana-pro-preview was hidden from the image
pill before).
- providerSupportsBuiltinWebSearch takes the model id so Gemini image
models hide the Search pill (mirrors the backend skip).
- providerSupportsBuiltinCodeExecution uses the same isGeminiImageModel
guard for nano-banana ids.
- GEMINI_THINKING_PRO_PREFIXES gains gemini-3.5-pro; gemini-3-pro
tightened to gemini-3-pro-preview to avoid the image-id overlap.
- Updated 3 callers of providerSupportsBuiltinWebSearch to thread the
selected model id through.
Tests (test_gemini_provider.py): 34 -> 42, all green
- test_image_models_skip_thinking_config
- test_image_models_drop_text_only_tools
- test_gemini_35_pro_recognized_as_pro_thinking
- test_legacy_openai_base_url_normalized
- test_finish_reason_swaps_to_tool_calls_when_function_call_emitted
- test_thought_signature_round_trips_into_gemini_function_call
- test_thought_signature_emitted_in_tool_call_delta
- test_usage_chunk_includes_thoughts_tokens
Verification:
- Backend pytest 1518/1519 passing (one unrelated Qwen3.5 flash-attn
test fails on main as well; nothing in this PR touches that path).
- Frontend npx tsc -b clean.
- Live e2e 16/16 against generativelanguage.googleapis.com through the
patched _stream_gemini code path (all 11 chat models single + multi
turn, all 3 image models returned image bytes, web_search and
code_execution tools both emit the expected envelope).
- Live /api/providers/models against the patched backend surfaces 16
ids (gemini-3-pro-preview correctly filtered via deny_exact).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address second-pass review findings on Gemini (PR #5720)
Round-2 reviewer.py flagged a phantom web_search card on image
turns (12/12 reviewers), route-layer stripping of tool_calls /
tool_call_id / name, an over-narrow image-mode tool guard, and
silent safety blocks. This patch fixes all four.
Backend (external_provider.py):
- web_search_active is now derived from the outbound tools_array
(whether googleSearch was actually forwarded), not the raw
enabled_tools intent. Image-mode turns dropped the tool above so
the inbound stream no longer emits a phantom "search complete"
tool_start / tool_end on those turns.
- text_tools_allowed now uses is_image_model (covers both `-image`
/ `nano-banana` picker models AND text models that requested
`image_generation` via enabled_tools). Verified against the live
Gemini API which rejects both googleSearch and codeExecution
alongside responseModalities=["TEXT","IMAGE"] with explicit 400s
("Search as tool is not enabled for this model", "Code execution
is not enabled for this model").
- promptFeedback.blockReason is surfaced as a 400 content-filter
error chunk instead of returning an empty successful assistant
response. The streaming loop closes the response before exiting.
Route (routes/inference.py):
- _build_external_messages now propagates tool_calls (assistant),
tool_call_id, and name (tool result) through every code path
(string content, multimodal content, non-vision fallback). Without
this Gemini 3 function-call round trips lost their thoughtSignature
+ tool_call_id at the route boundary, and functionResponse.name
arrived empty on the second turn.
- Assistant messages with content=None and tool_calls populated are
preserved as a synthetic empty-string content turn so the
Gemini translator can rebuild the functionCall part.
Tests (test_gemini_provider.py): 42 -> 45, all green
- test_image_models_suppress_phantom_web_search_card
- test_image_generation_tool_drops_text_tools
- test_prompt_feedback_block_reason_surfaces_as_error
Verification:
- Backend pytest 1736 / 1736 (the two pre-existing unrelated fails
on main, test_help_output and Qwen3.5 flash-attn pin, are skipped).
- Frontend npx tsc -b clean.
- Live e2e 16/16 against generativelanguage.googleapis.com:
11 chat models single + multi turn, 3 image models returning
image bytes, web_search and code_execution both PASS.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix third-pass Gemini findings (PR #5720)
Round 3 review follow-ups:
Backend (studio/backend/core/inference/external_provider.py):
- Close response AND aiter_lines iterator in a finally so normal,
prompt-block, and cancellation exits all clean up (eliminates the
RuntimeWarning about aclose never being awaited).
- Pair the synthetic web_search tool_start with a tool_end on the
promptFeedback.blockReason path so the UI does not leave a stuck
"searching..." spinner after the error toast.
- Preserve native id and thoughtSignature on executableCode and
codeExecutionResult tool events under google.native_part, and pair
the tool_end on the code-exec id so multi-turn code-execution
replays do not lose Gemini-required history.
- Carry part-level thoughtSignature on text deltas via
delta.extra_content.google.thought_signature and on inline image
tool_end via google.thought_signature so Gemini 3 image editing
and tool turns round-trip the signature on the next request.
- Guess remote image_url MIME from the URL path so PNG / WebP / GIF
inputs are not silently relabeled as JPEG.
- Roll usageMetadata.toolUsePromptTokenCount into translated input
tokens and surface thoughtsTokenCount as
completion_tokens_details.reasoning_tokens in _build_usage_chunk.
- Only normalize the Google-hosted /v1beta/openai legacy base URL;
custom proxies whose paths happen to end in /openai are left
untouched.
- Forward ChatCompletionRequest.tools and tool_choice through
stream_chat_completion into _stream_gemini, translating to
tools[].functionDeclarations and toolConfig.functionCallingConfig.
Frontend:
- chat-adapter: when Gemini image-generation is enabled for the turn,
also disable Search and Code so the request, builder, and active
pills agree with what the backend actually sends (the backend
already strips text tools when image_generation is in enabled_tools).
- chat-adapter: consume OpenAI-shape delta.tool_calls chunks so
Gemini function-call deltas without text surface as tool-call parts.
- shared-composer: disable Search and Code pills while Gemini image
mode is active so the UI matches the request.
Tests (studio/backend/tests/test_gemini_provider.py): adds coverage
for proxy base-url gating, remote image MIME inference,
toolUsePromptTokenCount, reasoning_tokens propagation, prompt-block
web_search tool_end pairing, native code-exec id/thoughtSignature
metadata, inline image thoughtSignature, text-chunk extra_content,
OpenAI tools/tool_choice translation, and image-model tool drop.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: Gemini 3 thinkingLevel + image-model Search grounding (PR #5720)
Gemini 3.x migrated to a string `thinkingConfig.thinkingLevel`
(MINIMAL/LOW/MEDIUM/HIGH) and rejects `thinkingBudget`+`thinkingLevel`
in the same request. Gemini 3 also cannot turn thinking fully off, so
the lowest position is "minimal" (Flash) or "low" (Pro rejects
"minimal").
- external_provider._stream_gemini: split thinking translation by
family. Gemini 3.x (3 / 3.1 / 3.5 + gemini-pro-latest /
gemini-flash-latest / gemini-flash-lite-latest) emits
thinkingConfig.thinkingLevel; effort none/off coerces to "low" on
Pro and "minimal" on Flash. Gemini 2.5 stays on thinkingBudget.
- external_provider._stream_gemini: allow `tools: [{googleSearch: {}}]`
on the Gemini 3 image family (gemini-3-pro-image-preview,
gemini-3.1-flash-image-preview, nano-banana-pro). Google's docs
document Search grounding on these. codeExecution stays blocked
on image mode (still mutually exclusive with responseModalities).
- provider-capabilities.ts: mirror the Gemini 3 effort ladders in
resolveGeminiReasoningCapabilities (Pro: low/medium/high; Flash:
minimal/low/medium/high; 2.5 Flash keeps the off-position).
- provider-capabilities.ts: providerSupportsBuiltinWebSearch now
returns true on the documented Gemini 3 image models so the pill
is reachable; older image ids (gemini-2.5-flash-image) still hide.
Tests: splits the existing thinkingBudget cases by family (Gemini 3
checks thinkingLevel; Gemini 2.5 keeps thinkingBudget), adds positive
googleSearch coverage for Gemini 3 image models and negative
googleSearch coverage for legacy image models.
References:
- https://ai.google.dev/gemini-api/docs/thinking
- https://ai.google.dev/gemini-api/docs/gemini-3
- https://ai.google.dev/gemini-api/docs/models/gemini-3-pro-image-preview
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: attach Gemini code_execution inline images to the code card (PR #5720)
When a text Gemini turn wires codeExecution and the sandbox produces a
matplotlib plot, the inline image part ships right after the
codeExecutionResult. Previously this surfaced as a separate empty
image_generation card. Track the most recent code_execution
tool_call_id + result text and, when an inline image follows with
code_execution active, emit a second tool_end on the same id that
appends the image as a data: URI under the `__IMAGES__:` marker the
chat-adapter already understands.
Image-picker turns (`-image` / `nano-banana`) keep the standalone
image_generation envelope so Nano Banana outputs render the same way.
Tests: covers the merged code-execution card emission with no
standalone image_generation event when code_execution is the active
tool.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix fourth-pass Gemini findings (PR #5720)
Round 4 review follow-ups:
Backend:
- `_is_openai_compatible` + `_auth_headers` detect Gemini connections
pointed at a custom OpenAI-compatible proxy (non-Google host whose
path ends in `/openai`) and route them through the OpenAI-compat
surface with `Authorization: Bearer ...` instead of the native
`_stream_gemini` translator + `x-goog-api-key`. Google-hosted Gemini
keeps the native dispatch path it migrated to in this PR.
- `_stream_gemini` thinkingLevel handling for Gemini 3 Pro now coerces
both "minimal" and "medium" effort to "low" / "high" respectively
(Pro tier only accepts low/high per
https://ai.google.dev/gemini-api/docs/thinking).
- `providers.py` `default_models` restores the advertised
`gemini-3.5-pro` and the rolling `gemini-pro-latest` /
`gemini-flash-latest` / `gemini-flash-lite-latest` aliases that the
allowlist already admits.
Frontend:
- chat-adapter: lean on `providerSupportsBuiltinWebSearch` (which
already encodes the Gemini 3 image-model Search allowance) instead
of blanket-disabling Search whenever Gemini image mode is active.
Code execution stays blocked because Gemini image mode rejects it.
- shared-composer: mirror the same gate -- only the Code pill is
unconditionally disabled in Gemini image mode; the Search pill is
driven by `supportsBuiltinWebSearch`.
- provider-capabilities: Gemini 3 Pro reasoning levels now expose only
"low" and "high" (no Medium pill) to match the API.
Tests: covers the Gemini 3 Pro medium / minimal coercion, the custom
proxy OAI-compat dispatch + Authorization Bearer auth, and the
native-vs-proxy detection. Also closes the mocked httpx.AsyncClient
inside the test event loop so the Python 3.13 `aclose was never
awaited` warning no longer fires.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix fifth-pass Gemini findings (PR #5720)
Round 5 review follow-ups:
Backend:
- `_is_openai_compatible` + `_auth_headers` now treat ANY non-Google
Gemini base URL as OpenAI-compat (LiteLLM / custom OAI gateways /
OpenAI-compat vLLM routers), not just paths ending in `/openai`.
Pre-existing saved Gemini proxies on `/v1` keep working.
- Gemini 3 thinkingLevel coercion narrowed to the documented
inconsistencies: only "minimal" is coerced to "low" on Pro tier.
"medium" passes through (Gemini 3.1 Pro accepts it per
https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/3-1-pro).
- `_stream_gemini` only flips `responseModalities=[TEXT,IMAGE]` when
the selected model is image-capable. A stale
`enabled_tools=["image_generation"]` on a text model is silently
dropped instead of producing an invalid Gemini request.
- `_stream_gemini` validates the model id against
`[A-Za-z0-9._-]+` before URL interpolation so a model like
`../cachedContents/x` cannot redirect the request to an unintended
endpoint with the configured API key attached.
- Empty-text Gemini parts that still carry `thoughtSignature` emit a
content-free delta with `extra_content.google.thought_signature` so
Gemini 3 turns that end with a signature-only fragment do not lose
the replay state.
- ConnectError / ReadTimeout / generic HTTPError paths in
`_stream_gemini` now close the synthetic web_search tool_start
with a matching tool_end before the error chunk so the UI does not
leave a stuck "searching..." card on transport failure.
- `providers.py` default_models drop the non-existent
`gemini-3.5-pro` (Google launched only `gemini-3.5-flash` at
I/O 2026; Pro tier remains `gemini-3.1-pro-preview`).
- `routes/inference.py` only forwards `payload.top_k` when the caller
explicitly set it on the request (Pydantic `model_fields_set`).
Omitted top_k stays omitted, restoring the pre-PR behavior where
Gemini uses its server default.
- `ChatCompletionRequest.enable_prompt_caching` adds a `mode="before"`
validator that coerces the canonical string literals "true"/"false"
back to bool so historical opt-out callers keep working after the
field widened to `Union[bool, str]` for Gemini cache resource names.
Frontend:
- `providerSupportsBuiltinWebSearch` / Code / Image now accept the
saved connection `baseUrl` and return false for custom OAI-compat
Gemini proxies. Backend skips `_stream_gemini` for those bases, so
native tool envelopes never reach them; hiding the pills keeps the
request, builder, and UI consistent.
- `provider-capabilities.ts` Gemini 3 Pro effort ladder restores
`["low", "medium", "high"]` to match Google's documented levels.
- Call sites in `chat-page.tsx` and `chat-adapter.ts` pass through
`provider.baseUrl` so the proxy gate fires.
Tests: covers Gemini 3 Pro medium pass-through, custom proxy dispatch
on `/v1` and `/openai` bases, path-traversal model id rejection,
top_k omission when not explicit, text-model image_generation drop,
empty-text + thoughtSignature surfacing, and
enable_prompt_caching string coercion.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix sixth-pass Gemini findings (PR #5720)
Round 6 review follow-ups:
Frontend:
- chat-adapter `delta.tool_calls` accumulates fragments by `id` /
`index` instead of pushing a new tool-call card per chunk. The
standard OpenAI Chat Completions stream contract sends `id`/`name`
on the first chunk and partial `function.arguments` on subsequent
chunks; our previous handler parsed each fragment as a standalone
tool call. Local llama.cpp and OAI-compat providers that stream
fragments now reassemble into a single function-call part.
- chat-adapter also preserves `extra_content` on streamed tool-call
deltas so Gemini 3 `thoughtSignature` survives to the next turn.
- provider-capabilities Gemini 3 Pro restores "medium" in the
reasoning-effort ladder (Google's official Gemini API thinking
doc lists low/medium/high for Gemini 3.1 Pro; my earlier round 4
coercion was wrong).
- provider-capabilities orders `gemini-2.5-flash-lite` ahead of the
broader `gemini-2.5-flash` prefix so Flash-Lite falls into the
"no native thinking knob" branch as documented.
* Studio: round-trip Gemini tool_calls and tool results (PR #5720)
Recurring round 3-6 P1: the chat-adapter renders Gemini function-call
parts and code-execution events but `toOpenAIMessage` only serialized
text + image content, so the next turn lost the assistant
`tool_calls[]` (including Gemini 3's required
`extra_content.google.thought_signature`) and the matching
`role="tool"` result. Gemini 3 multi-turn function calling and code
execution failed validation on the second turn.
Frontend:
- types/api.ts widens OpenAIChatMessage to permit `role="tool"`,
`tool_calls`, `tool_call_id`, `name`, and `content: null`. Adds
OpenAIToolCallPart with `extra_content` for the Gemini round-trip.
- chat-adapter: new `toOpenAIMessages` expands an assistant turn with
tool-call parts into [assistant w/ tool_calls + extra_content,
role=tool result, ...]. tool result content is JSON-serialized so
the backend translator can rebuild Gemini's `functionResponse`
shape.
- chat-adapter outbound history now uses `flatMap(toOpenAIMessages)`
so each assistant tool-call round-trips through the standard OAI
shape the backend's `_stream_gemini` already understands.
* Studio: replay Gemini code_execution and image native parts on history (PR #5720)
Multi-turn Gemini history previously lost the native executableCode,
codeExecutionResult, and inlineData parts because the outbound
translator regenerated a generic functionCall for every assistant
tool_call. Stow the native dict on tool_end (frontend) and replay it
verbatim with thoughtSignature (backend) so follow-up turns preserve
the prior execution and image generation state. Skip role="tool"
fan-out for server-side builtin tools so Gemini does not 400 on a
functionResponse with no matching user-declared function.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: complete Gemini built-in tool replay round-trip (PR #5720)
Round 7 follow-up to the multi-turn native-part work. Three asymmetric
storage/consume gaps remained between the backend translator and the
chat adapter, so realistic Gemini follow-up turns degraded to generic
functionCalls instead of native history.
- Frontend collectAssistantToolCalls now drops web_search outright,
drops code_execution / image_generation when the native part is
missing, and promotes args.google to extra_content.google so the
backend native_part replay branch actually fires.
- Backend image_generation tool_end now emits google.native_part
with the inlineData (mimeType + base64) and thoughtSignature so the
follow-up image-edit turn can replay the prior image as a native
Gemini model part.
- Backend code-execution plot tool_end now stows google.native_part
with the inlineData so the merged code-exec card can round-trip
executableCode + codeExecutionResult + inlineData on the same id.
- Added regression tests for image-gen native-part replay and the
code-exec plot native_part stow.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 8 Gemini follow-ups (PR #5720)
- Text-part thoughtSignature: stow on the assistant message during
streaming and replay onto the last text part on the next turn so
Gemini 3 strict function-calling does not reject history.
- Function declarations: recursively strip Gemini-unsupported OpenAPI
keys (additionalProperties, $schema, $defs, strict, etc.) so OpenAI
strict tools stop 400ing as INVALID_ARGUMENT on Gemini.
- OpenAI-compat fallback: forward tools/tool_choice so custom Gemini
proxies (LiteLLM, gateways) keep function-calling.
- enable_prompt_caching: cover the Pydantic v1 legacy off/on/f/n/t/y
string set so explicit opt-outs stay opt-out (Gemini was sending
cachedContent: "off" otherwise).
- Frontend collectAssistantToolCalls / collectToolResultMessages: use
google.native_part + result presence to disambiguate provider
builtins from same-named user-declared functions.
- Added regression tests for text-signature replay and schema
sanitization.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 9 Gemini follow-ups (PR #5720)
Two round-9 convergent finds across the 12 reviewers:
- Server-side web_search was leaking onto the next turn as a fake
user functionCall/functionResponse. The previous heuristic (skip
builtin only when no native_part AND no result) let it through
because the synthetic tool card has a non-empty result string.
Always skip web_search by name on both serializers, accept that a
user-declared function literally named "web_search" must use a
different name.
- Assistant `extra_content` was dropped by ChatMessage validation
before _stream_gemini could replay text-part thought signatures.
Add the field to ChatMessage and forward it through
_build_external_messages so the multi-turn signature path actually
carries data.
Includes a regression test for the ChatMessage round-trip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 10 Gemini follow-ups (PR #5720)
Three convergent round-10 reviewer findings closed:
- Tag synthetic provider-side builtins with `args._server_tool=True`
via a central helper that runs in every `_emit_tool_event` /
`_emit_synthetic_tool_event` path. The frontend filter now skips
on that marker instead of on the public tool name, so local
llama.cpp `web_search` and OpenAI function tools literally named
`web_search` / `code_execution` / `image_generation` round-trip
cleanly while Gemini grounding / hosted code-exec / hosted image
cards stay skipped.
- Gate Gemini image-mode (responseModalities=[TEXT,IMAGE]) on the
Images pill (enabled_tools containing `image_generation`).
Selecting an image-capable model with the pill off no longer forces
image output the UI says is disabled.
- Frontend missing-key guard now exempts custom Gemini OAI-compat
proxies (LiteLLM, gateways) the same way the backend already
does, so a saved Gemini connection on `http://localhost:4000/v1`
with no API key stops being blocked.
Existing tests updated to pass `enabled_tools=["image_generation"]`
on image-mode capture paths.
* Studio: round 11 Gemini follow-ups (PR #5720)
Four round-11 findings closed:
- Kimi _stream_kimi_web_search's local _synthetic_chunk helper now
runs through _stamp_server_tool_marker so Kimi search history is
not replayed as a fake user functionCall on the next turn (was an
asymmetric miss after the round-10 tagging work).
- OpenAI Responses path (/v1/responses for gpt-5.x) forwards
caller-supplied tools / tool_choice, translating the Chat
Completions function-tool shape into the Responses native shape.
Without this, standard OpenAI tools silently dropped on
Responses-routed traffic.
- Decoupled the Gemini image-tier model-id guards (text-tool /
thinking strip) from the Images pill flip
(responseModalities=[TEXT,IMAGE]). gemini-2.5-flash-image with
Search/Code on and the Images pill OFF no longer forwards
googleSearch + thinkingConfig (Gemini 400s on those for legacy
image ids).
- Gemini-only extra_content is now forwarded by
_build_external_messages only when provider_type=="gemini" so
Google's thought_signature does not leak into OpenAI / Mistral /
Kimi / OpenRouter request bodies as an unknown field.
Added a regression test for the image-tier strict-guard split and
extended the extra_content test to cover the non-Gemini suppression.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 12 Gemini follow-ups (PR #5720)
Three round-12 convergent findings closed:
- extra_content leak to custom Gemini OAI-compat proxies (8/12
reviewers). _build_external_messages now gates extra_content on
the native generativelanguage.googleapis.com host, not just
provider_type=="gemini", so LiteLLM / custom gateways routed
through /chat/completions do not get an unknown top-level field.
- OpenAI Responses function-tool round-trip (5/12 reviewers). I
added user `tools` forwarding in round 11 but did not parse the
matching response.output_item.done items of type=function_call.
The parser now translates them into Chat Completions
delta.tool_calls and the terminal chunk reports
finish_reason="tool_calls" when the model invoked a user
function.
- Image-tier model with Images pill OFF (2/12). Google's image
models default to text+image when responseModalities is omitted,
so the previous fix silently still billed image output. Force
responseModalities=["TEXT"] when the Images pill is off and the
selected model is image-capable.
Updated the two pre-existing tests that pinned the synthetic-tool
arguments shape to include the new `_server_tool: True` marker, and
added a regression test for the Responses function-call output
translation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 13 Gemini/Responses follow-ups (PR #5720)
Three round-13 convergent findings closed:
- OpenAI Responses function_call indices: my round-12 translator
hardcoded every emitted tool_calls[*].index to 0, so parallel
function calls collapsed for index-keyed clients. Track and
increment function_call_index per emit (mirrors the Gemini
branch's distinct-index pattern). 10/12 reviewers flagged.
- _SERVER_SIDE_BUILTIN_TOOL_NAMES now includes web_fetch so
Anthropic-hosted web_fetch cards carry the _server_tool marker
and the frontend history serializer doesn't replay them as fake
user functions. 4 reviewers flagged.
- OpenAI Responses follow-up tool results now serialize as
Responses-shape function_call / function_call_output items keyed
by call_id, instead of Chat Completions role="tool" content.
Skips assistant tool_calls tagged with _server_tool so hosted
builtins don't round-trip as user functions. 2 reviewers flagged.
Updated the Anthropic code_execution and web_fetch test argument
pins to include the new _server_tool marker, and added two
regression tests (distinct indices on parallel function_call,
function_call_output round-trip).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 14 Gemini follow-ups (PR #5720)
Three round-14 findings closed:
- Remote `image_url` translation (5 reviewers convergent). Public
HTTPS image URLs can't be sent as `fileData.fileUri` -- Gemini
reserves that path for Files API URIs and YouTube. Fetch the
bytes server-side and inline them as base64 `inlineData`,
mirroring the pre-PR OpenAI-compat behaviour. YouTube URLs and
generativelanguage.googleapis.com/v1beta/files/* stay as
`fileData`.
- Nullable JSON Schema type arrays. OpenAI strict tools commonly
use `"type": ["string", "null"]`; the Gemini sanitizer now
flattens that to `"type": "string", "nullable": true` so strict
function tools stop 400ing.
- Parallel functionResponses now ride on one user content block
with multiple `functionResponse` parts, matching Google's
parallel tool docs. Consecutive `role="tool"` messages merge
into the previous user turn instead of splitting into separate
Gemini user turns.
Three regression tests added (remote URL fetch + inline, Files
API / YouTube fileData preservation, schema nullable flattening,
parallel-tool grouping).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: SSRF harden Gemini remote image fetch (PR #5720)
Round 15 convergent finding (12/12 reviewers). My round-14 fix to
download user-controlled image URLs for inlineData inlining was an
SSRF / data-exfiltration path: no scheme check, no private-host
guard, no size cap, no Content-Type validation, redirects could
bounce to internal services, and the full URL was logged.
Replace the inline fetch with `_safe_fetch_image_for_gemini`:
- Require https:// (reject http, file, data, ftp, etc).
- Resolve the hostname via socket.getaddrinfo and reject if ANY
resolved address is private / loopback / link-local / multicast /
reserved / unspecified (covers 127.0.0.0/8, 10/8, 172.16/12,
192.168/16, ::1, 169.254/16 metadata, RFC 6890).
- Block IP-literal URLs that resolve into those same ranges.
- Cap response body at 10 MB (Content-Length pre-check + streamed
byte counter).
- Require Content-Type to start with `image/`.
- Disable redirect following so a 302 to a private host can't slip
past the address check.
- Use a short 15s timeout and a tiny connection pool dedicated to
these fetches.
- Log only the host name + error class -- no full URL, no signed
querystring leak.
If the guard rejects, the image part is silently dropped (instead
of forwarding raw bytes or a fileData fallback). Files API URIs
and YouTube URLs still ride as `fileData.fileUri` unchanged.
Tests: replaced the live-fetch test with a `_safe_fetch_image_for_gemini`
monkeypatch, added four new SSRF-guard tests (non-https rejected,
loopback / private IP literals rejected, hostnames that resolve to
private IPs rejected).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 16 Gemini follow-ups (PR #5720)
- IP-pinned image fetch (`_safe_fetch_image_for_gemini`): reuse the
validated-once-then-pin pattern from `tools._fetch_page_text` via
`asyncio.to_thread`, so DNS rebinding between validation and the
HTTP connect cannot redirect us at a private/metadata address.
Catch malformed-bracketed IPv6 urlparse errors. Follow up to 4
redirect hops with per-hop SSRF re-validation.
- Replace contains-substring detection of Gemini Files API + YouTube
URLs with parsed scheme/host/path checks, so attacker URLs like
`https://evil.example/path/youtube.com/x.png` no longer skip the
safe-fetch path and serialize as `fileData.fileUri`.
- `_build_external_messages`: strip per-tool-call `extra_content`
for non-native-Gemini providers; the Gemini-only
`thought_signature` payload was leaking through `tool_calls[]`
into /chat/completions on OpenAI, Anthropic, and custom Gemini
OAI-compat gateways.
- `_server_tool` marker now gated on the function name being one of
the canonical builtin names (`web_search`, `web_fetch`,
`code_execution`, `image_generation`) AND the marker being set,
so a user function whose schema happens to define an
`_server_tool` field is no longer dropped. Frontend filter mirrors
the same gate, plus a backward-compat fallback for pre-PR
persisted server-tool cards (no marker) routed via name +
native_part / web-tool heuristic.
- Gemini schema sanitizer collapses `anyOf: [{X}, {"type":"null"}]`
to `{X, "nullable": true}` so Optional[X] tool args from
OpenAI/Pydantic schemas no longer 400 the Gemini request.
- Frontend tool-result serializer emits `{"result":""}` for empty
string outputs so the ChatMessage validator does not reject
`role="tool"` with empty content.
- Coerce `medium` thinkingLevel to `high` for legacy
`gemini-3-pro*` / `gemini-3-pro-preview*` (only low/high
documented; shut down 2026-03-09); 3.1+ Pro still passes through.
- Hide Gemini native thinking ladder on custom OAI-compat Gemini
gateways by routing `getExternalReasoningCapabilities` through
`isGeminiCustomOpenAICompatBase(baseUrl)`; thread baseUrl through
all four call sites.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 17 Gemini follow-ups (PR #5720)
- Frontend `collectAssistantToolCalls` and `collectToolResultMessages`
no longer drop unmarked `web_search` / `web_fetch` cards by name
alone: a user-defined function with one of those names must
round-trip. Pre-PR persisted `code_execution` / `image_generation`
cards still get filtered via a shape heuristic (kind/command/code/
prompt fields) instead of bare name.
- `_build_external_messages._filter_tool_calls` now drops marked
server-side builtin `tool_calls` entirely for non-native-Gemini
providers, not just their `extra_content`. An assistant turn whose
only payload was a marked builtin is dropped completely so the
receiving provider does not see an orphan tool_call.
- `_stream_anthropic` translates OpenAI top-level `tool_calls` into
Anthropic native `{type:"tool_use", id, name, input}` content
blocks, and translates `role="tool"` follow-ups into `role:"user"`
messages carrying a `tool_result` block. Anthropic's native
Messages API rejects the OpenAI shapes.
- `_safe_fetch_image_for_gemini_sync` factors URL validation through
`_safe_parse_https`, so malformed `port` access (e.g.
`https://host:bad/x.png`) and malformed redirect targets (e.g. a
302 to `https://[bad/x.png`) drop the image instead of raising mid-
request.
- `tool_choice="none"` now disables hosted builtins (Gemini
googleSearch / codeExecution and OpenAI Responses web_search /
shell / image_generation), not just user function declarations.
- Schema sanitizer handles multi-type `anyOf` with null
(`Union[str, int, None]`): keep the slim non-null anyOf and add
`nullable: true` so Gemini does not reject `{"type":"null"}`.
- Image fetch falls back to the caller-provided MIME (guessed from
URL extension) when the server omits Content-Type instead of
dropping the image as `non-image content-type=<none>`.
- Per-request aggregate caps on remote image inlining (8 images,
20MB total) so a single chat request cannot force unbounded
backend downloads.
- Frontend exposes the reasoning ladder for `gemini-2.5-flash-lite`
(`none/minimal/low/medium/high/max`) so the UI can drive the
thinkingBudget the backend already supports.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 18 Gemini follow-ups (PR #5720)
- `tool_choice="none"` now opts out of hosted builtin tools on every
provider path, not just Gemini and OpenAI Responses. Anthropic
web_search / web_fetch / code_execution, Kimi `$web_search` early
return, and OpenRouter `plugins:[{id:"web"}]` are all gated on
`tool_choice_disabled`. Passing `enabled_tools=[...]` with
`tool_choice="none"` no longer triggers provider-side search /
code execution for any provider.
- `_stream_anthropic` accepts `tool_choice` and threads it through;
the dispatcher in `stream_chat_completion` forwards it.
- Frontend `isServerSideBuiltinToolPart` simplified to drop only on
(marker) OR (canonical name + native_part). The previous shape
heuristic on `args.kind`/`args.command`/`args.code`/`args.prompt`
dropped real user-declared `code_execution` / `image_generation`
functions. Pre-PR persisted hosted cards lacking the marker now
leak to non-native providers on switch -- preferred to silently
deleting legitimate function-call history.
- Backend `_is_marked_server_builtin_tool_call` and the OpenAI
Responses translator's matching filter accept BOTH `_server_tool`
marker AND `args.google.native_part` as durable provider-side
signals so Gemini code_execution / image_generation cards are
still dropped on a provider switch.
- Per-request remote image count cap now counts ATTEMPTS, not just
successful inlines, so 100 failing/slow URLs cannot each consume
the 15s fetch timeout. Data: URL images now share the same count
and byte caps as fetched remote URLs.
- OpenAI Responses translator tracks skipped server-builtin
`function_call` ids and drops their matching `role="tool"`
follow-ups, preventing orphan `function_call_output` items in the
outbound body.
- Gemini schema sanitizer preserves multi-type unions with null:
`{"type":["string","integer","null"]}` becomes
`anyOf:[{string},{integer}] + nullable:true` instead of being
flattened to the first non-null type.
- Gemini model id validation moved to the top of `_stream_gemini`
so an invalid model id rejects the request before any remote
image fetch / message translation side effect.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 19 Gemini follow-ups (PR #5720)
- `_build_external_messages` now skips an empty assistant turn when
`_filter_tool_calls` drops every synthetic builtin tool_call (was
guarded only on the `content is None` branch; the string-content
and list-content branches still forwarded
`{"role":"assistant","content":""}` which several providers
reject). Also tracks the dropped server-builtin tool_call ids and
skips the matching `role="tool"` follow-ups so the receiving
provider does not see an orphan tool_result.
- OpenRouter `web_search_active` (the synthetic tool_start /
tool_end emitter) is now also gated on `tool_choice_disabled` so
a request with `tool_choice="none"` does not surface a fake
web_search card in the chat UI even though the plugin was
correctly stripped from the outbound body.
- `_stream_anthropic` translates an OpenAI role="tool" with list
content (`content=[{"type":"text","text":"..."}]`) into a native
`tool_result` block on a user message; previously only the
string-content shape was translated, so list-content tool results
were forwarded as invalid `role:"tool"` messages.
- Gemini `data:` URL image_url parts now require an `image/*` MIME
type; a `data:text/html;base64,...` is dropped instead of being
forwarded as `inlineData.mimeType="text/html"` (Gemini rejects
the malformed image part). Symmetric with the fetched-remote
image fetch path that already rejects non-image Content-Type.
- YouTube `fileData.fileUri` now declares `video/mp4` as the
mimeType instead of `image/jpeg` guessed from the URL path. The
YouTube/fileData input is the documented Gemini video path; the
guessed image MIME made valid YouTube inputs malformed.
- OpenAI Responses translator preserves `response.output` ordering
on assistant turns that emitted both text and a function_call:
assistant text is now serialized BEFORE the function_call item
so the subsequent function_call_output (the matching role=tool
follow-up) lands in the right position. Previously the order
was function_call -> assistant text -> function_call_output,
which can confuse multi-turn function-calling flows.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 20 Gemini follow-ups (PR #5720)
Convergent reviewer findings from round 20:
- tool_choice="none" no longer flips responseModalities=[TEXT,IMAGE]
on image-tier Gemini models. Forced-function tool_choice (e.g.
{type:function, function:{name:lookup}}) also drops hosted Search /
code execution from the Gemini body so the caller's pinned user
function is not silently joined by hosted builtins.
- Gemini code-execution thoughtSignature replay now uses an ordered
parts list (native_part.parts[]) so per-part signatures stay
attached to the exact part Gemini emitted. The previous merged
shape fanned one top-level thoughtSignature across executableCode
+ codeExecutionResult + inlineData and tripped Gemini 3 strict
validators. Backward-compat fallback keeps pre-round-21 persisted
history working: a legacy native_part with a single subpart still
replays the signature on that subpart; merged legacy objects pin
the signature to executableCode only.
- Remote-image fetch threads the remaining per-request byte budget
into _safe_fetch_image_for_gemini, so over-budget URLs are
refused via Content-Length pre-check / short read instead of
fully downloaded then discarded after the aggregate cap check.
- Gemini role=tool with OpenAI list-form content
([{type:text,text:result}]) now flattens text parts before
building functionResponse.response.result; previously the parts
arrived as the result value instead of the actual tool output.
- Frontend chat-adapter merges native_part by concatenating parts
lists (preserving per-part thoughtSignature). Wire types expose
enable_prompt_caching as boolean|string (Gemini cached-content
name) and OpenAIChatDelta now carries tool_calls and extra_content.
- Test test_openrouter_no_synthetic_web_search_event_on_tool_choice_none
reads _toolEvent from the top-level SSE payload so a backend
regression cannot mask the assertion.
Adds 7 regression tests covering image_generation gate, forced-function
gate, native_part list replay, legacy fallback, list-content
functionResponse flattening, fetch byte-budget threading, and wire
types.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Apply forced-function tool_choice gate to Anthropic, OpenRouter, Kimi
Previously only the Gemini path treated `tool_choice={"type":"function",
"function":{"name":...}}` as a hosted-tool opt-out. Anthropic,
OpenRouter, and Kimi still attached hosted web_search / web_fetch /
code_execution when the caller explicitly pinned a user function plus
`enabled_tools=[...]`. That contradicts the explicit function pin and
bills the caller for unwanted server-side calls.
Mirror the Gemini gate symmetrically:
- Anthropic web_search / web_fetch / code_execution
- OpenRouter `plugins:[{id:"web"}]` + the synthetic web_search SSE
event the same path emits at stream close
- Kimi `_stream_kimi_web_search` dispatch
Adds 4 regression tests:
- test_anthropic_forced_function_tool_choice_drops_hosted_tools
- test_openrouter_forced_function_tool_choice_drops_web_plugin
- test_kimi_forced_function_tool_choice_skips_web_search_helper
- test_openrouter_no_synthetic_web_search_event_on_forced_function_tool_choice
All 146 existing backend tests still pass.
* Strip Gemini-only synthetic tool history on local-GGUF dispatch
After a Gemini chat that ran code_execution / image_generation, switching
the same thread to a local GGUF model used to forward the synthetic
provider-side tool_calls (tagged with `args._server_tool` or carrying a
Gemini `args.google.native_part` payload) and the message-level
`extra_content` to llama-server. The receiving backend has no tool
declaration for those names and no use for Gemini thoughtSignature
metadata; in the worst case it can produce an orphan tool_call_id and a
confused continuation.
Add `_strip_provider_synthetic_tool_history()` and wire it through the
two local message builders:
- `_openai_messages_for_passthrough` (OAI-compat passthrough)
- `_openai_messages_for_gguf_chat` (standard GGUF chat path)
Real user-function `tool_calls` and their matching `role="tool"` replies
survive unchanged; only synthetic provider-side cards and Gemini-only
`extra_content` are stripped. If the synthetic call was the assistant
turn's only payload, the now-empty turn is dropped too so llama-server
does not reject the request.
Adds 2 regression tests:
- test_strip_provider_synthetic_tool_history_drops_synthetic_only
- test_strip_provider_synthetic_tool_history_drops_empty_assistant
142 existing backend tests still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Disable Search/Code composer pills for Gemini image-tier models
For external Gemini image-tier models (gemini-2.5-flash-image,
gemini-3.x-image-preview, etc.), the backend unconditionally strips
code_execution and strips web_search on older image ids. Search is
still allowed on Gemini 3.x Pro/Flash image models, which
supportsBuiltinWebSearch already encodes per model.
Before this commit the composer pill gates were:
searchDisabled = !modelLoaded || !(supportsTools || supportsBuiltinWebSearch)
codeDisabled = !modelLoaded || !(supportsTools || supportsBuiltinCodeExecution) || imageModeDisablesCode
`supportsTools` here is a local-runtime fallback that becomes true when
any tool-capable local model has been loaded in the session. With a
local tool-capable runtime active, switching the chat to an external
Gemini image-tier model used to leave Search/Code clickable, even
though the backend will silently drop the tool on the wire.
Detect "external provider is Gemini AND the model is image-tier" (via
supportsBuiltinImageGeneration) and gate the two pills strictly on the
provider's own builtin support in that case. Non-Gemini paths and
non-image Gemini models keep the supportsTools fallback unchanged.
* Apply forced-function tool_choice gate to OpenAI Responses path
Round 22 added the gate for Gemini / Anthropic / OpenRouter / Kimi but
missed the OpenAI Responses translator. When a caller pinned a user
function via `tool_choice={"type":"function","function":{"name":...}}`
plus `enabled_tools=["web_search","code_execution","image_generation"]`,
the Responses body still attached `{"type":"web_search"}`,
`{"type":"shell"}`, and `{"type":"image_generation"}` server tools. The
function pin should suppress those for the same privacy + billing reason
the other provider paths now do.
Compute `_responses_tool_choice_forced_function` next to
`_responses_tool_choice_none` and gate each hosted-tool append on
`_responses_hosted_builtins_allowed = not none and not forced_function`.
The fix has to be applied in TWO places: the initial body builder and
`_build_body()` (called by the container-expiry retry path). User
function declarations still flow through so the pin has something to
target, and the Responses-shape `{type:"function", name:"..."}`
`tool_choice` is forwarded unchanged.
Adds regression test `test_openai_responses_forced_function_tool_choice_drops_hosted_tools`.
All 166 existing backend tests across Gemini + Responses + image-gen +
code-exec suites still pass.
* Round 24 P1s: SSRF shared-address gap + extra_content text-only leak + custom-Gemini model list
Three convergent P1s from round 24 review:
1. SSRF: the shared SSRF validator in `tools._validate_and_resolve_host`
used a denylist (is_private / loopback / link_local / multicast /
reserved / unspecified). Python classifies shared address space
(100.64.0.0/10 carrier-grade NAT, plus 240.0.0.0/4, benchmarking
ranges, etc.) with `is_private=False` AND `is_global=False`. The new
Gemini server-side image fetcher therefore accepts URLs whose
hostname resolves to 100.64.0.1 in cloud/VPC deployments. Add
`not ip.is_global` as the primary gate -- a single source of truth
that covers every current and future non-global range.
2. _strip_provider_synthetic_tool_history previously only stripped
message-level `extra_content` when the assistant turn had tool_calls.
A plain text Gemini reply carrying
`extra_content.google.thought_signature` flowed through to
llama-server when the thread was switched to a local GGUF backend.
Always strip message-level `extra_content` on assistant turns.
3. routes/providers.list_provider_models applied Gemini's native
`model_id_allowlist` regex to every Gemini provider, including
custom OAI-compatible bases (LiteLLM, deployment gateways). IDs like
`google/gemini-2.5-flash` and team-prefixed deployment aliases got
filtered out even though the chat-dispatch path now routes them via
the OpenAI-compatible client. Skip registry-level model-id filters
when the configured Gemini base_url host is not the canonical
`generativelanguage.googleapis.com`, mirroring the chat-dispatch
gate.
Three regression tests added:
- test_validate_and_resolve_host_blocks_shared_address_space
- test_strip_provider_synthetic_tool_history_drops_text_only_extra_content
- test_gemini_custom_oai_compat_base_skips_native_allowlist
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Round 25 P1s: skip synthetic server-tool replay + inline $ref/$defs into Gemini schema
Two convergent reviewer findings on the native Gemini path:
1. _stream_gemini's tool_calls replay loop falls through to a generic
functionCall emission whenever it sees an assistant tool_call. Marked
server-side builtin cards (web_search / web_fetch tagged with
_server_tool or args.google.native_part) hit that fallthrough with no
replayable native_part, which produces an outbound functionCall whose
name is not a declared user function. The Gemini turn 400s on the
undeclared name. Guard the loop to drop those entries instead, while
keeping the existing code_execution / image_generation native-part
replay branch intact.
2. _sanitize_gemini_schema uses a strict allowlist that drops local
$ref / $defs references. Pydantic-generated tool schemas hoist nested
object shapes into $defs and reference them via {"$ref": "#/$defs/X"},
so a property like address: {"$ref": "#/$defs/Address"} collapsed to
{} on the wire and the model lost the nested fields, types, and
required keys. Resolve local #/... pointers against the schema root
and inline the referenced subtree, with local siblings overriding
the reference (normal JSON Schema composition) and a seen-ref guard
for self-referential schemas.
Added regression coverage:
- test_gemini_native_skips_synthetic_server_builtin_replay
- test_function_declarations_inline_local_refs_into_gemini_schema
- test_function_declarations_inline_local_refs_in_anyof_and_items
- test_function_declarations_self_referential_schema_terminates
All 145 Gemini provider tests pass; touched provider regression set
(OpenAI Responses, code execution, image generation, Anthropic code
execution, Anthropic web_fetch) also 43/43 green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Round 26 P1s: drop orphan Gemini functionResponse + Anthropic /messages synthetic-history strip
Reviewer round 26 surfaced two convergent asymmetric-fix bugs.
1. _stream_gemini drops a synthetic server-tool tool_call (web_search /
web_fetch tagged _server_tool) and also replays code_execution /
image_generation tool_calls as Gemini-native executableCode /
codeExecutionResult / inlineData parts. The matching role="tool"
follow-up was still falling through to the generic functionResponse
branch, producing either an orphan functionResponse (synthetic case)
or a duplicate response pointing at a name with no
functionDeclarations entry (native-part case). Both forms 400 the
next Gemini turn. Track skipped + native-replayed tool_call_ids in
_gemini_skip_tool_result_ids and short-circuit the role="tool"
branch on a match.
2. The Anthropic-compatible local /v1/messages route only called
_drop_empty_assistant_sentinels on the OpenAI-translated history,
while the sibling /v1/chat/completions and GGUF passthrough builders
chain that with _strip_provider_synthetic_tool_history. An Anthropic
caller replaying a prior provider-side tool_use therefore forwarded
fake builtin tool history straight into local llama-server. Apply
the same strip on the Anthropic route after the
anthropic_messages_to_openai conversion.
Regression coverage added:
- test_gemini_native_skips_orphan_function_response_for_dropped_builtin
- test_gemini_native_skips_orphan_function_response_for_native_part_replay
Gemini suite 147/147; touched provider regression set 43/43.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Round 27 P1s: native_part location fallback + Gemini image request budget for base64
Two convergent reviewer findings on the native Gemini path.
1. _stream_gemini's synthetic-builtin detector at lines 3519-3524
recognizes args.google.native_part as a server-tool marker, but
_native_part was only loaded from tc.extra_content.google.native_part.
A direct OpenAI-compatible API caller or imported third-party thread
round-trips the payload through function.arguments because
tool_calls[].extra_content is not in the OpenAI spec. The round-25
guard then saw a synthetic builtin with no _native_part and dropped
the entire assistant turn, so the next native Gemini request lost
the prior executableCode / inlineData / codeExecutionResult context.
Fall back to args.google.native_part when extra_content path is
missing, mirroring what the synthetic detector already accepts.
2. _GEMINI_REMOTE_IMAGE_MAX_TOTAL_BYTES capped DECODED bytes at 20MB.
Gemini receives images base64-encoded inside JSON, and base64
inflates payload size by ~4/3. With 20MB decoded the actual JSON
body is ~26.7MB plus prompt overhead, well over Gemini's ~20MB
request limit. Drop the decoded cap to 14MB so realistic multi-
image turns stay safely under 20MB encoded.
Added regression test test_gemini_native_part_falls_back_to_args_google
covering an OpenAI-compat-shaped image_generation tool_call whose
native_part lives only in function.arguments.
Gemini suite 148/148.
* Fix TS build errors from main merge: restore imageParts + refusal return [] + cast image-edit ref
Three errors in chat-adapter.ts surfaced by the frontend tsc step after merging
main into feat/gemini-provider:
1. The Anthropic refusal early-return used main's but
toOpenAIMessages returns SerializedMessage[]; flip to .
2. Restore -- the line
was lost when removing main's conflict block from the function body.
3. selectedImageEditReference splice was inserting OpenAIChatMessage
into a SerializedMessage[] array; the shapes differ on tool_calls.id
nullability. Cast the reference message through unknown -- it carries
no tool_calls, so the runtime payload is structurally compatible.
Reproduced locally with `tsc -b --pretty false` (now passes). Build
also failing in the in-repo `npm run build` step on PR CI; this commit
unblocks all 12 failing UI/API workflows.
* Tighten verbose comments in external_provider.py + chat-adapter.ts
Compress multi-line explanatory comments in the Gemini translator
and the chat adapter without changing any behaviour. All 148 Gemini
provider tests still pass; tsc --noEmit clean.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com>
* feat(recipes): round-trip local model variants
* feat(recipes): add local model selector
* feat(recipes): wire selector into model editors
* fix(recipes): clear stale model state on relink
* feat(recipes): load selected local models for jobs
* chore(frontend): simplify biome scripts
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(recipes): handle local selector edge cases
* fix(recipes): polish local model selector behavior
* fix(recipes): delay local model restore until terminal runs
* fix(recipes): accept resolved default gguf variants
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: add Anthropic fast_mode toggle + surface streaming refusals
Fast mode (beta `fast-mode-2026-02-01`) lets Claude Opus 4.6 and 4.7
generate output tokens up to 2.5x faster at 6x standard Opus
pricing. The toggle lives in Configuration → Provider when the
selected Anthropic model is Opus 4.6 or 4.7 and is otherwise
hidden. Backend gates the same prefixes a second time so a stale
frontend cannot make Anthropic 400 the request, and the
`fast-mode-2026-02-01` beta header is merged onto whatever other
betas the request already needed (code-execution, compaction).
Streaming refusals (`message_delta.delta.stop_reason="refusal"` on
Claude 4 models) now surface a short user-facing notice in the
assistant message before the translated OpenAI chunk emits the
existing `finish_reason="content_filter"`. Previously the chat
bubble truncated silently because the SSE stopped mid-stream with
no visible explanation. Per the upstream docs the conversation
must be reset before continuing, so the notice tells the user
exactly that.
Reference:
- https://platform.claude.com/docs/en/build-with-claude/fast-mode
- https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/handle-streaming-refusals
Tests:
- studio/backend/tests/test_anthropic_fast_mode_and_refusal.py (8 cases
pinning fast_mode pass-through on 4.6/4.7, silent drop on Sonnet /
Haiku / older Opus / None / False, and the refusal notice + finish
reason on a synthetic refusal stream).
* Studio: drop refused Anthropic turns from the next request
Anthropic's streaming-refusal guidance says the refused assistant
turn must be removed or updated before the next call -- otherwise
the safety classifier keeps refusing. The PR only added a
user-visible notice; the partial assistant output (plus the notice
itself) still rode the next request via toOpenAIMessage.
Tag the refusal turn with an HTML-comment sentinel emitted alongside
the notice. The chat-adapter checks for that sentinel in
toOpenAIMessage and returns null, so the refused turn is excluded
from outboundMessages. The notice still renders in the transcript
(HTML comments don't display), so users keep the explanation.
* Studio: filter None finish_reason entries in test helper
test_refusal_maps_to_content_filter expects only ['content_filter']
in the finish_reasons list, but the post-PR refusal path emits a
user-visible content notice chunk first. Every _content_chunk
carries 'finish_reason: None' by construction; the helper was
appending those, so the assertion saw [None, 'content_filter']
instead of ['content_filter'].
None is not a finish reason -- it's just mid-stream delta noise.
Skip None values in _finish_reasons so the helper reflects what
the test names actually claim to check. Same fix applies cleanly
to the other helper usages (pause_turn test expects [] and the
sibling stop test expects ['stop'], both unaffected).
* Studio: cover Anthropic fast-mode edge cases
Adds 19 cases on top of the 9 in test_anthropic_fast_mode_and_refusal.
The base file pins the happy path; this file fills in the cliffs:
* Dated-snapshot prefix matching: claude-opus-4-7-2026-02-01 and
claude-opus-4-6-2026-02-01 still gate fast_mode through, while
claude-opus-4-5-2025-08-01 and claude-sonnet-4-6-2026-02-01 do not.
* Strict opt-in: a future claude-opus-4-8 or claude-opus-5 does NOT
auto-enable fast_mode -- the prefix tuple must be bumped explicitly
when a new family is whitelisted upstream.
* Beta-header merge: fast_mode coexists with code-execution-2025-08-25
and compact-2026-01-12 in one comma-separated anthropic-beta header
with no duplicates and no truncation. Pins the value to the exact
fast-mode-2026-02-01 docs token so a typo would fail CI.
* Non-destruction: fast_mode=None produces byte-identical outbound
body and headers to the version that omits the argument entirely.
Same for fast_mode=False. Guarantees the upgrade path is
non-breaking on existing Anthropic streams.
* Refusal stream ordering: the user-visible notice precedes the
finish_reason chunk so a streaming UI paints text before flipping
to content_filter. Refusal sentinel emitted exactly once. Notice
rides a normal content delta chunk with finish_reason still null.
Partial assistant deltas survive before the notice.
* Provider-side refusal coverage: a refusal on Sonnet (not just Opus)
still emits the notice + sentinel + content_filter mapping, since
refusal handling is not gated on fast-mode capability.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Persist fastMode, drop refused user message on retry
Two follow-ups on #5715:
1) sanitizeInferenceParams stripped fastMode. fastMode is in
PERSISTED_INFERENCE_PARAM_KEYS but the storage sanitizer only kept
numeric fields plus systemPrompt and trustRemoteCode, so the new
toggle was silently dropped on reload and on the
/api/chat/settings round-trip. Save it the same way trustRemoteCode
is saved.
2) Refusal recovery now also drops the triggering user turn.
Returning null from toOpenAIMessage on the assistant side left the
user prompt that caused the refusal in the outbound history, so
the very next request would re-trigger the same classifier.
Anthropic's refusal-handling guidance is explicit on this: remove
the refused turn AND the user message that triggered it before
the next call. Implemented via a pre-pass that pops the trailing
user message when an assistant carries the refusal sentinel.
Typecheck clean.
* Studio: out-of-band refusal signal + fast-mode prefix/usage/pricing fixes
The text sentinel for the Anthropic refusal drop signal was spoofable:
any assistant message containing the literal
<!--studio:anthropic-refusal--> would prune the prior user + assistant
pair on the next request. Move the signal onto a separate _toolEvent
chunk that the chat adapter latches into
assistant.metadata.custom.anthropicRefusal; assistant text can no
longer control the pruner.
Tighten the fast-mode model gate (backend + frontend) to require a "-"
family boundary so claude-opus-4-70 / claude-opus-4-7b style IDs do
not get speed: "fast" on a naive startswith match.
Use survivingMessages for the image / audio attachment scan so a
refused user turn does not gate or mis-attribute the next non-refused
turn.
Propagate Anthropic usage.speed onto the OpenAI-style usage chunk and
apply the documented 6x fast-mode multiplier in the cost calculator
(stacks with prompt-cache multipliers per the docs); expose the new
multiplier on the pricing snapshot for the UI tooltip.
Tests cover the tool-event chunk shape, the prefix-collision rejects,
usage.speed propagation, the 6x pricing math, and that the visible
refusal text carries no embedded sentinel.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shorten fast-mode and refusal comments for PR #5715
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: PDF / document attachments for Anthropic + OpenAI
Studio's local-GGUF chat already supports image attachments via the
`image_url` content part shape. PDFs and other documents had no
plumbing for the external-provider path: there was no normalised
content type the frontend could send that translated to Anthropic's
native `document` block or OpenAI's `input_file`.
Add a Studio-side `input_document` content part on assistant /
user messages with three shapes:
{type: "input_document",
file_data: "data:application/pdf;base64,<DATA>",
filename?: "name.pdf",
media_type?: "application/pdf"}
{type: "input_document",
file_url: "https://example.com/doc.pdf",
filename?: "doc.pdf"}
Translation:
- Anthropic Messages API: emits a `document` block with
`{source: {type:"base64", media_type, data}}` or
`{source: {type:"url", url}}`, plus an optional `title` from
`filename`. PDFs are extracted server-side by Anthropic per their
vision/document docs and counted toward input tokens.
- OpenAI Responses API: emits `{type:"input_file", file_data |
file_url, filename?}`. PDFs are extracted server-side.
Empty / unparseable `input_document` parts are silently dropped so
a malformed frontend payload can't blow up the request.
Tests:
- New `test_multimodal_document.py` with 6 cases pinning the
outbound body shape for base64 + URL inputs on both providers,
and the empty-part drop behavior on both.
- The Anthropic assertions strip the prompt-cache wrapper
(`cache_control:{type:ephemeral}` that the tail-message caching
layer adds) before comparing the document core fields, so this
test stays focused on the translation, not the caching layer.
Live verified end-to-end against both providers: a 363-byte
single-page "HELLO" PDF, base64-encoded, attached as a `document`
block to Opus 4.7 and as an `input_file` to gpt-5.5. Both models
correctly extracted the word "HELLO" from the PDF.
Follow-up (out of scope):
- Pydantic schema entry on ChatMessage.content for `input_document`
(today it rides through because ChatCompletionRequest uses
extra=allow). Will tighten when the frontend attach button lands.
- Frontend file-picker UX for non-image attachments on the external
provider path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: gate empty-content msg + skip empty data-URI payload
Gemini High + Codex P2 on PR #5689:
1. Anthropic translation appended an empty `anthropic_parts` array
when every part was dropped (e.g. user sent only an unparseable
input_document). Anthropic 400s on "messages.N.content: at least
one block is required". Skip the whole-message append when no
parts survived. The OpenAI Responses path already had the
equivalent guard, so this brings the two providers into parity.
2. `data:application/pdf;base64,` with no payload (or whitespace-only)
parses to an empty `source.data` string. Anthropic rejects that
with 400 as well. Skip the document block before constructing it.
Plus 2 new test cases pinning both behaviors:
- `test_anthropic_empty_only_document_drops_whole_message`: confirms
a turn whose only content is an unparseable input_document does
NOT make it onto the outbound `messages` array.
- `test_anthropic_empty_data_uri_payload_is_dropped`: confirms an
empty-payload data-URI is filtered out at translation time.
(Note re: gemini's other High note about adding `input_document` to
the Pydantic ContentPart union -- ChatCompletionRequest is configured
with `extra=allow` so the part rides through today. Tightening the
union belongs with the frontend attach-button PR that surfaces the
field; called out as follow-up in the PR description.)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: register input_document in ContentPart + builder
Reviewer caught that the translation code on the external_provider
side was unreachable from a real ChatCompletionRequest:
- ContentPart is a discriminated Union of (text, image_url) only, so
any `{"type": "input_document", ...}` part was rejected by Pydantic
at request parsing with a discriminator error before the helper
could see it.
- _build_external_messages in routes/inference.py only walked text
and image_url parts, so even with a permissive schema the document
parts would have been silently dropped instead of forwarded to
the per-provider translator.
Fixes:
- Add InputDocumentContentPart with optional file_data / file_url /
filename / media_type and Tag("input_document") on the Union.
- Extend _build_external_messages to pass input_document through as
a plain dict for vision-capable providers (so external_provider's
existing Anthropic `document` and OpenAI Responses `input_file`
mappers actually run) and strip them on non-vision providers.
Tests added: schema accepts input_document, builder passes it to
vision providers, builder strips it on non-vision providers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: validate file_data before preferring over file_url
Codex P2 caught that the OpenAI input_document translator treats any
truthy file_data as valid and never falls back to file_url. That
means a malformed `data:application/pdf;base64,` (empty payload) or
a whitespace-only data URI gets forwarded as `file_data=""` and
400s the whole turn, AND silently discards a perfectly recoverable
file_url on the same part.
Mirror the Anthropic-side guard onto the OpenAI Responses path:
treat any "data:" URI with no actual base64 payload as missing and
fall through to file_url. Standalone-empty data URIs (no fallback)
are dropped entirely instead of being sent to the wire.
Tests added: empty data URI + valid file_url -> file_url wins,
whitespace-only data URI + valid file_url -> file_url wins,
empty data URI without fallback -> part is dropped.
* Address review: Anthropic side also falls back to file_url on empty data URI
Codex P2 follow-up to my earlier fix: I added the empty-data-URI ->
file_url fallback to the OpenAI Responses translator but missed
the Anthropic translator, which still `continue`d on empty payloads
and discarded an otherwise valid file_url on the same part. Result:
when the frontend supplied both file_data (placeholder / broken)
AND a working file_url, Anthropic silently lost the attachment;
when the message contained only that part, the whole message could
be dropped before reaching the wire.
Mirrored the OpenAI guard: any "data:" URI with no actual base64
payload (`data:application/pdf;base64,` or whitespace-only) is
treated as missing, and the file_url branch takes over. The
all-parts-dropped guard further down already handles the
no-fallback case.
Tests added: empty data URI + valid file_url -> URL source on the
wire with the filename preserved; whitespace-only data URI + valid
file_url -> URL source on the wire.
* Address review: gate input_document passthrough to anthropic + openai
Codex P1: only `_stream_anthropic` and `_stream_openai_responses`
have explicit translation logic for input_document parts (the former
maps to {type:"document", source:...}, the latter to
{type:"input_file", file_data|file_url}). Every other provider
(gemini / mistral / kimi / openrouter / deepseek / qwen / custom)
goes through the generic /chat/completions passthrough that forwards
`messages` verbatim, so any input_document part on a non-vision
route on those providers would 400 with an unknown content_part
type.
Added `_INPUT_DOCUMENT_PROVIDERS = frozenset({"anthropic", "openai"})`
constant and gated the pass-through branch on `provider_type in
_INPUT_DOCUMENT_PROVIDERS`. Every other provider strips the part
(text content survives). Threaded provider_type through from
_proxy_to_external_provider's call site.
Tests updated: vision + provider in {anthropic, openai} still
forwards; six unmapped providers (gemini/mistral/kimi/openrouter/
deepseek/qwen) strip the part; missing provider_type strips
defensively. The existing non-vision drop test still passes.
* Fix stale web_fetch tool-version assertion after merging main
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: wire OpenAI Responses server-side context compaction
The OpenAI Responses API accepts a `context_management` field that
enables server-side compaction. When the rendered prompt crosses the
configured threshold, the API runs a server-side compaction step and
the request continues against the compacted prefix. No beta header
and no dated version pin are required, per the docs.
Changes:
- Add `compaction_threshold: Optional[int]` (ge=1_000, le=2_000_000)
to ChatCompletionRequest. Thread through `routes/inference.py` ->
`stream_chat_completion` -> `_stream_openai_responses`.
- In `_stream_openai_responses`, when threshold is set AND the base
URL points at cloud OpenAI (api.openai.com), attach
`context_management: [{type:"compaction", compact_threshold:N}]`
to the outbound body. Non-cloud bases (ollama, llama.cpp, "custom"
presets) silently drop the field so we don't 400 those servers.
- Add `test_openai_compaction.py` with 4 cases: cloud OpenAI sets
the field verbatim, low-threshold probe passes through (we don't
clamp on the OpenAI side because the API accepts whatever),
non-cloud base drops the field, omitted threshold leaves body
untouched.
Live verified against the real OpenAI API on gpt-5.5:
`context_management:[{type:"compaction", compact_threshold:200000}]`
returns 200 with no error.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: accept Azure OpenAI base URLs + raise compaction floor
Two reviewer follow-ups on the OpenAI compaction PR:
1. The `is_openai_cloud = "api.openai.com" in self.base_url` check
excluded Azure OpenAI Foundry, even though Azure exposes the
same /v1/responses extensions (context_management,
prompt_cache_retention, container shell). Users on Azure saw
their compaction toggle silently no-op. Broadened the check to
also match `*.openai.azure.com` and made it case-insensitive so
URLs copy-pasted from the Azure portal still resolve. Non-cloud
OpenAI-compatible servers (ollama / llama.cpp / vLLM / "custom"
preset) still fall outside the gate.
2. The schema floor on compaction_threshold was ge=1_000, which is
well below the upstream Responses API's effective minimum
(vercel/ai#12486, langchain-ai/langchain#35464 report
`compact_threshold is not enabled` 400s on Azure at 100k; cloud
uses 200k as the canonical example). Raised the floor to 10k
so obvious typos surface as a clean 422 from FastAPI rather than
an opaque upstream 400 the user has to debug from the SSE
stream.
Tests added: Azure base URL carries both context_management and
prompt_cache_retention; mixed-case Azure URLs match; schema rejects
9_999 and accepts 10_000.
* Address review: drop schema-level compaction floor (cross-provider regression)
Codex P2 follow-up on the previous floor bump: ge=10_000 was
enforced globally at the ChatCompletionRequest layer, but the field
is documented as a no-op on every non-cloud OpenAI base and every
non-OpenAI provider. With the global floor, an Anthropic / ollama
/ llama.cpp / custom request that happens to carry compaction_threshold
below 10k was rejected with 422 at request validation time instead
of being silently ignored as the description promised.
Reverted the schema floor to ge=1 (any positive int) and rewrote
the description to call out per-provider routing: OpenAI cloud's
effective floor is around 200k and surfaces upstream 400s below
that; _stream_anthropic clamps sub-50k values up. Per-provider
helpers stay the single source of truth on the floor.
Test updated to pin: zero is still rejected, but every positive
value (1, 5_000, 9_999, 10_000, 200_000) passes schema validation.
* Address CodeQL: hostname-anchored OpenAI cloud detection
CodeQL py/incomplete-url-substring-sanitization fired on
`".openai.azure.com" in _base`. An attacker who controls the
configured base_url could slip cloud-only request body fields
(prompt_cache_retention, context_management compaction, container
shell) to an arbitrary server with:
https://evil.com/api.openai.com/v1https://api.openai.com.attacker.com/v1https://attacker.com/.openai.azure.com/v1https://my-resource.openai.azure.com.attacker.com/openai/v1
Replaced the substring check with a `_is_openai_family_cloud`
helper that runs urllib.parse.urlparse on the URL and matches the
lowercased hostname exactly (`api.openai.com`) or via `endswith`
on the leading-dot suffix (`.openai.azure.com`). Both halves are
host-anchored so path / fake-subdomain bypasses fail.
Test added: every attacker-controlled bypass shape above must NOT
carry context_management OR prompt_cache_retention on the wire.
Existing Azure and openai.com tests still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: scope compaction_threshold description to OpenAI on this branch
Codex P2: the field description on this PR mentioned Anthropic
compaction behavior, but the Anthropic wiring lives on PR 5686
(separate branch). On feat/openai-compaction alone, _stream_anthropic
has no compaction_threshold parameter, so the field is silently
ignored for Anthropic requests and the doc claim was misleading.
Trimmed the description to OpenAI cloud + Azure Foundry only on
this branch. PR 5686 already re-adds the Anthropic clause via its
own change, so the rebase / merge order on main will land the
combined description naturally once both PRs ship.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: wire Anthropic server-side context compaction
Anthropic ships server-side context compaction as a beta
(`compact-2026-01-12`). When the rendered prompt crosses the
configured input-token threshold, Anthropic runs an extra LLM pass
that summarises older turns and the request continues against the
compacted prefix. The response carries the original top-level fields
plus a new `context_management` block (with `applied_edits`) and
`usage.iterations[]` accounting per pass.
Per the docs the feature is currently supported on Opus 4.6, Opus 4.7,
Sonnet 4.6, and Mythos preview. The minimum threshold is 50k tokens;
under-50k requests 400.
Changes:
- Add prefix gate + helper `_anthropic_supports_compaction` plus
constants `_ANTHROPIC_COMPACTION_PREFIXES`, `_ANTHROPIC_COMPACTION_BETA`,
`_ANTHROPIC_COMPACTION_TYPE`, `_ANTHROPIC_COMPACTION_MIN`.
- Add `compaction_threshold: Optional[int]` to ChatCompletionRequest
(50k ge bound, 2M le bound). Thread through `routes/inference.py`
-> `stream_chat_completion` -> `_stream_anthropic`.
- In `_stream_anthropic`, when threshold is set AND the model
accepts compaction, attach `context_management.edits[{type:
"compact_20260112", trigger:{type:"input_tokens", value:N}}]` to
the outbound body. Sub-50k values are clamped up to 50k to keep
the request well-formed.
- Refactor the anthropic-beta header builder to merge any combination
of `code-execution-2025-08-25` + `compact-2026-01-12` flags into
one header value. Unrelated betas added at the registry level still
pass through.
- Add `test_anthropic_compaction.py` with 16 cases: gate matrix
(every doc-listed model), correct body shape, threshold clamping,
beta header merge with code execution, silent no-op on unsupported
models, omitted-threshold pass-through.
Live verified end-to-end against the real Anthropic API:
`compact_20260112` accepted on Opus 4.7, response carries
`context_management.applied_edits` + `usage.iterations[]` as
documented. (The first WebFetch-summarised version of these docs
suggested `compact_20260120`; the actual API only accepts
`compact_20260112`, matching the beta-header date. Worth pinning
behind a test so a future doc update can't drift back.)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: drop ge=50_000 clamp + parse usage.iterations[]
Two reviewer follow-ups on the compaction PR:
1. Pydantic ge=50_000 on compaction_threshold was dead code.
FastAPI rejected sub-50k threshold values with a 422 before the
`max(int(...), _ANTHROPIC_COMPACTION_MIN)` clamp in
_stream_anthropic could ever fire. Relaxed the floor to ge=1 so
the in-helper clamp actually does its job; the schema comment
now explains why this is intentional. Added a regression test
that posts a value of 1 and 49_999 through the real request
schema.
2. Anthropic publishes per-iteration token counts in
`usage.iterations[]` whenever a fresh compaction has run, and
the top-level input_tokens / output_tokens cover only the
`message` iteration -- billing must add the compaction
iterations on top. Aggregate compaction iteration tokens into
`last_usage["compaction_input_tokens" / "compaction_output_tokens"]`
so the cost surface (PR 5690) can read them without re-walking
the array, and surface both figures in the closing stream
summary log. Added two tests: one that pins the aggregation on a
compacted turn and one that pins `None` when no fresh
iterations land (so re-applied compaction blocks don't double-bill).
Sourcing: https://platform.claude.com/docs/en/build-with-claude/compaction
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: round-trip Anthropic compaction blocks across turns
Codex P1: once context_management is enabled and Anthropic runs
server-side compaction mid-stream, the response carries a
`{type:"compaction", content:"<summary>"}` content block on the
assistant message. The translator only handled text_delta and
input_json_delta on content_block_delta, so the compaction block
was silently dropped. Worse, the request schema's ContentPart
discriminated Union didn't accept `type:"compaction"`, and
_build_external_messages didn't pass it through, so even a
hand-crafted assistant message carrying the block would 422 at
parse time. Net result: Anthropic re-compacted from scratch on
every subsequent turn, wasting input tokens and reasoning budget.
End-to-end backend wiring of the round-trip:
1. SSE translator. _stream_anthropic now tracks a `current_compaction`
state slot. content_block_start with type=="compaction" seeds it
(Anthropic may include the summary on the start event AND/OR
stream it via text_delta events on the same block index --
handle both). text_delta inside a compaction block routes into
the compaction buffer instead of the user-visible content
stream, since the summary is opaque internal state, not
assistant prose. content_block_stop emits a `compaction_block`
tool_event carrying the full summary so the chat-adapter can
persist it. compaction_blocks_seen is surfaced in the closing
summary log.
2. Pydantic schema. Added CompactionContentPart with Tag("compaction")
on the ContentPart Union so requests carrying the block parse
cleanly. Required `content` field with a docstring pointing at
the Anthropic docs.
3. Message builder. _build_external_messages forwards compaction
parts on both vision and non-vision paths; the per-provider
stream helper decides whether to forward to the wire (Anthropic
does; other providers ignore the part). When a non-vision route
ends up with a single text part, collapse back to a string
so providers that don't accept content arrays still get the
expected shape.
4. _stream_anthropic outbound translator. {type:"compaction"} parts
on an assistant message land on the wire verbatim. Empty/missing
`content` is skipped so a malformed stored block can't 400
Anthropic.
Tests added (5): stream emits compaction_block tool event with the
summary intact; user-visible content stream does NOT carry the
summary text; outbound body forwards compaction parts verbatim on
the next turn; Pydantic schema accepts the part; builder passes
it through on both vision and non-vision provider routes.
Frontend follow-up: the chat-adapter needs to persist the
compaction_block tool_event onto the stored assistant message so
turn N+1 includes it in payload.messages. Pinned in the PR
description.
Sourcing: https://platform.claude.com/docs/en/build-with-claude/compaction
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: gate compaction-part passthrough to Anthropic only
Codex P1: my previous round-trip change preserved {type:"compaction"}
parts on every provider route in _build_external_messages. That
meant a chat history with prior compaction state silently leaked
the Anthropic-specific block to OpenAI/DeepSeek/Mistral/Gemini/
Kimi/OpenRouter on a provider switch, where generic
/chat/completions passthrough hands the unknown content type to
the upstream API and 400s the whole turn.
Added a `provider_type` kwarg to _build_external_messages and
gated the compaction forwarder on `provider_type == "anthropic"`.
Every other value (including the legacy None for callers that
don't pass it yet) strips the part. The Anthropic stream helper
still maps it to a native `compaction` block on the wire.
Threaded provider_type through from _proxy_to_external_provider's
call site.
Tests updated: vision + provider="anthropic" still forwards; six
non-anthropic providers strip the part; missing provider_type
strips defensively; non-vision + anthropic still forwards; non-vision
+ non-anthropic collapses back to a text string.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: wire Anthropic web_fetch server-side tool
Studio's Anthropic passthrough only forwarded web_search and
code_execution when enabled_tools was set. Asking Claude through Studio
to fetch a URL produced no fetch (the tool was not in the outbound
tools array), so users had to fall back to web_search even when they
already had the exact URL they wanted.
This change opts in web_fetch_20250910 when enabled_tools contains
"web_fetch". The new tool entry is appended alongside any existing
web_search / code_execution entries:
{"type": "web_fetch_20250910", "name": "web_fetch", "max_uses": 5}
No anthropic-beta header is required (web_fetch is GA); the existing
code-execution-2025-08-25 flag continues to merge cleanly when both
tools are enabled in the same turn.
SSE translation mirrors the web_search path. A `server_tool_use` block
with name="web_fetch" emits a `tool_start` _toolEvent carrying the
URL the model asked to fetch; the matching `web_fetch_tool_result`
block emits a `tool_end` _toolEvent whose result string follows the
Title / URL / Snippet shape parseSourcesFromResult on the frontend
already expects, so the source pill renders identically. Error blocks
(`web_fetch_tool_error`) are surfaced as "Error: <error_code>" matching
the code_execution error path.
The final "Anthropic stream complete" log line picks up web_fetch_
requested / web_fetch_invocations / web_fetch_urls so support reports
of "the model did not fetch anything" can be triaged from the log.
Verified end to end against claude-haiku-4-5 with
`enabled_tools=["web_fetch"]`: the model emitted tool_start with
url=https://example.com and tool_end with the page Title + URL +
Snippet, plus the assistant message correctly read back "Example
Domain" as the title.
Tests:
- 5 new unit tests in test_anthropic_web_fetch.py covering tool
registration, the combined web_search + web_fetch + code_execution
request body, the pill-off case, and SSE translation for both
success and error paths.
- All 242 existing Anthropic + OpenAI provider tests still pass.
The enabled_tools field description in models/inference.py is updated
so OpenAPI consumers see the new option.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* web_fetch: title fallback to URL, log parse failures, drop dead checks
Three review nits on the previous commit:
1. `_format_web_fetch_result` left `title` empty when Anthropic omitted
`document.title`. The frontend `parseSourcesFromResult` only emits
a source pill when both `Title:` and `URL:` lines are present, so
fetches against pages without an HTML title tag silently lost
their citation in the UI. Fall back to `title = title or url`,
matching the web_search formatter.
2. The broad `except Exception` around `json.loads(buffer)` for the
web_fetch input swallowed the failure with no trace. Log at debug
so a malformed partial_json buffer can be triaged from the server
log without changing behavior.
3. `inner` was already sanitised to a dict at the matching
content_block_start and `_format_web_fetch_result` always returns
a non-empty string (defaulting to "(fetch complete)"), so the
`isinstance(inner, dict) else {}` guard and the
`result_text or "(fetch complete)"` fallback at the emit site
were dead code. Removed.
Added a test exercising the titleless path so the fallback stays
covered.
* chat-adapter: emit source pills for web_fetch tool calls
`parseSourcesFromResult` was only wired up for tool calls where
`toolName === "web_search"`, so the Title / URL / Snippet block the
backend formatter emits for `web_fetch_tool_result` never reached the
source-pill renderer. Users saw the raw tool result in the tool card
but the dedicated source-pill row at the message tail stayed empty.
Both web_search and web_fetch ship the same text shape today, so the
fix is to broaden the gate.
* Address review: wire web_fetch from Search pill + fix pause_turn truncation
Two reviewer follow-ups on the Anthropic web_fetch PR:
1. The backend tool wiring landed but the frontend chat-adapter
never put `web_fetch` in `enabled_tools`, so toggling the Search
pill only ever attached `web_search` -- web_fetch was unreachable
from the UI. Added providerSupportsBuiltinWebFetch() (Anthropic
today) and paired the entry with the existing Search pill, since
the canonical workflow is "search returns URLs, fetch reads
them" and there is no separate UI toggle yet.
2. `pause_turn` from Anthropic's stop_reason vocabulary fell through
the finish_reason map's "stop" default, which the OpenAI-format
client renders as end-of-message and truncates the answer. Per
the docs pause_turn means "Claude paused a long server-tool
turn (web_search / web_fetch) and will resume". Mapped to None
and skipped the chunk emission so the SSE stream still ends with
[DONE] on message_stop but no terminal finish_reason lands on
the client. While there: added explicit mappings for `tool_use`
(-> tool_calls) and `refusal` (-> content_filter) which were
also falling through to "stop".
Tests added: pause_turn emits no finish_reason, end_turn still
emits "stop", refusal maps to "content_filter".
Sourcing: https://platform.claude.com/docs/en/api/messages#response-stop-reason
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: support Anthropic 1h cache TTL via prompt_cache_ttl field
Anthropic exposes two ephemeral cache pools per request: the default
5-minute pool, and a 1-hour pool selected by attaching `ttl:"1h"` to
the `cache_control` marker. 1h writes are billed at 2x base input vs
1.25x for 5m, but reads stay at 0.1x for both, so a single extra read
landing more than 5 minutes after the write pays off the premium.
Studio hardcoded the 5m pool via `cache_control: {type:"ephemeral"}`
on both breakpoints. For chats with multi-minute idle gaps (people
juggling tabs, long-running tool calls between turns), the cache
expires before the next turn and every read becomes a cache_creation,
not a cache_read -- exactly the case where the 1h pool wins.
Changes:
- Add `prompt_cache_ttl: Optional[Literal["5m", "1h"]]` to
ChatCompletionRequest. Default (None) preserves today's 5m behavior.
- Thread through `routes/inference.py` ->
`stream_chat_completion` -> `_stream_anthropic`.
- Build a shared `cache_marker` dict in `_stream_anthropic`; attach
`ttl` only when the request asks for one of the two valid values.
Unknown TTL strings are silently dropped to avoid sending malformed
markers (the upstream API would 400).
- Apply the same marker to both existing breakpoints (system block at
line 1175 and the latest-message tail at line 1198 / 1213) so the
pool selection is consistent across the whole prefix.
- Add `test_anthropic_cache_ttl.py` with 11 parametrized cases
pinning the outbound body shape: omitted -> default marker;
explicit `5m`/`1h` -> ttl field set; unknown values dropped;
caching off -> no markers at all.
Verified upstream that `cache_control: {type:"ephemeral", ttl:"1h"}`
is accepted by the Anthropic API today; no beta header required.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Relax prompt_cache_ttl to Optional[str] (Codex P1)
Declaring `prompt_cache_ttl` as `Optional[Literal["5m", "1h"]]` made
FastAPI/Pydantic 422 the request before _stream_anthropic could even
see the field. The whole point of the downstream drop-unknown-values
behaviour was to keep a stale frontend from crashing the request;
the strict Literal at the request layer defeated that.
Loosen the schema to Optional[str]; the existing in-helper guard
already restricts forwarded values to {"5m", "1h"} (everything else
is silently dropped). Test suite stays unchanged -- the bogus-value
cases in test_anthropic_cache_ttl.py already pass arbitrary strings
through and assert they are dropped before the wire.
* Address review: confirm extended-cache-ttl beta header is GA
Reviewer asked whether the 1h cache TTL still requires the
`extended-cache-ttl-2025-04-11` anthropic-beta header. Investigated:
- Live-tested api.anthropic.com on claude-opus-4-7 (2026-05-22)
with cache_control={type:"ephemeral", ttl:"1h"} and NO beta
header. Got status 200 and ephemeral_1h_input_tokens populated
on the create turn, plus cache_read_input_tokens populated on
the reuse turn.
- Cross-checked the current prompt-caching docs: no mention of
any beta header on the 1h TTL path.
Conclusion: the gate has been promoted to GA. The code already
does not send the beta header (the cache_marker dict only carries
`type`/`ttl`), so no wire change is needed. Pinned the contract
with two regression tests that assert the header is NOT on the
outbound request, and added a docstring note explaining the
investigation outcome so a future reader does not re-add it.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix: Claude Code Anthropic API tool compatibility
* fix: merge Anthropic server tool selections
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix anthropic /v1/messages server-tool alias misrout
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: harden Anthropic /v1/messages tool validation
* fix: dispatch Anthropic server tools by only
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: reject Anthropic client tools missing 'name' at boundary
AnthropicTool.name was relaxed to Optional[str] to accommodate server-tool
declarations. A client tool with input_schema but no name now parses but
is silently dropped by anthropic_tools_to_openai, leaving tool calling
disabled. Surface as 400 instead.
* fix: reject Anthropic client tools with empty 'name'
isinstance(name, str) accepts an empty string, but anthropic_tools_to_openai
drops entries via 'if not name', producing the same silent-disable
fallthrough the boundary check is meant to prevent. Tighten to also reject
empty name.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
* studio: add --spec-draft-n-max toggle for MTP speculative decoding
Surface llama-server's --spec-draft-n-max as a first-class
LoadRequest field so users can tune the MTP draft tree size from
the chat settings panel. Default behaviour is unchanged: when the
caller omits spec_draft_n_max, the existing platform defaults still
apply (6 on GPU, 3 on CPU/Mac).
Why this matters: on context-constrained loads the draft KV cache
competes with the target model's KV cache for VRAM. Lowering
spec_draft_n_max reduces that pressure, lets a larger user context
fit, and recovers throughput; raising it pays off when draft
acceptance is high enough to amortise the extra cache.
Backend
- LoadRequest gains an optional spec_draft_n_max: int (1..16).
- LlamaCppBackend.load_model accepts and persists the override on
self._spec_draft_n_max, used in place of the hardcoded 6/3 in the
MTP emit branch.
- LoadResponse and InferenceStatusResponse echo the active value
(None when the platform default is in effect) so the UI can
hydrate the input on refresh.
- _already_in_target_state and _request_matches_loaded_settings
compare spec_draft_n_max alongside speculative_type so a value
change triggers a reload rather than no-op'ing.
- strip_shadowing_flags now strips inherited --spec-* extras when
either speculative_type or spec_draft_n_max is in fields_set, so
an inherited --spec-draft-n-max cannot last-wins-override a fresh
request's first-class field.
Frontend
- LoadModelRequest, LoadModelResponse, InferenceStatusResponse
TypeScript shapes get spec_draft_n_max.
- chat-runtime-store gains specDraftNMax / loadedSpecDraftNMax and
a setter, hydrated from /v1/status and /v1/load.
- chat-settings-sheet renders a "Draft Tokens" numeric input
directly under the Speculative Decoding switch when that switch
is on. Toggling the switch off clears the override; the Reset
button restores the loaded value.
Tests
- Four new regression tests cover _already_in_target_state with
matching / mismatching / non-MTP / unset spec_draft_n_max.
- Existing test_llama_server_args.py and test_llama_cpp_mtp_detection.py
green: 141 passed locally.
* studio: add --spec-draft-p-min and --spec-draft-p-split to spec strip set
llama.cpp server documents --spec-draft-p-min (default 0.75, min draft
acceptance probability) and --spec-draft-p-split (default 0.10). Both
are first-class spec-decoding knobs that should travel with the rest
of the --spec-* family when an Apply re-sets speculative_type, so an
inherited override doesn't leak across a fresh load.
* studio/tests: skip MTP capability-probe tests on Windows
The four probe_server_capabilities tests use a bash stub written to
tmp_path/llama-server, which Windows' subprocess can't execute
directly (no shebang resolution, .bat / .cmd would be needed). Mark
them skipif sys.platform == 'win32' so the rest of the MTP plumbing
suite stays green on Windows CI. Unix coverage is unchanged.
* studio: lower MTP GPU default --spec-draft-n-max from 6 to 2
Bench on B200 / Qwen3.6-27B-MTP-GGUF UD-Q4_K_XL across five prompt
types (essay, code, story, math, science) with greedy temp=0:
prompt OFF n=1 n=2 n=3 n=6
essay 79.1 93.4 93.8 84.7 64.6
code 79.1 104.4 116.6 113.5 103.0
story 79.1 99.2 105.7 101.8 88.9
math 79.1 100.8 110.8 111.8 98.2
science 79.1 100.1 110.8 110.8 102.9
The previous hardcoded GPU default of 6 was 17% SLOWER than spec-off
on the essay prompt (64.6 vs 79.1 t/s) and 11-50% slower than n=2 on
the rest. n=2 wins on 4/5 prompts with a 1.18x-1.47x speedup vs OFF;
n=3 wins on the math prompt by a hair. n=6 collapses once acceptance
rate drops past n=3 -- wasted draft decode dominates the per-step
budget.
Matches the dataset README ("n_max=2 is the sweet spot for 36 of 42
quants"). Keeps CPU/Mac default at 3, which empirically tracks the
narrower ngram+MTP chained budget on those platforms.
Users who want the old behaviour can pass spec_draft_n_max in
LoadRequest (the toggle this PR also adds) or --spec-draft-n-max via
llama_extra_args.
* studio: skip MTP auto-promote on sub-2B models, backfill chat usage
Two MTP-visibility fixes uncovered while bisecting llama.cpp post-#22673
on Qwen3.6-27B-MTP-GGUF UD-Q4_K_XL on B200.
Size gate. Direct llama-server bench (no Studio measurement loop) at
n_predict=192 across 9 prompts shows MTP regresses vs spec-off on
sub-2B dense models because draft cost exceeds savings:
Qwen3.5-0.8B Q4_K_XL GPU: 452.0 OFF -> 283.4 t/s n=2 (0.63x)
CPU: 84.5 OFF -> 64.9 t/s n=3 (0.77x)
Qwen3.5-4B Q4_K_XL GPU: 241.0 OFF -> 258.2 t/s n=2 (1.07x)
Qwen3.5-9B Q4_K_XL GPU: 201.6 OFF -> 228.9 t/s n=2 (1.14x)
Qwen3.5-27B Q4_K_XL GPU: 78.8 OFF -> 113.6 t/s n=2 (1.44x)
Qwen3.6-27B Q4_K_XL GPU: 78.8 OFF -> 113.6 t/s n=2 (1.44x)
Qwen3.6-35B-A3B Q4 GPU: 192.3 OFF -> 223.2 t/s n=2 (1.16x)
The 2B inflection is sharp. Skip auto-promote to draft-mtp when the
identifier reports <2.0B params; users can still force via --spec-type
or the Speculative Decoding toggle. Mirror the gate in the
reload-skip check so a sub-2B reload-with-default does not bounce a
spec-off backend.
Chat-completions usage. llama-server's final SSE chunk emits both an
OpenAI-style usage block and a custom timings block. timings.predicted_n
is always populated, but usage.completion_tokens is zero on some
server builds. The Studio chat UI computes generation t/s from
meta.usage.completion_tokens / totalStreamTime, so a zero
completion_tokens makes the UI fall back to wall-clock time
(including SSE / proxy / template overhead) which dilutes MTP gains and
makes ON look the same as OFF.
Add _backfill_usage_from_timings: if usage.completion_tokens is missing
or zero AND timings has predicted_n/prompt_n, synthesize a complete
usage dict. Apply at the streaming metadata yield in
generate_chat_completion and at the three accumulator/yield sites in
generate_chat_completion_with_tools so per-iteration counts are not
silently lost across tool calls.
Tests cover both the gate (sub-2B skips, 2B+ promotes) and the
backfill (zero usage filled, real usage preserved, empty timings
passthrough).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: probe + emit legacy ngram-mod flags for pre-rename llama-server
llama.cpp upstream renamed the ngram-mod tuning knobs:
--draft-max -> --spec-ngram-mod-n-max (and --spec-draft-n-max)
--draft-min -> --spec-ngram-mod-n-min (and --spec-draft-n-min)
--spec-ngram-size-n -> --spec-ngram-mod-n-match
The new names are real flags on post-rename builds and stub removal
entries on the same builds (with description "argument has been
removed"). Pre-rename builds only carry the legacy names as real
flags. Studio was emitting the new names unconditionally, so a user
running a pre-rename llama-server (e.g. an older prebuilt or a
hand-installed binary) would see "unknown argument" errors when the
ngram-mod path engages, or silent drop of the ngram knobs.
Extend `probe_server_capabilities` to parse the help text into
per-flag description blocks and tell real flags apart from removal
stubs by the "argument has been removed" marker. Add three new probe
fields: `ngram_mod_flavor` ("new" / "legacy" / None),
`supports_ngram_mod`, and `spec_draft_n_max_flag` (the actual n_max
flag the binary accepts). Cached by (path, mtime) the same way as
`mtp_token`.
Add `_build_ngram_mod_flags(caps, ...)` that picks the right flag
set, returning [] when neither is usable so callers can drop ngram
chaining entirely on minimal binaries.
Wire both call sites to use the probe-driven flag set:
- CPU/Mac MTP comma-chain (--spec-type ngram-mod,draft-mtp) emits
legacy or new knobs as appropriate. If neither set is available,
degrade to MTP-only (warn but still engage spec).
- Standalone --spec-type ngram-mod branch uses the same helper.
Tests cover post-rename detection, legacy detection, removal-stub
discrimination, minimal-binary case, and all three branches of
`_build_ngram_mod_flags` plus custom n_match/n_min/n_max values.
Verified against three real binaries (Studio bundled 726704a, my
build of 45b455e HEAD, and the MTP merge baseline 2555826) all
correctly reporting ngram_mod_flavor=new.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: sub-3B MTP falls back to ngram-mod, not off
Earlier sub-2B gate disabled speculative decoding entirely for tiny
dense MTP models because the MTP draft head's per-token cost exceeds
the acceptance savings at that scale. The "fully off" fallback was
conservative -- ngram-mod has near-zero idle cost on diverse content
and consistently outperforms both off and draft-mtp at sub-3B.
Clean-methodology bench (each of 9 distinct prompts run once after
two unrelated warmup prompts so the ngram-mod hash pool is
realistically populated but never holds the exact deterministic
output we're about to measure):
Q4_K_XL on B200:
0.8B OFF=451 draft-mtp n=2=263 (0.58x) ngram-only=498 (1.10x)
2B OFF=377 draft-mtp n=2=308 (0.82x) ngram-only=369 (1.00x)
4B OFF=240 draft-mtp n=2=260 (1.08x) -- 4B+ wins with MTP
Q4_K_XL on x86 48 cores:
0.8B OFF= 80 chained n=2= 69 (0.86x) ngram-only= 95 (1.19x)
2B OFF= 62 chained n=2= 51 (0.83x) ngram-only= 63 (1.01x)
4B OFF= 31 chained n=2= 41 (1.33x)
Change:
- Raise the MTP-skip threshold from 2.0B to 3.0B (2B falls below it).
- When skipping the MTP head, fall back to --spec-type ngram-mod via
the probe-driven _build_ngram_mod_flags helper. Works on both
post-rename and pre-rename llama-server builds.
- If the binary advertises neither ngram-mod flavor, fall back to
spec-off (older binaries that don't support ngram-mod at all).
- Mirror the same fallback in _already_in_target_state so a sub-3B
reload-with-default does not bounce a ngram-mod backend.
Tests updated: monkeypatch probe_server_capabilities so the gate
behavior is deterministic regardless of which llama-server happens
to be on the host. +1 new test for the "binary has no ngram-mod
support" branch; renamed prior 2B/0.8B tests to reflect new semantics.
This generalizes the size gate to be probe-driven instead of a hard
"disable spec" branch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: 5-mode Speculative Decoding dropdown (Auto / MTP / Ngram / MTP+Ngram / Off)
Replace the Chat Settings Speculative Decoding on/off Switch with a 5-option
Select. Auto preserves today's platform-aware resolver (MTP on MTP GGUFs,
ngram-mod fallback for sub-3B, --spec-default for non-MTP). The other 3 modes
force the user's choice on BOTH GPU and CPU: MTP emits draft-mtp only (no
ngram chain on CPU), Ngram emits ngram-mod only, MTP+Ngram emits the
ngram-mod,draft-mtp chain on both platforms. Off is the existing fully-off
state, kept so the Switch's "disable" capability isn't lost.
Backend
- New module-level _canonicalize_spec_mode(value) maps any accepted input
(canonical, legacy "default" / "draft-mtp" / "ngram-mod" / "ngram-simple",
or comma-chained "ngram-mod,draft-mtp") onto one of auto / mtp / ngram /
mtp+ngram / off / ngram-simple / None. Lets external callers and old
persisted UI state round-trip without breaking.
- LlamaCppBackend grows a _requested_spec_mode field + requested_spec_mode
property storing the canonical UI mode the user requested. Status
responses round-trip this instead of the resolved internal flag, so the
dropdown restores the picked value after reload / refresh (Auto on a 27B
MTP GGUF resolves to draft-mtp internally but the dropdown stays on
"Auto").
- The resolver block in load_model is extracted into a unit-testable
_build_speculative_flags method. Forced MTP / MTP+Ngram on a sub-3B or
non-MTP GGUF logs a warning and engages anyway (user override > the
Auto-path sub-3B fallback).
- _already_in_target_state and routes/inference._request_matches_loaded_settings
now compare canonical-requested mode, dropping the old auto-promotion
mirror. spec_draft_n_max still gates on the resolved spec so Auto + a
changed n_max still bounces a reload.
Frontend
- chat-settings-sheet.tsx: Switch swapped for Select modeled on the KV
Cache Dtype Select. Items: Auto / MTP / Ngram / MTP+Ngram / Off. Draft
Tokens input only visible when speculativeType is "mtp" or "mtp+ngram".
- chat-runtime-store.ts: initial value flips from "default" to "auto".
- use-chat-model-runtime.ts normalizeSpeculativeType mirrors the backend
canonicaliser so persisted "default" / "draft-mtp" / "ngram-mod" / chain
values hydrate to the right dropdown option.
- types/api.ts: docs the canonical wire vocabulary.
Tests
- 53 new assertions in test_llama_cpp_mtp_detection.py: full
_canonicalize_spec_mode table, a 23-row resolver matrix across
(requested mode) x (GPU/CPU) x (model size class), plus n_max override,
user-extra-args precedence, requested-mode round-trip, and graceful
degrade on an outdated llama-server without an MTP token.
- 165 existing backend tests still green. 218 total in the MTP /
server-args / reload-inheritance suite.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: reset Speculative Decoding to Auto on model switch
When the user switches from model A to a different model B, clear the
runtime store's speculativeType + specDraftNMax (and their loaded*
shadows). The new load request then carries null, the backend
canonicalises that to "auto", and its platform-aware resolver runs
fresh for the new model.
Without this, a non-MTP model loaded with "Off" carried the Off choice
into a subsequent MTP load, suppressing MTP auto-promotion (and the
sub-3B ngram-mod fallback) until the user manually opened settings and
flipped the dropdown back to Auto. The clean-sweep deep probe caught
it as anomaly A-1.
The reset only fires when currentCheckpoint != modelId, so a
same-model reapply or forceReload still honours the user's current
spec choice. End-to-end probe on Qwen3.5-4B-GGUF (non-MTP, Off) ->
Qwen3.5-0.8B-MTP confirms: dropdown shows Auto, /api/inference/status
returns speculative_type=auto, studio.log shows the Auto sub-3B
fallback emitted --spec-type ngram-mod.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: warn when llama.cpp prebuilt is at least 3 days behind
Layered on #5528. Generalises the MTP-specific staleness warning to
every llama.cpp prebuilt update, not just the ones that add MTP. If
the installed prebuilt is at least 3 days old AND its tag differs
from the latest published tag on the helper release repo (default
unslothai/llama.cpp), Studio nudges the user to run
"unsloth studio update".
How it works
Reads the install marker UNSLOTH_PREBUILT_INFO.json that
install_llama_prebuilt.py already writes to install_dir. The marker
carries the installed tag, the helper repo, and an installed_at_utc
timestamp. Studio compares those against the latest published tag
from the GitHub releases API for the helper repo.
GitHub fetch is cached at two levels:
- Process-level memo for /status hot path.
- Disk-level cache (24h TTL) at ~/.unsloth/studio/cache/llama_cpp_freshness/
so cold-start Studio launches do not always hit the API.
On a transient fetch failure (offline, rate-limited) we keep the
last-good disk value alive rather than poisoning the cache with None.
The check fails open: if anything is missing (marker, timestamp,
GitHub response), stale stays False so users never see a misleading
banner.
Surfaced in two places
1. Startup banner (logs + stderr) in main.py:lifespan(), alongside the
MTP capability probe added in #5528. Single line, e.g.:
WARNING: llama.cpp prebuilt is 5 days behind: installed b9190,
latest b9300. Run "unsloth studio update" to refresh.
2. /api/inference/status now returns:
llama_cpp_prebuilt_stale: bool
llama_cpp_installed_tag: str | None
llama_cpp_latest_tag: str | None
so the frontend can render a banner / popup with the actual tag
delta the user is missing.
3-day threshold
Mirrors the typical Unsloth llama.cpp release cadence. Anything
shorter would nag users who restart Studio at the wrong moment;
longer leaves real bugs sitting on the user's machine. Configurable
via the threshold_days kwarg if a future call site wants a different
window.
Tests
17 new cases in tests/test_llama_cpp_freshness.py cover marker
discovery in both cmake and root install layouts, missing / invalid
marker, GitHub fetch caching across process restarts (disk cache hit
after the in-memory cache is reset), the stale / not-stale decision
matrix (tag mismatch + age threshold), fail-open behaviour when
GitHub is unreachable, custom threshold, singular/plural day in the
warning string, and unparseable installed_at_utc. The broader
205-test inference regression suite still passes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Eyera
