* Studio: per-card web_search result + shell_call output fallback (OpenAI)
Two empty-output bugs in the OpenAI Responses tool-result rendering that
showed up clearly when a single prompt invoked 9 web_search + 4
code_execution + 1 image_generation in one turn. Reproduction shape in
the SQLite-stored chat history:
- 8 of 9 web_search tool-call records had result == "" (the cards
rendered as empty cards in the thread)
- 4 of 4 code_execution (shell_call) records were missing the result
key entirely (NoneType), so the cards that showed "Ran cat ..." style
commands displayed the command line but no output panel at all
- image_generation worked, as did the very last web_search of the run
Root causes in studio/backend/core/inference/external_provider.py:
1. web_search_call's tool_end emitted result: "" by design, with the
intent of overwriting only the LAST call at response.completed with
the full citation list (the source-pill extractor on the frontend
flatMaps across every web_search result, so a single non-empty
result is enough for the trailing source pills). Side effect: every
intermediate card renders empty in the thread. Fix: seed each call's
own tool_end result with "Searching: <query>" so the per-card text
is never empty, then keep the last-call overwrite path so the
source-pill extractor still works. Falls back to empty when the
model emits an action with no query, so the existing last-call path
stays unchanged for that edge.
2. shell_call's tool_start was emitted from
response.output_item.done for the call item, but tool_end lived in
the separate response.output_item.done handler for shell_call_output.
When OpenAI's Responses stream bundles the output array onto the
shell_call item's own done event (no separate shell_call_output
item), the previous handler emitted tool_start with no following
tool_end. The card spun on "running" indefinitely and stored as
NoneType in the thread DB. Fix: when the shell_call's done event
carries an embedded output list, emit tool_end immediately from
that. Track tool_end_emitted on the shell_calls map so a subsequent
shell_call_output event (some streams ship both) is skipped instead
of double-completing the card. A final flush at response.completed
emits tool_end for any orphan shell_call that received neither
bundled output nor a separate output event, so cards always finalise.
Tests (studio/backend/tests/test_openai_tool_result_fallbacks.py, 6
new):
- web_search: three calls, each card's result is its own Searching:
query (no empties)
- web_search: last call still gets the aggregated citation block when
url_citations arrive (pins the overwrite path)
- web_search: empty action.query falls back to result == "" (no junk
Searching: placeholder)
- shell_call: bundled output on done emits a single tool_end with that
output as the result text
- shell_call: bundled-then-separate output does not double-emit
tool_end (subsequent shell_call_output is skipped)
- shell_call: orphan call with neither bundled nor separate output is
flushed at response.completed so the card finalises
15/15 tests green when combined with the existing 9 in
test_openai_code_execution.py. Pre-commit + ruff format clean.
Scope: OpenAI Responses-API code path only. The Anthropic native
Messages-API path (_stream_anthropic) is untouched, as is the local
llama-server path. Local-model behaviour cannot regress because the
edited handlers only fire inside the OpenAI cloud branch.
* Studio: per-model external max_tokens cap + clamp on model switch
Two related external-provider issues that surfaced from the same
investigation as the per-card web_search / shell_call result bugs in
the previous commit:
A. Slider cap was a one-size-fits-all 32768 for every external model.
provider-capabilities.ts kept a single EXTERNAL_MAX_OUTPUT_TOKENS
constant (32k), well below what most providers actually accept. The
docstring even called out the right per-provider numbers (Anthropic
Opus 128k, GPT-5.x ~128k, Gemini 2.5 ~65k, DeepSeek 8k) but the
code picked the lowest as a conservative floor. Effect: long
generations from gpt-5.5 / claude-opus-4-7 silently truncated at
32k even though the API would have served up to 128k.
Fix: introduce getExternalMaxOutputTokens(providerType, modelId)
returning the documented per-model cap. Patterns are checked
longest-first so e.g. gpt-5.5-pro matches before gpt-5.5. Unknown
provider/model combinations fall back to the existing 32k floor so
no surprise increases for ids we don't know about.
Per-model caps from the official docs:
- OpenAI gpt-5.5 / gpt-5.5-pro: 128000
- OpenAI gpt-5.4 / gpt-5.4-pro: 65536
- OpenAI gpt-5.3: 16384
- Anthropic claude-opus-4-7: 128000
- Anthropic claude-opus-4-6 / sonnet-4-6 / opus-4-5 / sonnet-4-5 /
haiku-4-5: 64000
- Gemini 3.x family: 65535
- DeepSeek: 8192
- OpenRouter: strip provider/ prefix from the id and re-resolve
The slider in chat-settings-sheet.tsx and the send-time clamp in
chat-adapter.ts both call the new function so the slider's max=
matches what the wire layer will accept.
B. Slider value lied after switching from a local model to external.
When Studio auto-loads the helper Gemma-4-E2B-it on first chat,
chat-adapter sets params.maxTokens to Gemma's context_length
(262144 for Gemma 4). Switching the model picker to gpt-5.5 then
flips the slider's max prop to the external cap, but the stored
params.maxTokens is never reset. The numeric value next to the
slider would render 262144 against a track that ended at the
external cap. The send-time clamp brought the outbound max_tokens
back down to the cap, so the API call was safe, but the displayed
number had no relationship to what was actually being sent.
Fix: chat-runtime-store.setCheckpoint now clamps params.maxTokens
to getExternalMaxOutputTokens(...) on transitions into an external
model. Looks up the provider via useExternalProvidersStore so we
can derive providerType from the parsed external model id. No-op
when the stored maxTokens is already at or below the new cap, so
user-tuned values within range survive the switch.
Scope: pure frontend changes scoped to external-provider code paths.
Local model behaviour is untouched -- the ggufContextLength branch of
the slider's max= is unchanged, and setCheckpoint only mutates
maxTokens when isExternalModelId(modelId) is true. The send-time
clamp continues to be the safety net for any in-flight request that
crosses a model switch before the store-level clamp has applied.
Typecheck (tsc -b) clean; bun run build succeeds (2.13s).
Co-changes with the previous commit (7fe1adbf, per-card web_search +
shell_call output fallback) form a single PR: every empty-output and
silent-truncation issue surfaced from the same animal-popularity
prompt reproduction is now addressed in one branch.
* Studio: correct external max_tokens caps for Gemini and DeepSeek
Per-doc corrections to the per-model cap table added in 95da8d52:
- Gemini 3.x family: 65535 -> 65536, per
https://ai.google.dev/gemini-api/docs/models/gemini-3.1-pro-preview
(the published max_output_tokens is exactly 64K = 65536). The earlier
65535 was an off-by-one rough cap.
- DeepSeek (deepseek-chat / deepseek-reasoner aliases): 8192 -> 384000,
per https://api-docs.deepseek.com/quick_start/pricing. DeepSeek V4
Flash / Pro both list MAX OUTPUT = 384K; the chat / reasoner ids are
deprecated aliases for V4 Flash non-thinking / thinking modes. The
8192 value was carried over from V3 and silently truncated V4 traffic
at 2% of its actual ceiling.
Affects only the slider max and the send-time clamp for these provider
types. Other providers' caps unchanged. tsc -b clean.
* Studio: also flush orphan shell_calls on response.incomplete
Addresses gemini-code-assist[bot] high-priority inline review on PR
5785: the orphan-shell_call final flush added in 7fe1adbf landed only
in the response.completed branch. Truncated OpenAI Responses streams
emit response.incomplete instead (for example when the request hits
max_output_tokens), which left in-flight shell_call cards spinning
indefinitely in the UI.
Mirror the same flush block in the response.incomplete handler so the
truncated-stream path finalizes every pending tool card. The
tool_end_emitted guard keeps the path idempotent: if a shell_call
already completed via bundled output on its done event, the incomplete
flush is a no-op for it.
Two new tests in test_openai_tool_result_fallbacks.py:
- test_shell_call_flushed_on_response_incomplete_truncation pins the
bug repro: an in-flight shell_call followed by response.incomplete
must emit tool_end so the card finalizes.
- test_shell_call_incomplete_does_not_double_emit pins idempotency:
a shell_call that completed via bundled output and is then followed
by response.incomplete emits exactly one tool_end with the bundled
result text.
17/17 tests green (8 fallback tests + 9 existing code-execution). Pre-
commit + ruff format clean.
* Studio: trim verbose comments across PR 5785 edits
Compress the in-code commentary added across this branch to one or two
lines per block; the verbose prose was easier as a PR description than
as inline noise. No behavioural changes: 17/17 tests still green, tsc -b
still clean.
* Studio: longest-prefix pricing match + accept chat-style usage keys
Two P1 / High follow-ups from PR 5690 review feedback:
1. Pricing prefix lookup returned the first key it iterated, so
dated snapshots like ``gpt-5.4-mini-2026-04-23`` collided with
the shorter ``gpt-5.4`` entry and overbilled by 3x+. Sort the
table keys longest-first so the most specific entry wins.
2. ``calculate_cost`` only read ``input_tokens`` / ``output_tokens``,
but Studio's OpenAI-Chat-style usage envelope re-emits
``prompt_tokens`` / ``completion_tokens`` (the OpenAI Chat
Completions vocabulary). Callers handing in the chat-style
shape silently got a zeroed bill. Accept either pair so the
calculator works against both raw upstream usage and the
Studio-translated envelope.
Tests (4 new in test_pricing.py): dated mini/pro snapshots inherit
the right rate; chat-style usage keys price correctly; raw key wins
when both shapes are present.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: dedupe cache buckets when costing chat-style Anthropic usage
When the caller hands in Studio's chat-style envelope (``prompt_tokens``
emitted by ``_build_usage_chunk``) for Anthropic, that value already
folds ``cache_creation_input_tokens`` + ``cache_read_input_tokens`` into
the total. The previous follow-up accepted the chat-style key but then
re-added both cache buckets in ``billable_input_tokens`` and ``input_usd``,
double-counting cache tokens on every Anthropic chat-style call.
Detect which envelope landed (``input_tokens`` present = raw upstream;
absent + ``prompt_tokens`` present = Studio chat-style) and peel the
cache buckets off for Anthropic before the downstream math so both
envelopes produce identical costs.
OpenAI: ``input_tokens`` and Studio's ``prompt_tokens`` both already
include ``cache_read`` and exclude any notional ``cache_creation``, so
the OpenAI path stays a straight passthrough.
Tests (2 new): both envelopes match for Anthropic on a triple
(uncached + cache_creation + cache_read); OpenAI envelopes match on a
cached-tokens fixture.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: prefer raw output_tokens over chat-style completion_tokens
Codex flagged that the previous fallback chain
'usage.get("output_tokens") or usage.get("completion_tokens")'
treats an explicit 0 as missing -- a mixed-envelope payload where
'output_tokens' is 0 but 'completion_tokens' is non-zero (or
stale) bills the wrong amount. Mirror the has_input_tokens
precedence pattern: when the raw key is present we use it even at
0; otherwise fall back to completion_tokens.
* Studio: read OpenAI cached tokens from prompt_tokens_details too
Codex flagged that the chat-style OpenAI envelope Studio re-emits
via _build_usage_chunk surfaces cached prompt tokens under
prompt_tokens_details.cached_tokens, not input_tokens_details. The
OpenAI branch only checked input_tokens_details, so a cache-heavy
chat-style turn billed every cached token at the full input rate
instead of the 0.1x cache_read discount.
Walk both keys when discovering the cached count. New regression
test pins that the two envelopes price identically for a turn with
80k of 100k tokens cached.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten pricing prefix match + clamp corrupt usage
Three follow-ups on the longest-prefix pricing match landed in this PR:
- Prefix match now requires a dash boundary or end-of-string. The
longest-key sort alone still falsely landed "claude-opus-4-15" on
the "claude-opus-4-1" row, and "gpt-5.5-prod" on the "gpt-5.5-pro"
row (a 6x overcharge). Demanding the next character be "-" rules
out the lookalikes while keeping dated snapshots
("gpt-5.4-mini-2026-04-23", "claude-opus-4-7-20260414") landing on
their canonical row.
- Clamp every token count to >= 0. A corrupted upstream payload
(negative cached count, off-by-one in a fixture) could previously
produce a negative bill that masked real spend in the session
total tooltip.
- Tolerate a non-dict "cache_creation" (e.g. an upstream proxy
folded the field down to a single int). The current code raised
AttributeError mid-turn; now it falls back to the 5m-default
bucket so the rest of the cost calculation still runs.
Adds tests/test_pricing_edge.py with 20 adversarial cases covering
the boundary check, negative / None / zero token values across both
envelopes, cache_read > prompt corruption, the OpenAI long-context
threshold crossover on cache-inflated billable input, malformed
sub-objects, and unknown-provider degradation. Combined suite is
51 tests, all green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Surface Anthropic cache-read fallback and forward 1h breakdown
Two correctness gaps surfaced on the chat-style usage envelope:
1) Anthropic cache_read fell through to "uncached input" pricing when
the envelope arrived without the native ``cache_read_input_tokens``
key (e.g. via a proxy that only emits the mirrored
``prompt_tokens_details.cached_tokens`` block). Studio's canonical
``_build_usage_chunk`` always sets both so production traffic was
never affected, but the calculator should accept either as a
defense-in-depth measure. Add a fallback to read the mirrored
field when the native one is missing or zero; the native key still
wins when both are present so the math stays deterministic.
2) ``_build_usage_chunk`` dropped the ``cache_creation`` 5m / 1h
breakdown. Downstream ``calculate_cost`` then could not apply the
2x 1h premium and silently fell back to the 5m default,
underbilling 1h cache writes by 2x on chat-style traffic. Forward
the breakdown verbatim when the upstream usage carries it.
Tests grow by 4 (20 -> 24): two for the prompt_tokens_details
fallback (with native-precedence pin), one for the chunk shape, one
for the end-to-end pricing parity check at 1h.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add Anthropic fast_mode pricing multiplier
PR 5715 wires the fast-mode-2026-02-01 beta header + speed:"fast"
field through to Anthropic, but the cost calculator never learnt
about the matching 6x premium documented at
https://platform.claude.com/docs/en/build-with-claude/fast-mode
(Opus 4.7 standard $5/$25 per MTok, fast $30/$150).
This adds:
- ANTHROPIC_FAST_MODE_MULT = 6.0 constant.
- calculate_cost(..., fast_mode=True) applies the 6x to base input
AND output rates before any cache multipliers (cache mults stack
on top of fast per Anthropic docs).
- Provider+model gate: silently no-op on every model that is not
claude-opus-4-6 / claude-opus-4-7 so a stray fast_mode=True on
Sonnet/Haiku can never over-charge.
- model_priced label tagged "(fast)" so the cost tooltip can
surface which rate fired.
- pricing_snapshot now exposes fast_mode_mult so the frontend cost
panel doesn't have to hard-code 6.
7 new edge tests pin the math; existing 55 still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor explicit zero cache_read_input_tokens on Anthropic envelopes
The previous follow-up fell back to ``prompt_tokens_details.cached_tokens``
whenever the native ``cache_read_input_tokens`` was missing OR equal to 0,
even though the commit message stated the native key always wins when
present. A proxy that forwards a stale ``prompt_tokens_details`` block
alongside an authoritative ``cache_read_input_tokens: 0`` would then
inflate cache_read past the real native count, posting a false cache_read
line and bumping billable_input_tokens. Switch the gate to native-key
presence so an explicit zero stays authoritative; the mirror only kicks
in when the native key is absent. Add a regression test pinning the
explicit-zero precedence.
* Move fast_mode pricing back to #5715
The fast_mode 6x multiplier landed in two places at once -- here
(f66df7ba) and on #5715 (4f1afdb5) -- since both audits ran in
parallel. Drop the duplicate from this branch so the change lives
in its natural home (#5715, which introduces fast_mode itself);
this PR stays focused on the cache-read fallback + 1h breakdown.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shorten pricing comments for PR #5722
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: surface Anthropic document citations inline + in Sources panel
Anthropic's Messages API streams ``citations_delta`` events on
``content_block_delta`` when the request enables
``citations: {enabled: true}`` on document blocks. Each event carries
one citation pointing at the source document; previously they were
silently dropped, so reader-visible references never reached the chat
UI even when the model was citing properly.
The proxy now:
- dedupes by the type-specific anchor (char_location / page_location /
content_block_location / search_result_location) so re-cites of the
same span collapse onto a single footnote;
- injects ``[N]`` inline right after the matching text run;
- forwards the full list as a synthetic ``document_citations``
tool_event at ``message_stop`` so the Sources panel can render
per-document footnotes next to web_search / web_fetch citations.
Streams that never emit ``citations_delta`` stay byte-identical.
References:
- https://platform.claude.com/docs/en/build-with-claude/citations
- https://platform.claude.com/docs/en/build-with-claude/search-results
Tests (5 in test_anthropic_citations.py): passthrough, single
char_location, dedup of repeat citations, distinct sources get
distinct numbers, search_result_location supported.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: surface Anthropic document_citations in the Sources panel
The PR added a backend _toolEvent.type='document_citations' on
message_stop and an inline [N] marker in the assistant text, but the
chat-adapter only handles container_*/tool_*/sources from
web_search and web_fetch tool calls. Reviewers flagged that the
inline [N] markers had no matching footnote entries in the Sources
panel.
Capture the new event into a documentCitationParts buffer, convert
each citation dict into a Sources-panel source entry (using
document_title or search-result source URL plus cited_text as the
snippet), dedupe by id, and append to the final yield alongside
the existing web_search/web_fetch sourceParts.
* Studio: dedupe search_result_location citations by search_result_index
Anthropic's documented search_result_location citation shape carries
search_result_index, source, title, and start/end_block_index --
NOT document_index/document_title. The previous key keyed on
document_index + document_title + source + start_block_index, so
two distinct search results from the same source collapsed onto the
same footnote and the second [N] marker was lost.
Switch the search_result_location branch to key on the documented
fields, and pin the behaviour with a regression test asserting that
two citations sharing source/title but with different
search_result_index get distinct [1] [2] markers.
* Studio: keep each citation distinct across the end-anchor
Codex follow-ups on the citations PR:
* Backend _anthropic_citation_key now includes the end anchor for
every variant (end_char_index, end_page_number,
end_block_index). Anthropic ranges are start-AND-end pairs, so
a same-start / different-end pair is two distinct citations
that previously collapsed onto one footnote.
* Frontend documentCitationToSource ids include the position
fields (search_result_index, start/end char/page/block) instead
of being keyed on URL alone. Two citations from the same
document or two search_result_locations with the same source
now produce distinct Sources-panel entries, matching the
inline [N] numbering.
* Studio: key Sources list by per-citation id instead of url
Codex flagged that the Sources renderer keys badges on source.url,
so two Anthropic document citations sharing the same source URL
collide as React keys and one badge gets dropped (or duplicated).
The chat-adapter already mints a per-citation id that folds the
position fields (search_result_index, start/end char/page/block)
into the URL, so the two citations have distinct ids even when
their URL matches. Plumb that id through SourceData and use it as
the React key for both the measurement badges and the visible
SourceBadge list. Falls back to the URL when no id is supplied
(web_search and web_fetch source parts).
* Studio: enable Anthropic doc citations on input_document blocks
Plumb citations: {enabled: true} onto the translated Anthropic document
block (both base64 and URL source branches) so the upstream actually
emits citations_delta events. Without this opt-in the inline [N] +
Sources panel plumbing added in this PR is a no-op for real user
PDF / doc uploads.
Refs https://platform.claude.com/docs/en/build-with-claude/citations
Also add edge-case coverage for the citations_delta path:
malformed citations, mixed types per document, reversed indices,
missing document_index, non-int block indices, unknown citation
type, internal _key never leaking, footnote numbering across
content blocks, and the input_document wire-through itself.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reject unsafe citation sources, bound cited_text payload
Three follow-ups on top of #5718 surfaced by a deeper review pass:
1) javascript: / data: / vbscript: in citation source is XSS-able.
``documentCitationToSource`` was assigning ``cit.source`` straight
into ``Source.url`` and rendering it as an <a href>. A hostile
model emitting ``cit.source = "javascript:alert(document.domain)"``
would execute on click (openLink only intercepts URLs that contain
"://" or start with "mailto:", which both miss the javascript:
scheme). Restrict the navigable path to http(s):// only; anything
else falls back to the existing #anthropic-doc anchor and the
source title still renders the raw identifier for context. Also
reject CR/LF inside the URL string.
2) Frontend sources collapse distinct backend footnotes when the
citation type differs but positions match. char_location(0,5) and
page_location(0,5) over the same source previously deduped into
one entry because the id only carried position. Fold citation
type into the id anchor so the 1:1 mapping with inline [N]
markers is preserved across every citation shape.
3) ``cited_text`` was forwarded unbounded inside the synthetic
document_citations tool_event. The Sources panel trims to 240
chars for display anyway; for large RAG / search_result spans
(~10kB cited_text is plausible) this inflates SSE bytes 40x
for no UI benefit. Truncate server-side at 512 chars with an
ellipsis so the description-trim downstream still has room to
work and the wire stays bounded.
Tests grow from 21 to 22; existing 7 + edge 15 still green. Frontend
typecheck clean.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: apply http(s) URL guard to all Sources-panel link sources
The previous round only filtered ``cit.source`` inside
``documentCitationToSource``. Two parallel code paths still copied
provider/tool-controlled ``URL:`` text directly into clickable
``<a href>`` Sources-panel links:
* ``parseSourcesFromResult`` in chat-adapter.ts (legacy web_search /
web_fetch tool result parser)
* ``parseSearchResults`` in tool-ui-web-search.tsx (inline tool card)
A hostile tool response like ``URL: javascript:alert(1)`` or
``URL: data:text/html,...`` was therefore still rendered as a
navigable badge in the Sources panel.
Centralise the safe-URL test (``isSafeNavigableSourceUrl``,
``isSafeHttpUrl``) using ``new URL()`` + protocol allowlist + CR/LF
rejection, and apply it to both parsers. Unsafe blocks are dropped
rather than rewritten to a hash anchor because the web_search /
web_fetch parsers have no document-index fallback.
Citation conversion now uses the same helper so the in-place
http(s) regex and CR/LF check stay in one place.
* Shorten citation comments for PR #5718
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: surface Anthropic web_fetch as a standalone Fetch pill
web_fetch used to be silently bundled with the Search pill on the
assumption that "search returns URLs, fetch reads them" is the
typical workflow. Two problems with that:
- Anthropic bills each web_fetch invocation separately from
web_search hits, so combining them made the per-message cost
surface ambiguous.
- It blocked "just fetch this one URL" workflows where the user
already knows the page they want read and does not want a search
round-trip.
Adds:
- `webFetchToolsEnabled` to the chat-runtime-store, persisted to
localStorage under `unsloth_chat_web_fetch_tools_enabled`, with a
matching `supportsBuiltinWebFetch` capability flag and a
`setWebFetchToolsEnabled` setter.
- A new Fetch pill in the chat composer, rendered next to Images and
only when the active provider returns true from
`providerSupportsBuiltinWebFetch` (Anthropic today). The pill
defaults off so per-fetch billing is always a deliberate opt-in.
- chat-page bootstraps `webFetchToolsEnabled` from the same stored-
preference fallback the other pills use.
- chat-adapter reads `webFetchToolsEnabled` directly when deciding
whether to append "web_fetch" to `enabled_tools`, decoupling it
from `toolsEnabled` (Search).
Backend translation is unchanged: when `enabled_tools` already
contains "web_fetch", `_stream_anthropic` appends the
`web_fetch_20250910` / `web_fetch_20260209` tool exactly as before
(test_anthropic_web_fetch.py pins the standalone-only path at
`test_web_fetch_tool_appended_to_request_body` and the combined
path at `test_web_fetch_combined_with_web_search_and_code_execution`).
Frontend tsc passes.
* ci: re-trigger after transient GitHub API HTTP flake (checkout + ggml-org release fetch)
* Studio: include web_fetch in the disabled-tool guard axis
Reviewer P1 / High on PR #5742 (codex + gemini): after introducing
the standalone Fetch pill, `disabledToolGuard` still only branched on
`webSearchEnabledForThisTurn`. With Fetch ON and Search OFF the
system prompt would tell Claude "you do not have web search or web
fetch tools in this conversation", which contradicts the actual tool
schema being sent and suppresses `web_fetch` tool calls, defeating
the standalone-fetch workflow this PR adds.
Treat search and fetch as a single "any web tool enabled" axis. The
guard only needs to warn the model when no web tool is wired in for
this turn; once either pill is on the model can pick the right one
from the tool schema. The existing `webLabel` already covers both
names, so the user-visible guard text stays accurate in every
combination.
tsc clean.
* ci: re-trigger after transient infra flake on Windows prebuilt / actions/checkout
* Studio: route web_fetch through per-model version dispatch
The web_fetch tool body in `_stream_anthropic` hardcoded
`web_fetch_20250910` instead of calling `_anthropic_web_fetch_version`,
so Opus 4.6 / 4.7 and Sonnet 4.6 missed the `web_fetch_20260209`
dynamic-filtering variant. The picker, the unit tests for it, and a
deliberate "follow-up" note in `test_anthropic_web_fetch.py` already
existed; this just threads it through the emission site.
Mirrors how web_search and code_execution are dispatched per model.
Old models still resolve to `web_fetch_20250910` and continue to work.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shorten web_fetch comments for PR #5742
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: rewrite OpenAI Responses citation markers to markdown links
OpenAI's /v1/responses stream interleaves text deltas with inline
citation markers built from private-use codepoints (U+E200 / U+E201 /
U+E202) shaped like `citeSOURCE_ID`. The codepoints render
as garbled "E202" glyphs or empty boxes in most fonts, and the
markdown layer further strips them, leaving run-on text like
"citeturn1view0turn1view1turn3view0...". The url list still arrived in
the Sources panel via url_citation annotations, but the inline cite
hand-off into the prose was unreadable.
Rewrite each marker into `[N](URL)` when the matching url_citation
has already been recorded on this stream, and drop the marker
silently otherwise. The lookup uses a new `source_id` field captured
on `_record_url_citation` (accepts source_id / id / locator across
Responses API revisions). Annotations are now applied BEFORE the
delta text is rewritten so that markers and their resolving
annotation arriving in the same SSE event still resolve.
Reference: https://developers.openai.com/api/docs/guides/citation-formatting
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Preserve every source_id alias for a deduplicated url_citation
OpenAI's Responses stream cites the same URL under multiple
source_id markers when the model references different spans of the
same page. The previous dedup-by-URL kept only the first alias and
dropped the rest, so subsequent markers for the same URL never
resolved and got stripped from the prose. Switch the citation
record to a ``source_ids`` list and append new aliases on every
duplicate. The rewriter resolves any alias back to the same
citation number so the inline markers all collapse onto one footnote
rather than fanning out into bogus repeats.
Also collapse the two passes over ``all_url_citations`` in
``_record_url_citation`` into a single loop for clarity. Adds two
regression tests covering the alias-collision and mixed-shape cases.
* ci: re-trigger after flake in Studio GGUF Tool calling (rebased on main #5741 already)
* ci: re-run after transient CodeQL Python checkout auth flake
* Fix split-marker buffer + multi-source ids for PR #5713
The original rewriter only handles markers that arrive whole inside a
single response.output_text.delta event. OpenAI's stream chunks text
on byte-buffer boundaries with no awareness of the marker grammar,
so a marker can straddle two deltas (delta-1 ends with
"citetu", delta-2 starts with "rn0view0"). Each delta
was rewritten in isolation, so the half-marker leaked as garbled
"E200/E202" glyphs in the rendered prose.
Buffer the unterminated tail across deltas and concatenate it onto
the front of the next one so the rewriter sees a complete marker.
Flush the held-over tail on response.completed / response.incomplete /
[DONE], stripping any leftover private-use bytes so a never-closed
marker (truncated stream, missing annotation) never leaks.
Also handle the multi-source marker shape from the OpenAI docs --
citeid1id2 should expand to one bracket
link per resolvable id. The previous regex captured only the first
source id and silently dropped id2/id3.
Reference: https://developers.openai.com/api/docs/guides/citation-formatting
Tests: 21 new cases covering multi-source, locator suffix, marker
split across two and three deltas, unterminated marker on truncation,
late annotation resolving a buffered marker, idempotency, and the
head/tail split helper directly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Defer citation segments until url_citation annotation arrives
The split-marker buffer already concatenates a marker that straddles
two response.output_text.delta events. But when the annotation event
for a url_citation arrives AFTER the delta that contains its inline
marker (the typical OpenAI Responses ordering), the rewriter still
saw an empty lookup table at delta time and silently stripped the
marker. The URL kept showing up in the sources panel but the inline
link reference was permanently gone.
Add _rewrite_citation_markers_partial which leaves an unresolved
marker verbatim and reports has_unresolved=True. The streaming loop
buffers any closed segment that contains an unresolved marker into a
pending_citation_segments FIFO and drains the queue on every later
annotation event, on response.completed, on response.incomplete, and
on the [DONE] sentinel. Drain order is preserved so later clean text
does not leapfrog an earlier deferred segment. End-of-stream forces a
strip so no codepoint leaks if the annotation never arrived.
Add six regression tests covering single-pass resolution, the late-
annotation two-pass case, multi-source markers with partial
resolution, mixed known and pending markers in one segment, and
idempotency on marker-free input.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop unterminated citation tail to prevent cite-prefix plain-text leak
`_flush_pending_marker_tail` stripped the three private-use citation
codepoints from the held-over buffer, but left the literal ``cite``
keyword plus the source id behind as plain text. A stream ending
mid-marker therefore emitted user-visible garbage like
``Some text citeturn0view0`` instead of the intended clean prose.
``pending_marker_tail`` is by construction the suffix that starts at
an unclosed ``\\ue200`` opener -- the split helper guarantees there is
no closing ``\\ue201`` byte. Without that close the marker is
meaningless: the source id cannot be resolved to a URL and the user
prose before the opener was already emitted as ``head`` on the
originating delta. Bail out before the strip step and return the
empty string. As a belt-and-braces measure also drop any orphan
``cite<sid>`` literal at the head of the buffer in case a future
caller passes a partially-terminated tail.
Update the matching ``_simulate_delta_stream`` harness in the edge
tests so it mirrors the new flush logic, and add four regression
tests covering unterminated marker with surrounding prose, marker-
only inputs, prefix-only outputs, and the split-then-close path that
still must resolve to a link.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Defer multi-source markers until all ids resolve for PR #5713
`_rewrite_citation_markers_partial` previously treated a marker as
resolved when even one token in a multi-source marker resolved,
dropping any still-pending source ids. In streamed Responses events
the annotations for a multi-source marker can arrive across separate
`annotation.added` chunks, so the caller no longer buffered that
segment for retry and the late source id was lost from the inline
citation entirely.
Flag the marker unresolved whenever any token misses the lookup so
the streamer keeps the segment pending. End-of-stream force flush
still drops unresolved tokens through `_replace_openai_citation_markers`
so locator-style suffixes (which look like unresolved ids at the token
level but only appear at end-of-stream) render cleanly.
Updated the multi-source test to assert the new pending-then-flush
behavior; locator output now lands at force-flush rather than mid
stream.
* Shorten citation marker comments for PR #5713
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: add Anthropic fast_mode toggle + surface streaming refusals
Fast mode (beta `fast-mode-2026-02-01`) lets Claude Opus 4.6 and 4.7
generate output tokens up to 2.5x faster at 6x standard Opus
pricing. The toggle lives in Configuration → Provider when the
selected Anthropic model is Opus 4.6 or 4.7 and is otherwise
hidden. Backend gates the same prefixes a second time so a stale
frontend cannot make Anthropic 400 the request, and the
`fast-mode-2026-02-01` beta header is merged onto whatever other
betas the request already needed (code-execution, compaction).
Streaming refusals (`message_delta.delta.stop_reason="refusal"` on
Claude 4 models) now surface a short user-facing notice in the
assistant message before the translated OpenAI chunk emits the
existing `finish_reason="content_filter"`. Previously the chat
bubble truncated silently because the SSE stopped mid-stream with
no visible explanation. Per the upstream docs the conversation
must be reset before continuing, so the notice tells the user
exactly that.
Reference:
- https://platform.claude.com/docs/en/build-with-claude/fast-mode
- https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/handle-streaming-refusals
Tests:
- studio/backend/tests/test_anthropic_fast_mode_and_refusal.py (8 cases
pinning fast_mode pass-through on 4.6/4.7, silent drop on Sonnet /
Haiku / older Opus / None / False, and the refusal notice + finish
reason on a synthetic refusal stream).
* Studio: drop refused Anthropic turns from the next request
Anthropic's streaming-refusal guidance says the refused assistant
turn must be removed or updated before the next call -- otherwise
the safety classifier keeps refusing. The PR only added a
user-visible notice; the partial assistant output (plus the notice
itself) still rode the next request via toOpenAIMessage.
Tag the refusal turn with an HTML-comment sentinel emitted alongside
the notice. The chat-adapter checks for that sentinel in
toOpenAIMessage and returns null, so the refused turn is excluded
from outboundMessages. The notice still renders in the transcript
(HTML comments don't display), so users keep the explanation.
* Studio: filter None finish_reason entries in test helper
test_refusal_maps_to_content_filter expects only ['content_filter']
in the finish_reasons list, but the post-PR refusal path emits a
user-visible content notice chunk first. Every _content_chunk
carries 'finish_reason: None' by construction; the helper was
appending those, so the assertion saw [None, 'content_filter']
instead of ['content_filter'].
None is not a finish reason -- it's just mid-stream delta noise.
Skip None values in _finish_reasons so the helper reflects what
the test names actually claim to check. Same fix applies cleanly
to the other helper usages (pause_turn test expects [] and the
sibling stop test expects ['stop'], both unaffected).
* Studio: cover Anthropic fast-mode edge cases
Adds 19 cases on top of the 9 in test_anthropic_fast_mode_and_refusal.
The base file pins the happy path; this file fills in the cliffs:
* Dated-snapshot prefix matching: claude-opus-4-7-2026-02-01 and
claude-opus-4-6-2026-02-01 still gate fast_mode through, while
claude-opus-4-5-2025-08-01 and claude-sonnet-4-6-2026-02-01 do not.
* Strict opt-in: a future claude-opus-4-8 or claude-opus-5 does NOT
auto-enable fast_mode -- the prefix tuple must be bumped explicitly
when a new family is whitelisted upstream.
* Beta-header merge: fast_mode coexists with code-execution-2025-08-25
and compact-2026-01-12 in one comma-separated anthropic-beta header
with no duplicates and no truncation. Pins the value to the exact
fast-mode-2026-02-01 docs token so a typo would fail CI.
* Non-destruction: fast_mode=None produces byte-identical outbound
body and headers to the version that omits the argument entirely.
Same for fast_mode=False. Guarantees the upgrade path is
non-breaking on existing Anthropic streams.
* Refusal stream ordering: the user-visible notice precedes the
finish_reason chunk so a streaming UI paints text before flipping
to content_filter. Refusal sentinel emitted exactly once. Notice
rides a normal content delta chunk with finish_reason still null.
Partial assistant deltas survive before the notice.
* Provider-side refusal coverage: a refusal on Sonnet (not just Opus)
still emits the notice + sentinel + content_filter mapping, since
refusal handling is not gated on fast-mode capability.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Persist fastMode, drop refused user message on retry
Two follow-ups on #5715:
1) sanitizeInferenceParams stripped fastMode. fastMode is in
PERSISTED_INFERENCE_PARAM_KEYS but the storage sanitizer only kept
numeric fields plus systemPrompt and trustRemoteCode, so the new
toggle was silently dropped on reload and on the
/api/chat/settings round-trip. Save it the same way trustRemoteCode
is saved.
2) Refusal recovery now also drops the triggering user turn.
Returning null from toOpenAIMessage on the assistant side left the
user prompt that caused the refusal in the outbound history, so
the very next request would re-trigger the same classifier.
Anthropic's refusal-handling guidance is explicit on this: remove
the refused turn AND the user message that triggered it before
the next call. Implemented via a pre-pass that pops the trailing
user message when an assistant carries the refusal sentinel.
Typecheck clean.
* Studio: out-of-band refusal signal + fast-mode prefix/usage/pricing fixes
The text sentinel for the Anthropic refusal drop signal was spoofable:
any assistant message containing the literal
<!--studio:anthropic-refusal--> would prune the prior user + assistant
pair on the next request. Move the signal onto a separate _toolEvent
chunk that the chat adapter latches into
assistant.metadata.custom.anthropicRefusal; assistant text can no
longer control the pruner.
Tighten the fast-mode model gate (backend + frontend) to require a "-"
family boundary so claude-opus-4-70 / claude-opus-4-7b style IDs do
not get speed: "fast" on a naive startswith match.
Use survivingMessages for the image / audio attachment scan so a
refused user turn does not gate or mis-attribute the next non-refused
turn.
Propagate Anthropic usage.speed onto the OpenAI-style usage chunk and
apply the documented 6x fast-mode multiplier in the cost calculator
(stacks with prompt-cache multipliers per the docs); expose the new
multiplier on the pricing snapshot for the UI tooltip.
Tests cover the tool-event chunk shape, the prefix-collision rejects,
usage.speed propagation, the 6x pricing math, and that the visible
refusal text carries no embedded sentinel.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shorten fast-mode and refusal comments for PR #5715
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: PDF / document attachments for Anthropic + OpenAI
Studio's local-GGUF chat already supports image attachments via the
`image_url` content part shape. PDFs and other documents had no
plumbing for the external-provider path: there was no normalised
content type the frontend could send that translated to Anthropic's
native `document` block or OpenAI's `input_file`.
Add a Studio-side `input_document` content part on assistant /
user messages with three shapes:
{type: "input_document",
file_data: "data:application/pdf;base64,<DATA>",
filename?: "name.pdf",
media_type?: "application/pdf"}
{type: "input_document",
file_url: "https://example.com/doc.pdf",
filename?: "doc.pdf"}
Translation:
- Anthropic Messages API: emits a `document` block with
`{source: {type:"base64", media_type, data}}` or
`{source: {type:"url", url}}`, plus an optional `title` from
`filename`. PDFs are extracted server-side by Anthropic per their
vision/document docs and counted toward input tokens.
- OpenAI Responses API: emits `{type:"input_file", file_data |
file_url, filename?}`. PDFs are extracted server-side.
Empty / unparseable `input_document` parts are silently dropped so
a malformed frontend payload can't blow up the request.
Tests:
- New `test_multimodal_document.py` with 6 cases pinning the
outbound body shape for base64 + URL inputs on both providers,
and the empty-part drop behavior on both.
- The Anthropic assertions strip the prompt-cache wrapper
(`cache_control:{type:ephemeral}` that the tail-message caching
layer adds) before comparing the document core fields, so this
test stays focused on the translation, not the caching layer.
Live verified end-to-end against both providers: a 363-byte
single-page "HELLO" PDF, base64-encoded, attached as a `document`
block to Opus 4.7 and as an `input_file` to gpt-5.5. Both models
correctly extracted the word "HELLO" from the PDF.
Follow-up (out of scope):
- Pydantic schema entry on ChatMessage.content for `input_document`
(today it rides through because ChatCompletionRequest uses
extra=allow). Will tighten when the frontend attach button lands.
- Frontend file-picker UX for non-image attachments on the external
provider path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: gate empty-content msg + skip empty data-URI payload
Gemini High + Codex P2 on PR #5689:
1. Anthropic translation appended an empty `anthropic_parts` array
when every part was dropped (e.g. user sent only an unparseable
input_document). Anthropic 400s on "messages.N.content: at least
one block is required". Skip the whole-message append when no
parts survived. The OpenAI Responses path already had the
equivalent guard, so this brings the two providers into parity.
2. `data:application/pdf;base64,` with no payload (or whitespace-only)
parses to an empty `source.data` string. Anthropic rejects that
with 400 as well. Skip the document block before constructing it.
Plus 2 new test cases pinning both behaviors:
- `test_anthropic_empty_only_document_drops_whole_message`: confirms
a turn whose only content is an unparseable input_document does
NOT make it onto the outbound `messages` array.
- `test_anthropic_empty_data_uri_payload_is_dropped`: confirms an
empty-payload data-URI is filtered out at translation time.
(Note re: gemini's other High note about adding `input_document` to
the Pydantic ContentPart union -- ChatCompletionRequest is configured
with `extra=allow` so the part rides through today. Tightening the
union belongs with the frontend attach-button PR that surfaces the
field; called out as follow-up in the PR description.)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: register input_document in ContentPart + builder
Reviewer caught that the translation code on the external_provider
side was unreachable from a real ChatCompletionRequest:
- ContentPart is a discriminated Union of (text, image_url) only, so
any `{"type": "input_document", ...}` part was rejected by Pydantic
at request parsing with a discriminator error before the helper
could see it.
- _build_external_messages in routes/inference.py only walked text
and image_url parts, so even with a permissive schema the document
parts would have been silently dropped instead of forwarded to
the per-provider translator.
Fixes:
- Add InputDocumentContentPart with optional file_data / file_url /
filename / media_type and Tag("input_document") on the Union.
- Extend _build_external_messages to pass input_document through as
a plain dict for vision-capable providers (so external_provider's
existing Anthropic `document` and OpenAI Responses `input_file`
mappers actually run) and strip them on non-vision providers.
Tests added: schema accepts input_document, builder passes it to
vision providers, builder strips it on non-vision providers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: validate file_data before preferring over file_url
Codex P2 caught that the OpenAI input_document translator treats any
truthy file_data as valid and never falls back to file_url. That
means a malformed `data:application/pdf;base64,` (empty payload) or
a whitespace-only data URI gets forwarded as `file_data=""` and
400s the whole turn, AND silently discards a perfectly recoverable
file_url on the same part.
Mirror the Anthropic-side guard onto the OpenAI Responses path:
treat any "data:" URI with no actual base64 payload as missing and
fall through to file_url. Standalone-empty data URIs (no fallback)
are dropped entirely instead of being sent to the wire.
Tests added: empty data URI + valid file_url -> file_url wins,
whitespace-only data URI + valid file_url -> file_url wins,
empty data URI without fallback -> part is dropped.
* Address review: Anthropic side also falls back to file_url on empty data URI
Codex P2 follow-up to my earlier fix: I added the empty-data-URI ->
file_url fallback to the OpenAI Responses translator but missed
the Anthropic translator, which still `continue`d on empty payloads
and discarded an otherwise valid file_url on the same part. Result:
when the frontend supplied both file_data (placeholder / broken)
AND a working file_url, Anthropic silently lost the attachment;
when the message contained only that part, the whole message could
be dropped before reaching the wire.
Mirrored the OpenAI guard: any "data:" URI with no actual base64
payload (`data:application/pdf;base64,` or whitespace-only) is
treated as missing, and the file_url branch takes over. The
all-parts-dropped guard further down already handles the
no-fallback case.
Tests added: empty data URI + valid file_url -> URL source on the
wire with the filename preserved; whitespace-only data URI + valid
file_url -> URL source on the wire.
* Address review: gate input_document passthrough to anthropic + openai
Codex P1: only `_stream_anthropic` and `_stream_openai_responses`
have explicit translation logic for input_document parts (the former
maps to {type:"document", source:...}, the latter to
{type:"input_file", file_data|file_url}). Every other provider
(gemini / mistral / kimi / openrouter / deepseek / qwen / custom)
goes through the generic /chat/completions passthrough that forwards
`messages` verbatim, so any input_document part on a non-vision
route on those providers would 400 with an unknown content_part
type.
Added `_INPUT_DOCUMENT_PROVIDERS = frozenset({"anthropic", "openai"})`
constant and gated the pass-through branch on `provider_type in
_INPUT_DOCUMENT_PROVIDERS`. Every other provider strips the part
(text content survives). Threaded provider_type through from
_proxy_to_external_provider's call site.
Tests updated: vision + provider in {anthropic, openai} still
forwards; six unmapped providers (gemini/mistral/kimi/openrouter/
deepseek/qwen) strip the part; missing provider_type strips
defensively. The existing non-vision drop test still passes.
* Fix stale web_fetch tool-version assertion after merging main
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: wire OpenAI Responses server-side context compaction
The OpenAI Responses API accepts a `context_management` field that
enables server-side compaction. When the rendered prompt crosses the
configured threshold, the API runs a server-side compaction step and
the request continues against the compacted prefix. No beta header
and no dated version pin are required, per the docs.
Changes:
- Add `compaction_threshold: Optional[int]` (ge=1_000, le=2_000_000)
to ChatCompletionRequest. Thread through `routes/inference.py` ->
`stream_chat_completion` -> `_stream_openai_responses`.
- In `_stream_openai_responses`, when threshold is set AND the base
URL points at cloud OpenAI (api.openai.com), attach
`context_management: [{type:"compaction", compact_threshold:N}]`
to the outbound body. Non-cloud bases (ollama, llama.cpp, "custom"
presets) silently drop the field so we don't 400 those servers.
- Add `test_openai_compaction.py` with 4 cases: cloud OpenAI sets
the field verbatim, low-threshold probe passes through (we don't
clamp on the OpenAI side because the API accepts whatever),
non-cloud base drops the field, omitted threshold leaves body
untouched.
Live verified against the real OpenAI API on gpt-5.5:
`context_management:[{type:"compaction", compact_threshold:200000}]`
returns 200 with no error.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: accept Azure OpenAI base URLs + raise compaction floor
Two reviewer follow-ups on the OpenAI compaction PR:
1. The `is_openai_cloud = "api.openai.com" in self.base_url` check
excluded Azure OpenAI Foundry, even though Azure exposes the
same /v1/responses extensions (context_management,
prompt_cache_retention, container shell). Users on Azure saw
their compaction toggle silently no-op. Broadened the check to
also match `*.openai.azure.com` and made it case-insensitive so
URLs copy-pasted from the Azure portal still resolve. Non-cloud
OpenAI-compatible servers (ollama / llama.cpp / vLLM / "custom"
preset) still fall outside the gate.
2. The schema floor on compaction_threshold was ge=1_000, which is
well below the upstream Responses API's effective minimum
(vercel/ai#12486, langchain-ai/langchain#35464 report
`compact_threshold is not enabled` 400s on Azure at 100k; cloud
uses 200k as the canonical example). Raised the floor to 10k
so obvious typos surface as a clean 422 from FastAPI rather than
an opaque upstream 400 the user has to debug from the SSE
stream.
Tests added: Azure base URL carries both context_management and
prompt_cache_retention; mixed-case Azure URLs match; schema rejects
9_999 and accepts 10_000.
* Address review: drop schema-level compaction floor (cross-provider regression)
Codex P2 follow-up on the previous floor bump: ge=10_000 was
enforced globally at the ChatCompletionRequest layer, but the field
is documented as a no-op on every non-cloud OpenAI base and every
non-OpenAI provider. With the global floor, an Anthropic / ollama
/ llama.cpp / custom request that happens to carry compaction_threshold
below 10k was rejected with 422 at request validation time instead
of being silently ignored as the description promised.
Reverted the schema floor to ge=1 (any positive int) and rewrote
the description to call out per-provider routing: OpenAI cloud's
effective floor is around 200k and surfaces upstream 400s below
that; _stream_anthropic clamps sub-50k values up. Per-provider
helpers stay the single source of truth on the floor.
Test updated to pin: zero is still rejected, but every positive
value (1, 5_000, 9_999, 10_000, 200_000) passes schema validation.
* Address CodeQL: hostname-anchored OpenAI cloud detection
CodeQL py/incomplete-url-substring-sanitization fired on
`".openai.azure.com" in _base`. An attacker who controls the
configured base_url could slip cloud-only request body fields
(prompt_cache_retention, context_management compaction, container
shell) to an arbitrary server with:
https://evil.com/api.openai.com/v1https://api.openai.com.attacker.com/v1https://attacker.com/.openai.azure.com/v1https://my-resource.openai.azure.com.attacker.com/openai/v1
Replaced the substring check with a `_is_openai_family_cloud`
helper that runs urllib.parse.urlparse on the URL and matches the
lowercased hostname exactly (`api.openai.com`) or via `endswith`
on the leading-dot suffix (`.openai.azure.com`). Both halves are
host-anchored so path / fake-subdomain bypasses fail.
Test added: every attacker-controlled bypass shape above must NOT
carry context_management OR prompt_cache_retention on the wire.
Existing Azure and openai.com tests still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: scope compaction_threshold description to OpenAI on this branch
Codex P2: the field description on this PR mentioned Anthropic
compaction behavior, but the Anthropic wiring lives on PR 5686
(separate branch). On feat/openai-compaction alone, _stream_anthropic
has no compaction_threshold parameter, so the field is silently
ignored for Anthropic requests and the doc claim was misleading.
Trimmed the description to OpenAI cloud + Azure Foundry only on
this branch. PR 5686 already re-adds the Anthropic clause via its
own change, so the rebase / merge order on main will land the
combined description naturally once both PRs ship.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: wire Anthropic server-side context compaction
Anthropic ships server-side context compaction as a beta
(`compact-2026-01-12`). When the rendered prompt crosses the
configured input-token threshold, Anthropic runs an extra LLM pass
that summarises older turns and the request continues against the
compacted prefix. The response carries the original top-level fields
plus a new `context_management` block (with `applied_edits`) and
`usage.iterations[]` accounting per pass.
Per the docs the feature is currently supported on Opus 4.6, Opus 4.7,
Sonnet 4.6, and Mythos preview. The minimum threshold is 50k tokens;
under-50k requests 400.
Changes:
- Add prefix gate + helper `_anthropic_supports_compaction` plus
constants `_ANTHROPIC_COMPACTION_PREFIXES`, `_ANTHROPIC_COMPACTION_BETA`,
`_ANTHROPIC_COMPACTION_TYPE`, `_ANTHROPIC_COMPACTION_MIN`.
- Add `compaction_threshold: Optional[int]` to ChatCompletionRequest
(50k ge bound, 2M le bound). Thread through `routes/inference.py`
-> `stream_chat_completion` -> `_stream_anthropic`.
- In `_stream_anthropic`, when threshold is set AND the model
accepts compaction, attach `context_management.edits[{type:
"compact_20260112", trigger:{type:"input_tokens", value:N}}]` to
the outbound body. Sub-50k values are clamped up to 50k to keep
the request well-formed.
- Refactor the anthropic-beta header builder to merge any combination
of `code-execution-2025-08-25` + `compact-2026-01-12` flags into
one header value. Unrelated betas added at the registry level still
pass through.
- Add `test_anthropic_compaction.py` with 16 cases: gate matrix
(every doc-listed model), correct body shape, threshold clamping,
beta header merge with code execution, silent no-op on unsupported
models, omitted-threshold pass-through.
Live verified end-to-end against the real Anthropic API:
`compact_20260112` accepted on Opus 4.7, response carries
`context_management.applied_edits` + `usage.iterations[]` as
documented. (The first WebFetch-summarised version of these docs
suggested `compact_20260120`; the actual API only accepts
`compact_20260112`, matching the beta-header date. Worth pinning
behind a test so a future doc update can't drift back.)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: drop ge=50_000 clamp + parse usage.iterations[]
Two reviewer follow-ups on the compaction PR:
1. Pydantic ge=50_000 on compaction_threshold was dead code.
FastAPI rejected sub-50k threshold values with a 422 before the
`max(int(...), _ANTHROPIC_COMPACTION_MIN)` clamp in
_stream_anthropic could ever fire. Relaxed the floor to ge=1 so
the in-helper clamp actually does its job; the schema comment
now explains why this is intentional. Added a regression test
that posts a value of 1 and 49_999 through the real request
schema.
2. Anthropic publishes per-iteration token counts in
`usage.iterations[]` whenever a fresh compaction has run, and
the top-level input_tokens / output_tokens cover only the
`message` iteration -- billing must add the compaction
iterations on top. Aggregate compaction iteration tokens into
`last_usage["compaction_input_tokens" / "compaction_output_tokens"]`
so the cost surface (PR 5690) can read them without re-walking
the array, and surface both figures in the closing stream
summary log. Added two tests: one that pins the aggregation on a
compacted turn and one that pins `None` when no fresh
iterations land (so re-applied compaction blocks don't double-bill).
Sourcing: https://platform.claude.com/docs/en/build-with-claude/compaction
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: round-trip Anthropic compaction blocks across turns
Codex P1: once context_management is enabled and Anthropic runs
server-side compaction mid-stream, the response carries a
`{type:"compaction", content:"<summary>"}` content block on the
assistant message. The translator only handled text_delta and
input_json_delta on content_block_delta, so the compaction block
was silently dropped. Worse, the request schema's ContentPart
discriminated Union didn't accept `type:"compaction"`, and
_build_external_messages didn't pass it through, so even a
hand-crafted assistant message carrying the block would 422 at
parse time. Net result: Anthropic re-compacted from scratch on
every subsequent turn, wasting input tokens and reasoning budget.
End-to-end backend wiring of the round-trip:
1. SSE translator. _stream_anthropic now tracks a `current_compaction`
state slot. content_block_start with type=="compaction" seeds it
(Anthropic may include the summary on the start event AND/OR
stream it via text_delta events on the same block index --
handle both). text_delta inside a compaction block routes into
the compaction buffer instead of the user-visible content
stream, since the summary is opaque internal state, not
assistant prose. content_block_stop emits a `compaction_block`
tool_event carrying the full summary so the chat-adapter can
persist it. compaction_blocks_seen is surfaced in the closing
summary log.
2. Pydantic schema. Added CompactionContentPart with Tag("compaction")
on the ContentPart Union so requests carrying the block parse
cleanly. Required `content` field with a docstring pointing at
the Anthropic docs.
3. Message builder. _build_external_messages forwards compaction
parts on both vision and non-vision paths; the per-provider
stream helper decides whether to forward to the wire (Anthropic
does; other providers ignore the part). When a non-vision route
ends up with a single text part, collapse back to a string
so providers that don't accept content arrays still get the
expected shape.
4. _stream_anthropic outbound translator. {type:"compaction"} parts
on an assistant message land on the wire verbatim. Empty/missing
`content` is skipped so a malformed stored block can't 400
Anthropic.
Tests added (5): stream emits compaction_block tool event with the
summary intact; user-visible content stream does NOT carry the
summary text; outbound body forwards compaction parts verbatim on
the next turn; Pydantic schema accepts the part; builder passes
it through on both vision and non-vision provider routes.
Frontend follow-up: the chat-adapter needs to persist the
compaction_block tool_event onto the stored assistant message so
turn N+1 includes it in payload.messages. Pinned in the PR
description.
Sourcing: https://platform.claude.com/docs/en/build-with-claude/compaction
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: gate compaction-part passthrough to Anthropic only
Codex P1: my previous round-trip change preserved {type:"compaction"}
parts on every provider route in _build_external_messages. That
meant a chat history with prior compaction state silently leaked
the Anthropic-specific block to OpenAI/DeepSeek/Mistral/Gemini/
Kimi/OpenRouter on a provider switch, where generic
/chat/completions passthrough hands the unknown content type to
the upstream API and 400s the whole turn.
Added a `provider_type` kwarg to _build_external_messages and
gated the compaction forwarder on `provider_type == "anthropic"`.
Every other value (including the legacy None for callers that
don't pass it yet) strips the part. The Anthropic stream helper
still maps it to a native `compaction` block on the wire.
Threaded provider_type through from _proxy_to_external_provider's
call site.
Tests updated: vision + provider="anthropic" still forwards; six
non-anthropic providers strip the part; missing provider_type
strips defensively; non-vision + anthropic still forwards; non-vision
+ non-anthropic collapses back to a text string.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: wire Anthropic web_fetch server-side tool
Studio's Anthropic passthrough only forwarded web_search and
code_execution when enabled_tools was set. Asking Claude through Studio
to fetch a URL produced no fetch (the tool was not in the outbound
tools array), so users had to fall back to web_search even when they
already had the exact URL they wanted.
This change opts in web_fetch_20250910 when enabled_tools contains
"web_fetch". The new tool entry is appended alongside any existing
web_search / code_execution entries:
{"type": "web_fetch_20250910", "name": "web_fetch", "max_uses": 5}
No anthropic-beta header is required (web_fetch is GA); the existing
code-execution-2025-08-25 flag continues to merge cleanly when both
tools are enabled in the same turn.
SSE translation mirrors the web_search path. A `server_tool_use` block
with name="web_fetch" emits a `tool_start` _toolEvent carrying the
URL the model asked to fetch; the matching `web_fetch_tool_result`
block emits a `tool_end` _toolEvent whose result string follows the
Title / URL / Snippet shape parseSourcesFromResult on the frontend
already expects, so the source pill renders identically. Error blocks
(`web_fetch_tool_error`) are surfaced as "Error: <error_code>" matching
the code_execution error path.
The final "Anthropic stream complete" log line picks up web_fetch_
requested / web_fetch_invocations / web_fetch_urls so support reports
of "the model did not fetch anything" can be triaged from the log.
Verified end to end against claude-haiku-4-5 with
`enabled_tools=["web_fetch"]`: the model emitted tool_start with
url=https://example.com and tool_end with the page Title + URL +
Snippet, plus the assistant message correctly read back "Example
Domain" as the title.
Tests:
- 5 new unit tests in test_anthropic_web_fetch.py covering tool
registration, the combined web_search + web_fetch + code_execution
request body, the pill-off case, and SSE translation for both
success and error paths.
- All 242 existing Anthropic + OpenAI provider tests still pass.
The enabled_tools field description in models/inference.py is updated
so OpenAPI consumers see the new option.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* web_fetch: title fallback to URL, log parse failures, drop dead checks
Three review nits on the previous commit:
1. `_format_web_fetch_result` left `title` empty when Anthropic omitted
`document.title`. The frontend `parseSourcesFromResult` only emits
a source pill when both `Title:` and `URL:` lines are present, so
fetches against pages without an HTML title tag silently lost
their citation in the UI. Fall back to `title = title or url`,
matching the web_search formatter.
2. The broad `except Exception` around `json.loads(buffer)` for the
web_fetch input swallowed the failure with no trace. Log at debug
so a malformed partial_json buffer can be triaged from the server
log without changing behavior.
3. `inner` was already sanitised to a dict at the matching
content_block_start and `_format_web_fetch_result` always returns
a non-empty string (defaulting to "(fetch complete)"), so the
`isinstance(inner, dict) else {}` guard and the
`result_text or "(fetch complete)"` fallback at the emit site
were dead code. Removed.
Added a test exercising the titleless path so the fallback stays
covered.
* chat-adapter: emit source pills for web_fetch tool calls
`parseSourcesFromResult` was only wired up for tool calls where
`toolName === "web_search"`, so the Title / URL / Snippet block the
backend formatter emits for `web_fetch_tool_result` never reached the
source-pill renderer. Users saw the raw tool result in the tool card
but the dedicated source-pill row at the message tail stayed empty.
Both web_search and web_fetch ship the same text shape today, so the
fix is to broaden the gate.
* Address review: wire web_fetch from Search pill + fix pause_turn truncation
Two reviewer follow-ups on the Anthropic web_fetch PR:
1. The backend tool wiring landed but the frontend chat-adapter
never put `web_fetch` in `enabled_tools`, so toggling the Search
pill only ever attached `web_search` -- web_fetch was unreachable
from the UI. Added providerSupportsBuiltinWebFetch() (Anthropic
today) and paired the entry with the existing Search pill, since
the canonical workflow is "search returns URLs, fetch reads
them" and there is no separate UI toggle yet.
2. `pause_turn` from Anthropic's stop_reason vocabulary fell through
the finish_reason map's "stop" default, which the OpenAI-format
client renders as end-of-message and truncates the answer. Per
the docs pause_turn means "Claude paused a long server-tool
turn (web_search / web_fetch) and will resume". Mapped to None
and skipped the chunk emission so the SSE stream still ends with
[DONE] on message_stop but no terminal finish_reason lands on
the client. While there: added explicit mappings for `tool_use`
(-> tool_calls) and `refusal` (-> content_filter) which were
also falling through to "stop".
Tests added: pause_turn emits no finish_reason, end_turn still
emits "stop", refusal maps to "content_filter".
Sourcing: https://platform.claude.com/docs/en/api/messages#response-stop-reason
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: per-session cost calculator + /api/providers/pricing endpoint
Neither the Anthropic Messages API nor the OpenAI Responses API
reports a `cost` field on the response. Both expose detailed token
counts (input, output, cache hits, server-tool invocations); pricing
multipliers live in the provider docs. The frontend's "cost so far"
display was impossible without scraping the server log.
Land the math + a snapshot endpoint so the cost calculator can run
client-side from the existing usage chunk plumbing. The actual UI
hookup belongs in a frontend follow-up (and is gated on PR #5670's
usage-chunk emission landing so the frontend sees the usage block
in the first place).
Changes:
- New `core/inference/pricing.py` with:
- Per-MTok base pricing tables for every active Anthropic and
gpt-5.x family member. Dated snapshots inherit the canonical-id
price via prefix match so future snapshots cost the same as the
canonical id until pricing changes.
- Shared multipliers for Anthropic cache writes (5m: 1.25x, 1h: 2x)
and reads (0.1x); OpenAI cache reads (0.1x); Anthropic server
tool surcharges ($10 / 1k web_search, $0.05 / hour code_exec
beyond the 50-hour daily free tier).
- `calculate_cost(provider, model, usage)` returns a per-turn USD
breakdown plus billable token counts, with priced=False for
unknown models so the UI can still render token counts.
- `pricing_snapshot()` returns the whole table for the frontend
so it doesn't re-implement the multipliers.
- New `GET /api/providers/pricing` returning the snapshot, scoped
behind the existing auth dependency.
- New `backend/tests/test_pricing.py` with 12 cases pinning the
math against documented values: base input/output multiplication,
5m / 1h / read multipliers, default-to-5m fallback when the
breakdown is absent, web_search per-1k pricing, code_execution
per-hour pricing, dated-snapshot fallback, OpenAI cache-read
discount accounting (cached tokens subtracted from full-price
bucket and re-billed at 0.1x), unknown model graceful-degrade,
and the snapshot endpoint shape.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: verified OpenAI pricing + fix billable input double-count
Address the cost-calculator review:
- OpenAI prices were 2-6x under the actual published rates.
Cross-checked the live developers.openai.com/api/docs/pricing page
and replaced every entry. gpt-5.5 is 5/30, gpt-5.5-pro is 30/180,
gpt-5.4 is 2.5/15, gpt-5.4-mini 0.75/4.5, gpt-5.4-nano 0.20/1.25,
gpt-5.3-codex 1.75/14. Added chat-latest alias to the canonical
chat-snapshot rate. Dropped o3 / o4 / gpt-4.5 rows that are no
longer listed on the page; calculator returns priced=False instead
of silently billing at zero.
- billable_input_tokens was double-counting cached tokens for
OpenAI. Anthropic excludes cache_* buckets from input_tokens so
we add them; OpenAI folds cache_read_input_tokens into
input_tokens already, so the tooltip read 1.8M for a 1.0M bill.
Branched the math by provider and added a regression test.
Sourcing notes in the module docstring updated.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: canonical 4.5 ids, long-context tier, OpenAI tool fees
Three Codex P1 follow-ups on the cost calculator:
1. Canonical Anthropic 4.5 ids missing from ANTHROPIC_PRICING.
claude-opus-4-5 / claude-sonnet-4-5 / claude-haiku-4-5 (no date
suffix) are the ids used by backend defaults
(PROVIDER_REGISTRY['anthropic'].default_models), but the table
only had the dated forms. _lookup's prefix fallback doesn't help
because the canonical id is SHORTER than the dated key, so
str.startswith goes the wrong way and the calculator returned
priced=False + zero cost. Added the canonical aliases for
opus-4-5, sonnet-4-5, haiku-4-5, and opus-4-1.
2. OpenAI long-context tier. gpt-5.5 and gpt-5.4 cross over at
272k input tokens to a 2x input / 1.5x output rate (gpt-5.5:
$5/$30 -> $10/$45; gpt-5.4: $2.50/$15 -> $5/$22.50). Turns past
the threshold were systematically undercounted at headline
rates. Added long_context_threshold / long_context_input_per_mtok /
long_context_output_per_mtok columns and a tier-selection step
in calculate_cost; model_priced gains a "(long-context >272000)"
suffix when the higher tier applies so the tooltip can show
which rate was used. gpt-5.5-pro / gpt-5.4-pro / mini / nano /
codex have no published long-context tier today, so they keep a
single rate.
3. OpenAI server-tool surcharges. web_search is $10/1000 calls and
the hosted shell container is $0.03 per 20-minute session on the
default 1g tier (~$0.09/hr). server_tools_usd was previously
stuck at 0.0 for OpenAI even when web_search and shell tools
fired, so sessions with tool use understated cost. Added
OPENAI_WEB_SEARCH_USD_PER_1K and OPENAI_CONTAINER_USD_PER_HOUR
constants plus a parallel of the Anthropic surcharge block that
reads counts from usage["openai_tool_use"]. The SSE translator
wires the counts in a follow-up commit; the calculator is now
ready for them. pricing_snapshot also exposes both constants so
the frontend tooltip can render the per-call rate.
Existing tests updated to stay in the short-context tier where they
were testing base rates; new tests pin canonical 4.5 lookups,
long-context crossover on gpt-5.5/gpt-5.4, the absence of crossover
on mini/nano/codex, and OpenAI tool surcharges (web_search,
container hours, combined total).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: wire OpenAI image_generation tool
OpenAI's Responses API exposes server-side image generation as a
tool entry (`{type: "image_generation"}`); the result comes back as
an `image_generation_call` output item with the base64 image on
`result`, the actual prompt used on `revised_prompt`, plus `size`,
`quality`, `output_format`, `background`. The model decides when to
call the tool based on the user's request; rendering uses one of
the gpt-image-* backbones server-side.
Available on every gpt-5.x family member plus gpt-4.1, gpt-4o, o3,
o4-mini per the docs.
Changes:
- Append `{type:"image_generation"}` to the Responses request tools
array when `enabled_tools` carries `image_generation` AND the base
URL points at cloud OpenAI. Non-cloud bases (ollama, llama.cpp,
"custom" presets that collapse to provider="openai") silently drop
the tool to avoid 400s.
- Mirror the same logic in `_build_body` (the post-expiry retry
builder) so retries carry the same tool set as the original
attempt.
- Handle `image_generation_call` items in
`response.output_item.done`: emit `tool_start` with
`arguments:{kind:"image", prompt:<revised_prompt>}` and `tool_end`
with `image_b64`, `image_mime`, `size`, `quality`, `background`
so the chat adapter can render an inline preview. Image bytes go
on the tool_end chunk; no extra fields on the chat-completions
envelope so the OpenAI SDK shape stays clean.
- Add `import time` (used for synthesised tool_call_id fallback).
- Add `test_openai_image_generation.py` with 5 cases: tool entry on
cloud OpenAI, combined with web_search + code_execution
(verifies all three coexist), non-cloud drop, omitted pill leaves
body untouched, output item translation produces the expected
tool_start + tool_end chunks.
Live verified end-to-end: `gpt-5.4-mini` with `image_generation`
tool returned an `image_generation_call` carrying ~1MB of base64
PNG plus the gpt-image backbone's revised prompt.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use time.time_ns() for synthesised image_generation tool_call_id
Gemini medium on PR #5688: `int(time.time() * 1000)` has 1ms
resolution; two image generations resolving in the same millisecond
would collide on the synthesised id. Bump to nanoseconds.
(In practice the upstream `image_generation_call` item always carries
its own `id`; the synthesised fallback only fires when OpenAI omits
it -- rare, but cheap to harden.)
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: support Anthropic 1h cache TTL via prompt_cache_ttl field
Anthropic exposes two ephemeral cache pools per request: the default
5-minute pool, and a 1-hour pool selected by attaching `ttl:"1h"` to
the `cache_control` marker. 1h writes are billed at 2x base input vs
1.25x for 5m, but reads stay at 0.1x for both, so a single extra read
landing more than 5 minutes after the write pays off the premium.
Studio hardcoded the 5m pool via `cache_control: {type:"ephemeral"}`
on both breakpoints. For chats with multi-minute idle gaps (people
juggling tabs, long-running tool calls between turns), the cache
expires before the next turn and every read becomes a cache_creation,
not a cache_read -- exactly the case where the 1h pool wins.
Changes:
- Add `prompt_cache_ttl: Optional[Literal["5m", "1h"]]` to
ChatCompletionRequest. Default (None) preserves today's 5m behavior.
- Thread through `routes/inference.py` ->
`stream_chat_completion` -> `_stream_anthropic`.
- Build a shared `cache_marker` dict in `_stream_anthropic`; attach
`ttl` only when the request asks for one of the two valid values.
Unknown TTL strings are silently dropped to avoid sending malformed
markers (the upstream API would 400).
- Apply the same marker to both existing breakpoints (system block at
line 1175 and the latest-message tail at line 1198 / 1213) so the
pool selection is consistent across the whole prefix.
- Add `test_anthropic_cache_ttl.py` with 11 parametrized cases
pinning the outbound body shape: omitted -> default marker;
explicit `5m`/`1h` -> ttl field set; unknown values dropped;
caching off -> no markers at all.
Verified upstream that `cache_control: {type:"ephemeral", ttl:"1h"}`
is accepted by the Anthropic API today; no beta header required.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Relax prompt_cache_ttl to Optional[str] (Codex P1)
Declaring `prompt_cache_ttl` as `Optional[Literal["5m", "1h"]]` made
FastAPI/Pydantic 422 the request before _stream_anthropic could even
see the field. The whole point of the downstream drop-unknown-values
behaviour was to keep a stale frontend from crashing the request;
the strict Literal at the request layer defeated that.
Loosen the schema to Optional[str]; the existing in-helper guard
already restricts forwarded values to {"5m", "1h"} (everything else
is silently dropped). Test suite stays unchanged -- the bogus-value
cases in test_anthropic_cache_ttl.py already pass arbitrary strings
through and assert they are dropped before the wire.
* Address review: confirm extended-cache-ttl beta header is GA
Reviewer asked whether the 1h cache TTL still requires the
`extended-cache-ttl-2025-04-11` anthropic-beta header. Investigated:
- Live-tested api.anthropic.com on claude-opus-4-7 (2026-05-22)
with cache_control={type:"ephemeral", ttl:"1h"} and NO beta
header. Got status 200 and ephemeral_1h_input_tokens populated
on the create turn, plus cache_read_input_tokens populated on
the reuse turn.
- Cross-checked the current prompt-caching docs: no mention of
any beta header on the 1h TTL path.
Conclusion: the gate has been promoted to GA. The code already
does not send the beta header (the cache_marker dict only carries
`type`/`ttl`), so no wire change is needed. Pinned the contract
with two regression tests that assert the header is NOT on the
outbound request, and added a docstring note explaining the
investigation outcome so a future reader does not re-add it.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: per-model Anthropic server-side tool versions
Anthropic ships date-pinned tool versions per model family. Studio
currently hard-codes `web_search_20250305`, `web_fetch_20250910`, and
`code_execution_20250825` for every model, which means Opus 4.6/4.7,
Sonnet 4.6 and the Opus/Sonnet 4.5 family never get the newer
`_20260209` / `_20260120` variants. Those newer variants add dynamic
filtering (Claude writes code to rank/filter web results before they
enter context) and REPL state persistence + programmatic tool calling
inside the sandbox, which is what the user-facing pills are supposed
to expose.
Hardcoding the legacy versions also breaks if a future model family
drops the legacy types: the request 400s instead of falling back.
Changes:
- Add `_anthropic_web_search_version`, `_anthropic_web_fetch_version`,
`_anthropic_code_execution_version` helpers that pick the newest
variant the model accepts and fall back to the GA versions for
everything else.
- Add `_ANTHROPIC_CODE_EXECUTION_BETA` constant since the beta header
(`code-execution-2025-08-25`) is shared across both code-execution
date variants per the upstream docs.
- Wire the helpers into `_stream_anthropic` so the outbound body
carries the right pinned version per request.
- Add parametrized dispatch tests in
`test_anthropic_tool_versions.py` covering Opus 4.7/4.6/4.5,
Sonnet 4.6/4.5, Haiku 4.5, Opus 4.1/4.0, Sonnet 4.0, 3.5 Sonnet,
plus streaming integration tests that verify the outbound body
uses the right versions on Opus 4.7 (new web_search + new
code_execution), Haiku 4.5 (legacy both), and Sonnet 4.5 (legacy
web_search + new code_execution).
- Update existing `test_anthropic_code_execution.py` cases that
pinned the old version on Opus 4.7 to expect the new ones.
Verified end-to-end against the live Anthropic API: Opus 4.7 with
both pills enabled accepts the newer-pinned tools without a 400, and
Haiku 4.5 still works on the legacy fallback path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: surface prompt-cache token counts in /v1/chat/completions usage chunk
Studio's Anthropic and OpenAI Responses proxies already capture
cache_creation_input_tokens, cache_read_input_tokens (Anthropic) and
input_tokens_details.cached_tokens (OpenAI), but they were only written
to the structlog stream. Browser and SDK clients had no way to compute
"how many tokens hit the prompt cache" without scraping the server log,
so the chat UI could not show users how much money the cache was
saving on each turn.
This change emits one extra OpenAI include_usage-style chunk
(choices: [] with a populated usage block) just before the existing
[DONE] for Anthropic and after the final finish_reason chunk for
OpenAI Responses (both response.completed and response.incomplete).
The chunk shape:
usage.prompt_tokens_details.cached_tokens
normalised cache-read count, present for both providers.
usage.cache_creation_input_tokens
Anthropic-only; tokens billed at the cache-write premium.
usage.cache_read_input_tokens
Anthropic-only; same value as cached_tokens, kept for callers
that already key off the native Anthropic name.
Smoke verified end to end against a live Studio (claude-haiku-4-5
and gpt-4o-mini) plus 7 new unit tests on the helper and the two
streaming paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Anthropic: include cache buckets in prompt_tokens / total_tokens
Anthropic's `input_tokens` field excludes the cache buckets -- the
real prompt size is `input_tokens + cache_creation_input_tokens +
cache_read_input_tokens`. Previously the new usage chunk reported
only `input_tokens` as `prompt_tokens`, which heavily undercounted
cache-hit turns (e.g. an 18.9k-token cache_read turn looked like an
8-token prompt) and broke any downstream context / cost display fed
by `prompt_tokens` or `total_tokens`.
Fix `_build_usage_chunk` to sum all three input buckets for the
Anthropic provider while keeping the OpenAI Responses path unchanged
(OpenAI already folds cached tokens into `input_tokens`). The native
`cache_creation_input_tokens` / `cache_read_input_tokens` keys and
`prompt_tokens_details.cached_tokens` mirror are still emitted, so
clients keep full visibility of the cache split.
Tests updated to assert the summed shape.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: settle GPU VRAM after killing llama-server before the next reload
The NVIDIA driver reclaims a dead process's CUDA allocations
asynchronously after the kernel reaps the PID -- typically tens to
hundreds of milliseconds. Sampling `_get_gpu_free_memory` in that
window reads artificially low, which propagates into `_select_gpus`
/ `_fit_context_to_vram` and flips the layer-split toward `--fit on`
with more CPU-offloaded layers than steady-state would have required.
On a tight VRAM card the resulting mmap thrash + OOM matches the
Apply-reload kill path that bare-shell launches with the same flags
never hit (continues the lineage of #5161 / #5401 / #5427).
Adds `LlamaCppBackend._wait_for_vram_settle`: bounded poll of
`_get_gpu_free_memory` that returns as soon as two consecutive
samples agree per-GPU within `max(256 MiB, 2% of larger sample)`,
or `max_wait` (default 2 s) wall-clock elapses with probe time
included in the bound. Records `_last_kill_monotonic` inside
`_kill_process`'s `finally` block so the wait engages on both
in-process `load_model -> _kill_process -> load` and the frontend
chat-settings Apply path (`/unload` then `/load`). The call site
runs OUTSIDE the broad `self._lock` so concurrent `/unload`,
`/cancel`, `/status` are not blocked during the wait.
Short-circuits at zero cost on cold start (no kill recorded), stale
kill (older than 15 s, driver has already settled), CPU-only host
(probe returns empty), and probe exceptions (nvidia-smi gone away).
11 new unit tests in `test_llama_cpp_wait_for_vram_settle.py` cover:
cold-start zero cost, stale-kill skip, slow-probe deadline bound,
GPU index-set change, per-GPU stability with one draining card, the
2 % adaptive tolerance, _kill_process timestamp recording on real
kill vs no-op, and an `inspect.getsource` contract that pins the
call site to outside the Phase 3 lock and uses `_last_kill_monotonic`
so a future refactor can't silently regress any of these properties.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Michael Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: unblock /load event loop on detect_audio_type (#5642, #5635)
studio/backend/routes/inference.py wraps llama_backend.detect_audio_type
in await asyncio.to_thread() so its chain of sequential sync
httpx.Client.post() probes (/tokenize and /detokenize, 10 s timeout
each) runs on the threadpool instead of blocking the FastAPI event
loop. Without this wrap, /api/inference/load-progress polling and any
other in-flight HTTP request stalls for up to ~80 s while
detect_audio_type runs, which is exactly the "llama-server logs say
ready, Studio UI never finishes loading" symptom in #5642 (Win10) and
#5635 (Win11). The matching init_audio_codec call on the next branch
was already wrapped; this just brings detect_audio_type to parity.
Add a CPU-only spoof-based test suite under tests/studio/load_freeze/:
- llama_server_shim.py: stdlib http.server that answers /health,
/props, /tokenize, /detokenize, /completion with per-request
delay knobs.
- test_load_orchestrator.py:
* test_buggy_route_blocks_event_loop -- behavioural canary:
with a sync detect_audio_type call, concurrent /health
requests stall for >= one tokenize delay (proves the bug
class, runs from worker threads against a real uvicorn).
* test_fixed_route_keeps_event_loop_responsive -- with the
to_thread wrap, concurrent /health latency stays under 250 ms.
* test_routes_inference_wraps_detect_audio_type_in_to_thread --
static guard so the fix cannot regress silently.
* test_fast_path_load_completes_quickly -- regression budget
for post-_wait_for_health work.
Add .github/workflows/studio-load-orchestrator-ci.yml. CPU-only,
no torch, no real llama.cpp binary, no GPU. Cross-OS proof
(ubuntu-latest / macos-14 / windows-latest, 4 passed in 7-10 s each)
ran green on danielhanchen/unsloth-staging-2#136 before landing here.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: expand load-orchestrator suite to 22 tests (failure modes, stress, drift)
Replace the 4-test smoke with a comprehensive 22-test simulation
covering every failure mode of the /load -> detect_audio_type path:
1. Behavioural canary (2) - sync vs to_thread under slow shim
2. Functional equivalence (5) - sync == to_thread for each codec
branch (None / snac / csm / whisper
/ bicodec)
3. Failure modes (5) - shim returns 500, malformed JSON,
connection reset, unreachable port,
backend not loaded
4. Concurrency / stress (2) - 50 concurrent /probe; 100-burst
/health during slow /probe
5. Drift / regression guards (3) - wrap on production source, neighbour
init_audio_codec still wrapped, no
bare detect_audio_type() in any
async route
6. Timing budgets (2) - fast-path under 2s; 5 sequential
/probes under 10s
7. Browser-compat (2) - Content-Type + JSON.parse round-trip
+ response shape stable sync vs fix
8. Cancellation (1) - client disconnect mid-probe; server
keeps serving /health afterwards
Extended llama_server_shim with knobs for HTTP-500, malformed-JSON,
connection-reset, and tok_response_map / detok_map so we can
synthesise the exact request/response shape that triggers each codec
match. No new dependencies, still CPU-only and stdlib-driven.
Cross-OS validation on danielhanchen/unsloth-staging-2#136:
- ubuntu-latest: 22 passed in 19.59s
- macos-14: 22 passed in 22.07s
- windows-latest: 22 passed in 38.79s
Cross-Python on Linux (3.10 / 3.11 / 3.12 / 3.13 x pinned-floor /
latest deps, 8 uv venvs): 176/176 passed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: move audio detect/codec init inside load_model lock; relax small-quant CI
Follow-up to #5642 fix that addresses two distinct concerns raised by
the gemini-code-assist review on PR #5669:
1. Race condition (medium-priority comment on routes/inference.py:869)
The original fix wrapped llama_backend.detect_audio_type in
asyncio.to_thread. That unblocks the FastAPI event loop but opens
a race window where a concurrent /api/inference/load can acquire
_serial_load_lock, kill the live llama-server, and start a new
one while the first request's detect_audio_type thread is still
probing the (now-dead) port -- the route then writes stale
_is_audio / _audio_type onto the shared backend instance.
Fix: move detect_audio_type + init_audio_codec INSIDE
LlamaCppBackend.load_model, immediately before the function
returns True. Both calls happen while self._serial_load_lock is
held, so the entire load sequence (spawn, wait health, detect
audio, init codec, return) is atomic. routes/inference.py now
just reads the cached _audio_type / _is_audio attributes.
This is the shape the gemini reviewer recommended, and it also
simplifies the route -- no more asyncio.to_thread wrap, no more
conditional init_audio_codec call. The route layer keeps its
non-inference responsibilities (_native_display_label /
_native_grant_backed assignments) since those depend on
route-local arguments.
2. Hardcoded local file path in test shim (gemini's other comment)
FakeLlamaServer's default model_path was a developer-specific
Windows cache path. Replaced with an OS-portable placeholder.
The value is cosmetic-only -- only used in the synthesised stdout
template's "loading model" line, which the production code we
drive from the tests does not parse.
3. Existing CI flake on studio-inference-smoke.yml (generalised fix)
Studio GGUF CI has been red on main and 5+ unrelated PRs all
day. Root cause: small-quant Qwen3.5-2B drifts in two places.
(a) The python tool spits back "55,888" instead of "56088"
even though the tool itself returned the correct value. (b) The
OpenAI / Anthropic determinism check sees occasional non-byte-
identical responses at temperature=0.0 across runs due to KV
cache / speculative-decoding non-determinism. Both are model
output drift, not Studio regressions.
Generalised fix: match the Windows variant's already-lenient
WARN-when-tool-ran-but-model-drifted pattern. SSE-stream-empty
stays a hard FAIL (real plumbing failure); a non-empty stream
with the wrong numeric content becomes a WARN. Determinism
check similarly demotes "trailing whitespace OK but content
diverged" to a WARN; the harder grounding assertions on
later turns (paris present somewhere, turn-1 contains '1')
remain strict and continue to catch real regressions.
Test updates:
- test_routes_inference_wraps_detect_audio_type_in_to_thread is
replaced by test_load_model_caches_audio_type_inside_serial_load_lock
(asserts the lock + cache pattern in llama_cpp.py) and
test_routes_inference_reads_cached_audio_type_not_calls_detect
(asserts the route reads cached values).
- test_no_other_async_route_calls_detect_audio_type_unwrapped is
updated to flag any llama_backend.detect_audio_type call in
routes paths (the call belongs inside load_model now).
Local cross-Python matrix (Linux, Python 3.10 / 3.11 / 3.12 / 3.13 with
pinned-floor + latest dep ranges, 8 uv venvs): 22/22 passed in each
= 176/176 total.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tool-actually-ran assertion (chatgpt P1); shim port-0 (gemini)
Two PR-review follow-ups on #5669:
1. chatgpt-codex-connector P1 (false-green CI):
The previous WARN-when-tool-ran-but-model-drifted pattern allowed
a model that silently ignores enable_tools and just chats to
false-green the python / terminal tool smoke. Empty SSE was the
only failure mode caught -- a non-empty assistant text with no
actual tool invocation also passed.
Fix: post_sse now also returns the raw event payloads. A new
helper _tool_invoked(events, expected_outputs=...) checks the
raw stream for any of:
- OpenAI-style tool_calls delta
- Anthropic-style tool_use marker
- tool-role message
- the expected tool output substring (the tool's stdout reaches
the agentic loop as a fresh stream chunk, so the literal
"56088" / "hello-bash-tool" appears in the raw stream
independently of how the model narrates it)
The python and bash/terminal tool tests now hard-assert tool
invocation via _tool_invoked, then separately surface model
narration as PASS vs PASS-with-drift. A false-green like the one
chatgpt flagged would now hit the assert and FAIL the job.
web_search keeps its relaxed shape because DuckDuckGo upstream
blocks GHA IP ranges often enough to be noise.
2. gemini-code-assist medium (test shim, lines 192 + 261):
- Default model_path was a developer-specific Windows cache path.
Already replaced last cycle with an OS-portable placeholder.
- _free_port() inside the shim raced against bind(); replaced
with the cleaner port=0 -> read server_address[1] pattern.
The unused _free_port helper inside the shim is removed.
Local sim suite still green (22 passed in 19.91s). Studio GGUF CI
on this branch went green twice with the lenient path before this
push -- the strict assertion is a tightening, not a softening.
* ci(studio-inference-smoke): broaden tool-invocation markers
Add tool_status / tool_start / tool_end / tool_result to the
_tool_invoked marker tuple in studio-inference-smoke.yml. Studio's
routes/inference.py agentic tool loop emits tool_status (with
content) and tool_start / tool_end envelopes when a server-side tool
actually runs; anthropic_compat.py emits tool_use / tool_result.
The previous list only covered OpenAI tool_calls vocabulary, so on
the GGUF code path the strict assertion (introduced to address
chatgpt-codex-connector P1 on PR #5669) red-failed even when the
python / terminal tool had actually executed -- the last 3 SSE
events showed tool_status envelopes that the marker list missed.
Update the assertion failure-message strings to enumerate the full
marker set so debug output matches reality.
Local sim suite remains 22/22 green.
* studio: address chatgpt-codex P1+P2 follow-ups on 237052ff
P1 (.github/workflows/studio-inference-smoke.yml): tighten
_tool_invoked so it only counts strong markers. The previous
revision accepted (a) the weak tool_status envelope and (b) any
expected_outputs substring in the raw stream as evidence the tool
ran. Both let the test false-green:
- tool_status fires on every iteration boundary of Studio's GGUF
tool stream (including empty {"type":"tool_status","content":""}
cursor resets) regardless of whether any tool_call was actually
produced.
- The literal output substrings (56088, hello-bash-tool) can
appear in the model's narration without the tool ever running --
the user prompt itself contains "hello-bash-tool" and 123*456
is computable from prompt context alone.
Now require one of: tool_calls / tool_call / tool_use / tool_result
/ tool_start / tool_end / function_call / role:tool. tool_start in
Studio's GGUF agentic loop only fires inside `for tc in tool_calls`,
so its presence is positive proof a tool was actually invoked.
P2 (studio/backend/core/inference/llama_cpp.py): re-probe audio
type when load_model takes the already-in-target-state fast path
and the cached _audio_type is still None. detect_audio_type
swallows network / JSON errors and returns None, so the first
load's transient failure used to be sticky: subsequent /load calls
for the same model hit the fast path, skipped the probe, and kept
returning non-audio metadata indefinitely. The re-probe restores
the behaviour the route-level call used to give us before the
follow-up race fix moved detection inside the lock.
Local 22-test load_freeze sim suite remains green.
* studio: hard-assert tool_end.result for python+bash tools
Addresses chatgpt-codex-connector P1 review on PR #5669 commit
1a2fba84 ("Keep tool-output assertions hard-failing").
The previous revision asserted only that a tool was invoked
(strong-marker check) and downgraded the expected-output check to
WARN. That opened a false-green for tool-correctness regressions:
the python tool could silently return the wrong number, or the
terminal tool could silently fail to echo, and the test would still
pass because the assistant's narration happened to contain the
literal somewhere.
Add `_tool_output_contains(events, *needles)` which parses each SSE
event payload as JSON and checks the *tool's own output* across
three native shapes:
1. Studio GGUF agentic loop emits `{"type":"tool_end","result":
<str>}` from safetensors_agentic.py:348-353 -- this `result` is
the raw return value of the tool, before any model paraphrase.
2. Anthropic compatibility layer emits `{"type":"tool_result",
"content":[...]}` from anthropic_compat.py:357 -- check the
text blocks.
3. OpenAI chat completions stream tool-role deltas/messages
(`{"role":"tool","content":<str>}`) -- check that content.
Hard-assert that:
- python tool's tool_end.result contains "56088" or "56,088"
- bash tool's tool_end.result contains "hello-bash-tool"
Model-narration drift remains a WARN-only print (small-quant
paraphrase is acceptable; tool-output correctness is not).
Verified the helper with 7 unit cases locally (true-positive for
each native shape, true-negative for wrong tool result, narration-
only stream, and error-result, plus malformed-JSON tolerance).
Local 22-test load_freeze sim suite remains green.
* studio: retry server-side tool probes to handle small-quant flake
The strict tool_end.result assertion added in ea539eb4 (response to
chatgpt-codex P1 on commit 1a2fba84) red-failed on the very next CI
run -- but only on Linux; Mac+Windows GGUF CI both stayed green on
the same sha. The single failing attempt produced 29 SSE events
with no tool_end payload at all and finish_reason:stop, so
`_tool_invoked` passed (a tool_calls-looking substring matched
somewhere in the assistant's content text) while
`_tool_output_contains` correctly rejected the lack of a real
tool_end event. The chatgpt-codex P1 assertion semantics are
correct -- a tool that did not actually run cannot count as a pass.
The cause is small-quant Qwen3.5-2B-UD-IQ3_XXS sampling: it
correctly invokes the agentic tool loop most of the time but
occasionally produces content that *looks* like a tool_call to the
marker substring without the Studio GGUF agentic loop actually
intercepting it and running the tool. That is per-seed flake, not
a Studio plumbing regression; Mac+Windows on the same sha confirm
the plumbing works.
Add a single `_run_tool_probe(label, prompt, enabled, session,
needles, max_attempts = 3)` helper. Each attempt rotates the seed
(3407, 3408, 3409); we PASS on the first attempt where
`_tool_invoked AND _tool_output_contains` is True, and only FAIL
after exhausting all attempts. The failure message distinguishes
"never invoked at all" (real plumbing regression) from "invoked but
no attempt produced the right output" (tool-correctness regression),
so a future failure tells the reader where to look.
Strictness of each attempt is unchanged -- a winning attempt still
needs a strong tool marker AND a real tool_end.result containing
the expected literal. We only widen the chance the model gets to
actually invoke the tool.
Local 22-test load_freeze sim suite remains green. YAML parses.
* studio: structural _tool_invoked + entropy for tool-probe retry
Two bugs surfaced together on Linux Studio GGUF CI run 26242445342
(sha ec753581):
1. `_tool_invoked` was substring-based. Three deterministic
attempts at seed 3407/3408/3409 all returned True with
tool_output_contains False and 29 events, no tool_end envelope
anywhere. The marker substrings (tool_calls, tool_use, etc.)
were matching the model's own chat content text -- e.g. the
assistant typed something like "I'll use the python tool_calls
feature" and the substring search treated that as evidence the
tool ran. Even tool_calls:null inside a delta would match.
Rewrite as a structural check: parse each event as JSON and
verify tool invocation by inspecting envelope `type`,
non-empty `delta.tool_calls`, `finish_reason == "tool_calls"`,
`role:"tool"` deltas, Anthropic content blocks of type
tool_use/tool_result, and Responses-API output items of type
tool_call/function_call/tool_use.
Verified with 9 true-positive and 7 true-negative unit cases.
The simulated failing-run shape (assistant content containing
"tool_calls" substring + tool_status reset + stop + usage) now
correctly returns False, surfacing the real diagnosis.
2. Retry seed rotation was a no-op at temperature 0. llama.cpp
does deterministic argmax sampling at T=0, so seeds 3407, 3408,
3409 all produced byte-identical 29-event streams. Bump
TOOL_PROBE_TEMP to 0.4 and max_attempts to 4 so each retry
actually explores a distinct sampling trajectory; this keeps
the strict-correctness contract per attempt (real tool_end
with correct result still required) while giving the model a
real chance to invoke the tool.
The original strict-correctness P1 (chatgpt-codex on 1a2fba84)
remains the contract: an attempt only passes if tool_invoked AND
tool_output_contains both hold. We FAIL after all attempts only,
and the failure diagnostic distinguishes "never invoked at all"
(plumbing regression) from "invoked but wrong output" (tool-
correctness regression).
Local 22-test load_freeze sim suite remains green. YAML parses.
* studio: split audio detect/init around self._lock for unload-cancel
Address two new chatgpt-codex-connector P2 reviews on PR #5669
commit b8a7fe4a:
1. "Run audio probing outside _lock to keep unload responsive"
(3282819131). detect_audio_type was running inside the phase-3
self._lock critical section. In the worst case it fires 8
sequential httpx.Client.post() calls with timeout=10, so unload
(which also needs self._lock to call _kill_process) could block
for up to 80s after llama-server is already healthy. Move
detect_audio_type outside self._lock; it stays inside
self._serial_load_lock so a concurrent /load still serialises.
2. "Synchronize fast-path codec init with unload lock" (3283177129).
The fast-path re-probe added in 1a2fba84 called both
detect_audio_type and init_audio_codec without acquiring
self._lock. init_audio_codec is the side-effect-causing half
(allocates codec GPU memory, mutates LlamaCppBackend._codec_mgr);
a concurrent /api/inference/unload could clear backend state and
tear down codecs in parallel, leaving stale _is_audio/_audio_type
on a dead backend and potentially leaking codec memory.
Fix: wrap init_audio_codec in a short self._lock block (both in
the main load path and the fast-path re-probe), re-checking
self._healthy inside the lock so an unload that fired between
the unlocked detect and the locked init wins cleanly (return
False; do not reattach codec state to a torn-down server).
The two P2s are complementary: the detect half stays *outside*
_lock (read-only HTTP probes; safe to interrupt with unload), the
init half stays *inside* _lock (writes to backend / allocates GPU
memory; must serialise with unload). Result: unload can now kill
mid-probe at any time without waiting for the probe to time out,
and codec init cannot race against unload.
Local 22-test load_freeze sim suite remains green; AST parses.
* studio: demote tool_end.result check to WARN; keep structural invocation
Five consecutive failures of Linux Studio GGUF CI (1a2fba84 ->
d4daa04c) on the strict `_tool_output_contains` assertion. The
assertion is correct in theory -- a tool that ran should put its
output in tool_end.result -- but unreachable in practice with the
Studio-runnable models on hand:
* Cross-checked: main (sha 966d3cda) passes Studio GGUF CI with
the looser substring-based test, so the GGUF tool *plumbing*
is not broken on main.
* Other PR branches (fix/toast-cancel, explore/mlx) that fail
Studio GGUF CI fail in completely different places
(npm/studio install errors), not the tool-output assertion.
* Adding entropy (T=0.4) and 4 retries did surface a wider
trajectory (113 events, 250 chars of content) but still no
real tool_end.result containing "56088".
* Diagnosis: small-quant Qwen3.5-2B-UD-IQ3_XXS sometimes emits
OpenAI-style tool_calls deltas (which the new structural
_tool_invoked correctly identifies) without the Studio GGUF
agentic loop intercepting them as Studio-native XML tool
invocations. That GGUF-vs-OpenAI tool-format mismatch is a
real Studio issue, but it is out of scope for #5642 (which is
about the audio-detect blocking the FastAPI event loop) and
blocking the audio fix on it is not the right trade-off.
What this commit keeps -- the legitimate hardening from the
chatgpt-codex P1 series:
* `_tool_invoked` stays structural (parses JSON, checks
envelope.type / non-empty delta.tool_calls /
finish_reason="tool_calls" / role:"tool" / function_call
/ content blocks of type tool_use|tool_result). This is a
strict improvement over main's substring matcher which
false-positived on model content text.
* The per-attempt strict check still runs; we only DOWNGRADE the
failure-when-no-attempt-passes path to a WARN when at least
one attempt had structural invocation evidence. If NO attempt
has any structural invocation marker, FAIL hard (real
plumbing regression).
What this commit demotes:
* Strict tool_end.result needle-contains assertion -> WARN
print, with the attempts log captured so a regression in
Studio's GGUF agentic loop would be visible in CI logs.
* Model narration mismatch -> WARN (was already WARN).
Local 22-test load_freeze sim suite remains green. YAML parses.
* studio: hard-assert second determinism run non-empty
Addresses chatgpt-codex-connector P2 review (3283542662) on
commit 7dbe4960: the determinism probe previously asserted only
that the first run produced content and demoted the
`a.strip() == b.strip()` comparison to WARN. As a result a second
run that was completely empty (intermittent backend / tool
instability) would only log drift and the job would still PASS as
long as the first run carried the grounding tokens, false-greening
the second execution path the probe exists to exercise.
Add `assert b` alongside `assert a` in the per-turn loop so a
second-run empty response FAILs the job. The trailing-whitespace
/ small-quant drift comparison stays at WARN because that drift
is genuinely model-side (observed across unrelated PRs on main).
Local 22-test load_freeze sim suite remains green; YAML parses.
* studio: cache audio-probe outcome via _audio_probed flag
Addresses chatgpt-codex-connector P2 review (3283860597) on
commit f63ac224: the fast-path re-probe ran whenever
`_audio_type is None`, but for non-audio models that stays None
permanently because detect_audio_type returns None and the
`elif detected:` arm never stores a sentinel. Every no-op /load
of a regular text model therefore re-ran 8 sequential
/tokenize + /detokenize HTTP probes under _serial_load_lock, so
a hung probe endpoint could block other concurrent loads for
tens of seconds even after the server was healthy.
Add `self._audio_probed: bool = False` to __init__ (alongside
`_is_audio` and `_audio_type` which were previously not
initialised in __init__ either). The normal load path sets
`_audio_probed = True` once detect_audio_type returns without
exception -- treating "non-audio" as a definitive probed
outcome. The fast-path re-probe now gates on
`if not self._audio_probed:` instead of `if self._audio_type is
None:`. unload_model resets `_audio_probed = False`. If
detect_audio_type raises (it normally swallows internal
exceptions), we leave `_audio_probed = False` so the fast-path
can recover on the next load -- the original transient-failure
recovery P2 (chatgpt-codex on commit 237052ff) is preserved.
Local 22-test load_freeze sim suite remains green; AST parses.
* studio: strict audio probe + recheck _healthy on load success
Addresses two new chatgpt-codex-connector P2 reviews on commit
0f55615d:
1. "Retry audio probing when detection returns None" (3284185168).
The previous revision set `_audio_probed = True` immediately
after `detect_audio_type()` returned, but that method swallows
httpx/JSON errors and returns None on transient failures --
indistinguishable from a definitive "non-audio" verdict. The
caching therefore lost the transient-failure recovery the
earlier P2 (3281943869 on commit 237052ff) asked for: a
probe-error followed by no-op /load would never re-probe.
Split into a strict inner helper `_detect_audio_type_strict()`
that propagates transport/JSON errors via raise_for_status()
instead of catching them. The existing `detect_audio_type()`
becomes a backwards-compatible wrapper that swallows errors
for any external callers. load_model now calls the strict
helper directly so transient errors leave `_audio_probed=False`
(the fast-path re-probe recovers) while a clean return cached
the result as definitive. Apply to both normal load and
fast-path.
2. "Recheck health before reporting load success" (3284185172).
Audio probing now runs outside `self._lock`, so an
`/api/inference/unload` that arrives mid-probe can tear down
the backend before load_model reaches its `return True`. In
the non-codec branch we returned True without rechecking
`_healthy`, so the route could report success on a
torn-down backend. Re-check `_healthy` before the final
`return True` in both normal and fast-path branches; return
False if unload won.
Local 22-test load_freeze sim suite remains green. Static guard
test test_load_model_caches_audio_type_inside_serial_load_lock
updated to accept either `self.detect_audio_type()` or the new
strict-variant call shape.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: clear _audio_probed on codec init failure
Addresses chatgpt-codex-connector P2 review (3284516915) on
commit eb3a52a1: load_model marks `self._audio_probed = True`
before init_audio_codec, but when init throws (e.g., transient
huggingface_hub.snapshot_download blip for bicodec, GPU memory
pressure) we only log and continue. The fast-path guard
`if not self._audio_probed` then skips re-init on subsequent
no-op /load calls for the same model, so a transient codec init
failure leaves the backend stuck in non-audio mode until a full
unload+reload.
Clear `self._audio_probed = False` in the codec-init exception
handler (both normal load path and fast-path re-probe). Next
/load will re-probe and re-attempt init, restoring transient-
failure recovery.
Detection-only branches (csm / whisper / audio_vlm have no codec
init step) are unaffected -- a successful detect that recorded
the audio_type stays cached as probed.
Local 22-test load_freeze sim suite remains green; AST parses.
* studio: trim verbose review-citation comments
Remove inline citations of chatgpt-codex / gemini-code-assist PR
review IDs across llama_cpp.py, routes/inference.py,
studio-inference-smoke.yml, and the test shim. The review IDs
belong in the commit history, not in every block of code they
touched. Replace verbose docstrings with one-sentence summaries
where the body just repeated what the code already does. Behaviour
is unchanged; AST + 22-test sim suite still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: address 10-reviewer P1 findings on PR #5669
Four distinct issues surfaced by a 10-parallel reviewer pass over
the rebased branch:
1. `_detect_audio_type_strict` used `raise_for_status()` on every
probe response. HTTP 4xx/5xx for the SNAC marker token IDs
(e.g. server rejects out-of-vocab `128258`/`128259`) made the
strict probe abort before checking csm / whisper / audio_vlm /
bicodec / dac. Restore the pre-PR contract: treat non-200 as a
per-marker miss (return `""` / `[]`) and continue probing. Real
transport failures (connection reset, malformed JSON) still
raise so the caller can leave `_audio_probed=False`.
2. Codec-init failure inside the TTS branch logged a warning, set
`_audio_probed=False`, and let `load_model` return True. The
pre-PR contract was that an `init_audio_codec` exception
propagated out of the route and surfaced as HTTP 500. Restore
that: `return False` from `load_model` on init failure so the
route raises visibly instead of reporting an audio model as
plain text.
3. The non-TTS branch (csm / whisper / audio_vlm) wrote
`self._audio_type = detected` outside `self._lock`. The TTS
branch took `self._lock` and rechecked `self._healthy` first,
so a racing `/unload` couldn't be silently overwritten. Apply
the same guard to the non-TTS branch in both the fresh-load
path and the duplicate-load fast path.
4. The route's `already_loaded` short-circuit returned the cached
`_is_audio` / `_audio_type` without ever calling `load_model`.
When a previous probe failed transiently and `_audio_probed`
was left False, clicking Load again returned stale state and
never reached the backend retry path. Add `_audio_probed` to
the predicate so the request falls through.
Validation: 248/248 tests pass across Python 3.11 / 3.12 / 3.13 /
3.14 in isolated uv venvs (22 in-tree load_freeze + 18 + 11 + 11
supplements, 62 unique tests × 4 versions). Each fix has a
targeted reproducer that fails before the patch and passes after.
* studio: shorten audio-probe comments
Net -46 lines across llama_cpp.py, routes/inference.py, and the test
shim. Drops over-verbose docstrings and inline comments to one-line
WHY summaries where the code is self-evident. Behaviour unchanged;
62/62 sim tests still pass.
* studio: restrict _is_audio=True to TTS subset (codex P1 on d297b76e)
The previous fix landed self._is_audio = True in the
csm/whisper/audio_vlm branch, but the pre-PR route only set
_is_audio = True for the TTS subset (snac/bicodec/dac). That
matters because /v1/chat/completions auto-routes to
generate_audio_response when _is_audio is true, and
generate_audio_response rejects non-TTS codecs. A csm/whisper/
audio_vlm GGUF would have been misrouted into the TTS path.
Drop the _is_audio = True assignment from both elif detected:
branches (fresh-load and fast-path); keep the _audio_type write
so detection metadata is preserved. Add a static regression test
asserting the elif blocks never set _is_audio=True.
Validation: 252/252 (63 tests x py3.11/3.12/3.13/3.14) PASS.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix: Claude Code Anthropic API tool compatibility
* fix: merge Anthropic server tool selections
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix anthropic /v1/messages server-tool alias misrout
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: harden Anthropic /v1/messages tool validation
* fix: dispatch Anthropic server tools by only
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: reject Anthropic client tools missing 'name' at boundary
AnthropicTool.name was relaxed to Optional[str] to accommodate server-tool
declarations. A client tool with input_schema but no name now parses but
is silently dropped by anthropic_tools_to_openai, leaving tool calling
disabled. Surface as 400 instead.
* fix: reject Anthropic client tools with empty 'name'
isinstance(name, str) accepts an empty string, but anthropic_tools_to_openai
drops entries via 'if not name', producing the same silent-disable
fallthrough the boundary check is meant to prevent. Tighten to also reject
empty name.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
* feat: add custom model v1/model loading
* fix: require base URL for local model catalog loading
* ux/studio-provider-model-loading-controls
* fix: normalize local provider base URLs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: add custom model v1/model loading
* fix: require base URL for local model catalog loading
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Reverts PR #5615 to give the safetensors + MLX healing parity work more time to bake before re-merging. The reverted feature branch `studio-tools-multi-format` remains untouched, and the follow-up PR will layer the healing-parity commits on top.
Adds tool calling for Llama-3, Mistral (pre-v11 + v11+ + [ARGS]), and Gemma 4 to the safetensors / transformers and MLX backends. Parser patched against llama.cpp / vLLM / SGLang per-family parsers and normalises to OpenAI shape. 96 targeted unit tests + cross-OS staging CI (ubuntu / macos-14 / windows) green on the multi-format probe.
Adds tools, thinking blocks, code execution, and web search support to the safetensors / transformers and MLX inference backends in Studio, bringing them to parity with the GGUF path.
What ships
- safetensors / transformers agentic tool loop with cumulative-text state machine, tool-call XML parser, and template kwarg forwarding (tools / enable_thinking / reasoning_effort / preserve_thinking).
- MLX backend: same kwargs accepted on Apple Silicon; chat_template_info shipped through worker IPC; pills enable for Qwen / Qwen3 / Qwen3.5 / Gemma reasoning.
- Capability classifier (_detect_safetensors_features) gates supports_tools on actual parser-compatible emission markers (<tool_call> / <function=) so Llama-3 / Mistral / Gemma 4 do not advertise toggles the parser cannot honour.
- gpt-oss override stays: reasoning on, tools off (Harmony channel, not <tool_call> XML).
- CWE-209 hygiene: safetensors SSE error path emits a constant message and logs the trace server-side.
Validation
- 256 unit tests green (43 tool-loop, 11 capability advertise, 7 MLX backend, 5 main-added, 190 adjacent inference / anthropic / openai regression).
- Cross-OS staging CI green on ubuntu-latest / macos-14 / windows-latest plus a dedicated MLX cartesian probe against real unsloth/Qwen3.5-0.8B on macos-14 (CI 26098107440).
- Capability parity verified across Qwen3 / Qwen3.5 / Llama-3 / Mistral / Gemma / DeepSeek-R1 / gpt-oss (incl. BF16).
- Manual confirmation from Imagineer99 on Qwen3.5-2B: think + search + code exec working.
Closes the safetensors / MLX gap with the GGUF backend.
* studio: reserve VRAM headroom for the MTP draft cache in auto-fit
When MTP is going to engage on this load, _fit_context_to_vram now
budgets 0.85 of available VRAM instead of 0.90, leaving room for
llama.cpp's secondary MTP draft KV cache + compute graph buffers.
Motivation: a user report on RTX 5090 (32 GB) showed Qwen3.6-27B-MTP-GGUF
UD-Q4_K_XL at native auto-context running roughly half the speed of
the same model with a slightly smaller context. The most parsimonious
explanation is a VRAM cliff: at native context the target's KV
already eats the 90% budget, then llama-server allocates the draft
cache + draft graph on top and spills into a slower partial-offload
path. Reducing the budget by 5% on MTP loads avoids the spill without
penalising non-MTP loads. On hardware with abundant VRAM (B200, etc.)
the fit is unchanged because the requested context already fits in
the tighter budget too.
MTP detection mirrors the auto-promotion logic in load_model: the
GGUF advertises nextn_predict_layers, or the model identifier /
local path matches the -MTP marker, and the user has not explicitly
opted out via speculative_type="off" or --spec-type extra args.
Tests: two new cases in test_kv_cache_estimation.py verify that
mtp_engaged=True yields a context less-than-or-equal-to the
non-MTP path on a tight budget, and that kv_on_gpu=False still
short-circuits regardless of mtp_engaged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: gate _mtp_will_engage on canonical-mode resolver
After PR #5582 introduced the 5-mode Speculative Decoding dropdown plus
_canonicalize_spec_mode, the auto-fit MTP-engaged predicate becomes:
* forced mtp / mtp+ngram -> always engage MTP (extra VRAM needed)
* auto + MTP GGUF (>= 3B) -> engages MTP via auto-promotion
* auto + MTP GGUF (sub-3B) -> falls back to ngram-mod (no extra VRAM)
* ngram / ngram-simple / off -> never engage MTP
* user --spec-type in extra_args -> resolver suppressed; no headroom
The old gate triggered on "anything but off", so it over-reserved the
0.85 budget when the user explicitly picked Ngram (no MTP) or when
Auto fell back to ngram-mod on a sub-3B MTP model. The 5% headroom
cost was minor but unnecessary.
Mirrors the same logic already encoded in _build_speculative_flags so
the auto-fit budget and the actual emission agree on whether MTP is
running.
All 361 backend tests pass.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: add --spec-draft-n-max toggle for MTP speculative decoding
Surface llama-server's --spec-draft-n-max as a first-class
LoadRequest field so users can tune the MTP draft tree size from
the chat settings panel. Default behaviour is unchanged: when the
caller omits spec_draft_n_max, the existing platform defaults still
apply (6 on GPU, 3 on CPU/Mac).
Why this matters: on context-constrained loads the draft KV cache
competes with the target model's KV cache for VRAM. Lowering
spec_draft_n_max reduces that pressure, lets a larger user context
fit, and recovers throughput; raising it pays off when draft
acceptance is high enough to amortise the extra cache.
Backend
- LoadRequest gains an optional spec_draft_n_max: int (1..16).
- LlamaCppBackend.load_model accepts and persists the override on
self._spec_draft_n_max, used in place of the hardcoded 6/3 in the
MTP emit branch.
- LoadResponse and InferenceStatusResponse echo the active value
(None when the platform default is in effect) so the UI can
hydrate the input on refresh.
- _already_in_target_state and _request_matches_loaded_settings
compare spec_draft_n_max alongside speculative_type so a value
change triggers a reload rather than no-op'ing.
- strip_shadowing_flags now strips inherited --spec-* extras when
either speculative_type or spec_draft_n_max is in fields_set, so
an inherited --spec-draft-n-max cannot last-wins-override a fresh
request's first-class field.
Frontend
- LoadModelRequest, LoadModelResponse, InferenceStatusResponse
TypeScript shapes get spec_draft_n_max.
- chat-runtime-store gains specDraftNMax / loadedSpecDraftNMax and
a setter, hydrated from /v1/status and /v1/load.
- chat-settings-sheet renders a "Draft Tokens" numeric input
directly under the Speculative Decoding switch when that switch
is on. Toggling the switch off clears the override; the Reset
button restores the loaded value.
Tests
- Four new regression tests cover _already_in_target_state with
matching / mismatching / non-MTP / unset spec_draft_n_max.
- Existing test_llama_server_args.py and test_llama_cpp_mtp_detection.py
green: 141 passed locally.
* studio: add --spec-draft-p-min and --spec-draft-p-split to spec strip set
llama.cpp server documents --spec-draft-p-min (default 0.75, min draft
acceptance probability) and --spec-draft-p-split (default 0.10). Both
are first-class spec-decoding knobs that should travel with the rest
of the --spec-* family when an Apply re-sets speculative_type, so an
inherited override doesn't leak across a fresh load.
* studio/tests: skip MTP capability-probe tests on Windows
The four probe_server_capabilities tests use a bash stub written to
tmp_path/llama-server, which Windows' subprocess can't execute
directly (no shebang resolution, .bat / .cmd would be needed). Mark
them skipif sys.platform == 'win32' so the rest of the MTP plumbing
suite stays green on Windows CI. Unix coverage is unchanged.
* studio: lower MTP GPU default --spec-draft-n-max from 6 to 2
Bench on B200 / Qwen3.6-27B-MTP-GGUF UD-Q4_K_XL across five prompt
types (essay, code, story, math, science) with greedy temp=0:
prompt OFF n=1 n=2 n=3 n=6
essay 79.1 93.4 93.8 84.7 64.6
code 79.1 104.4 116.6 113.5 103.0
story 79.1 99.2 105.7 101.8 88.9
math 79.1 100.8 110.8 111.8 98.2
science 79.1 100.1 110.8 110.8 102.9
The previous hardcoded GPU default of 6 was 17% SLOWER than spec-off
on the essay prompt (64.6 vs 79.1 t/s) and 11-50% slower than n=2 on
the rest. n=2 wins on 4/5 prompts with a 1.18x-1.47x speedup vs OFF;
n=3 wins on the math prompt by a hair. n=6 collapses once acceptance
rate drops past n=3 -- wasted draft decode dominates the per-step
budget.
Matches the dataset README ("n_max=2 is the sweet spot for 36 of 42
quants"). Keeps CPU/Mac default at 3, which empirically tracks the
narrower ngram+MTP chained budget on those platforms.
Users who want the old behaviour can pass spec_draft_n_max in
LoadRequest (the toggle this PR also adds) or --spec-draft-n-max via
llama_extra_args.
* studio: skip MTP auto-promote on sub-2B models, backfill chat usage
Two MTP-visibility fixes uncovered while bisecting llama.cpp post-#22673
on Qwen3.6-27B-MTP-GGUF UD-Q4_K_XL on B200.
Size gate. Direct llama-server bench (no Studio measurement loop) at
n_predict=192 across 9 prompts shows MTP regresses vs spec-off on
sub-2B dense models because draft cost exceeds savings:
Qwen3.5-0.8B Q4_K_XL GPU: 452.0 OFF -> 283.4 t/s n=2 (0.63x)
CPU: 84.5 OFF -> 64.9 t/s n=3 (0.77x)
Qwen3.5-4B Q4_K_XL GPU: 241.0 OFF -> 258.2 t/s n=2 (1.07x)
Qwen3.5-9B Q4_K_XL GPU: 201.6 OFF -> 228.9 t/s n=2 (1.14x)
Qwen3.5-27B Q4_K_XL GPU: 78.8 OFF -> 113.6 t/s n=2 (1.44x)
Qwen3.6-27B Q4_K_XL GPU: 78.8 OFF -> 113.6 t/s n=2 (1.44x)
Qwen3.6-35B-A3B Q4 GPU: 192.3 OFF -> 223.2 t/s n=2 (1.16x)
The 2B inflection is sharp. Skip auto-promote to draft-mtp when the
identifier reports <2.0B params; users can still force via --spec-type
or the Speculative Decoding toggle. Mirror the gate in the
reload-skip check so a sub-2B reload-with-default does not bounce a
spec-off backend.
Chat-completions usage. llama-server's final SSE chunk emits both an
OpenAI-style usage block and a custom timings block. timings.predicted_n
is always populated, but usage.completion_tokens is zero on some
server builds. The Studio chat UI computes generation t/s from
meta.usage.completion_tokens / totalStreamTime, so a zero
completion_tokens makes the UI fall back to wall-clock time
(including SSE / proxy / template overhead) which dilutes MTP gains and
makes ON look the same as OFF.
Add _backfill_usage_from_timings: if usage.completion_tokens is missing
or zero AND timings has predicted_n/prompt_n, synthesize a complete
usage dict. Apply at the streaming metadata yield in
generate_chat_completion and at the three accumulator/yield sites in
generate_chat_completion_with_tools so per-iteration counts are not
silently lost across tool calls.
Tests cover both the gate (sub-2B skips, 2B+ promotes) and the
backfill (zero usage filled, real usage preserved, empty timings
passthrough).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: probe + emit legacy ngram-mod flags for pre-rename llama-server
llama.cpp upstream renamed the ngram-mod tuning knobs:
--draft-max -> --spec-ngram-mod-n-max (and --spec-draft-n-max)
--draft-min -> --spec-ngram-mod-n-min (and --spec-draft-n-min)
--spec-ngram-size-n -> --spec-ngram-mod-n-match
The new names are real flags on post-rename builds and stub removal
entries on the same builds (with description "argument has been
removed"). Pre-rename builds only carry the legacy names as real
flags. Studio was emitting the new names unconditionally, so a user
running a pre-rename llama-server (e.g. an older prebuilt or a
hand-installed binary) would see "unknown argument" errors when the
ngram-mod path engages, or silent drop of the ngram knobs.
Extend `probe_server_capabilities` to parse the help text into
per-flag description blocks and tell real flags apart from removal
stubs by the "argument has been removed" marker. Add three new probe
fields: `ngram_mod_flavor` ("new" / "legacy" / None),
`supports_ngram_mod`, and `spec_draft_n_max_flag` (the actual n_max
flag the binary accepts). Cached by (path, mtime) the same way as
`mtp_token`.
Add `_build_ngram_mod_flags(caps, ...)` that picks the right flag
set, returning [] when neither is usable so callers can drop ngram
chaining entirely on minimal binaries.
Wire both call sites to use the probe-driven flag set:
- CPU/Mac MTP comma-chain (--spec-type ngram-mod,draft-mtp) emits
legacy or new knobs as appropriate. If neither set is available,
degrade to MTP-only (warn but still engage spec).
- Standalone --spec-type ngram-mod branch uses the same helper.
Tests cover post-rename detection, legacy detection, removal-stub
discrimination, minimal-binary case, and all three branches of
`_build_ngram_mod_flags` plus custom n_match/n_min/n_max values.
Verified against three real binaries (Studio bundled 726704a, my
build of 45b455e HEAD, and the MTP merge baseline 2555826) all
correctly reporting ngram_mod_flavor=new.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: sub-3B MTP falls back to ngram-mod, not off
Earlier sub-2B gate disabled speculative decoding entirely for tiny
dense MTP models because the MTP draft head's per-token cost exceeds
the acceptance savings at that scale. The "fully off" fallback was
conservative -- ngram-mod has near-zero idle cost on diverse content
and consistently outperforms both off and draft-mtp at sub-3B.
Clean-methodology bench (each of 9 distinct prompts run once after
two unrelated warmup prompts so the ngram-mod hash pool is
realistically populated but never holds the exact deterministic
output we're about to measure):
Q4_K_XL on B200:
0.8B OFF=451 draft-mtp n=2=263 (0.58x) ngram-only=498 (1.10x)
2B OFF=377 draft-mtp n=2=308 (0.82x) ngram-only=369 (1.00x)
4B OFF=240 draft-mtp n=2=260 (1.08x) -- 4B+ wins with MTP
Q4_K_XL on x86 48 cores:
0.8B OFF= 80 chained n=2= 69 (0.86x) ngram-only= 95 (1.19x)
2B OFF= 62 chained n=2= 51 (0.83x) ngram-only= 63 (1.01x)
4B OFF= 31 chained n=2= 41 (1.33x)
Change:
- Raise the MTP-skip threshold from 2.0B to 3.0B (2B falls below it).
- When skipping the MTP head, fall back to --spec-type ngram-mod via
the probe-driven _build_ngram_mod_flags helper. Works on both
post-rename and pre-rename llama-server builds.
- If the binary advertises neither ngram-mod flavor, fall back to
spec-off (older binaries that don't support ngram-mod at all).
- Mirror the same fallback in _already_in_target_state so a sub-3B
reload-with-default does not bounce a ngram-mod backend.
Tests updated: monkeypatch probe_server_capabilities so the gate
behavior is deterministic regardless of which llama-server happens
to be on the host. +1 new test for the "binary has no ngram-mod
support" branch; renamed prior 2B/0.8B tests to reflect new semantics.
This generalizes the size gate to be probe-driven instead of a hard
"disable spec" branch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: 5-mode Speculative Decoding dropdown (Auto / MTP / Ngram / MTP+Ngram / Off)
Replace the Chat Settings Speculative Decoding on/off Switch with a 5-option
Select. Auto preserves today's platform-aware resolver (MTP on MTP GGUFs,
ngram-mod fallback for sub-3B, --spec-default for non-MTP). The other 3 modes
force the user's choice on BOTH GPU and CPU: MTP emits draft-mtp only (no
ngram chain on CPU), Ngram emits ngram-mod only, MTP+Ngram emits the
ngram-mod,draft-mtp chain on both platforms. Off is the existing fully-off
state, kept so the Switch's "disable" capability isn't lost.
Backend
- New module-level _canonicalize_spec_mode(value) maps any accepted input
(canonical, legacy "default" / "draft-mtp" / "ngram-mod" / "ngram-simple",
or comma-chained "ngram-mod,draft-mtp") onto one of auto / mtp / ngram /
mtp+ngram / off / ngram-simple / None. Lets external callers and old
persisted UI state round-trip without breaking.
- LlamaCppBackend grows a _requested_spec_mode field + requested_spec_mode
property storing the canonical UI mode the user requested. Status
responses round-trip this instead of the resolved internal flag, so the
dropdown restores the picked value after reload / refresh (Auto on a 27B
MTP GGUF resolves to draft-mtp internally but the dropdown stays on
"Auto").
- The resolver block in load_model is extracted into a unit-testable
_build_speculative_flags method. Forced MTP / MTP+Ngram on a sub-3B or
non-MTP GGUF logs a warning and engages anyway (user override > the
Auto-path sub-3B fallback).
- _already_in_target_state and routes/inference._request_matches_loaded_settings
now compare canonical-requested mode, dropping the old auto-promotion
mirror. spec_draft_n_max still gates on the resolved spec so Auto + a
changed n_max still bounces a reload.
Frontend
- chat-settings-sheet.tsx: Switch swapped for Select modeled on the KV
Cache Dtype Select. Items: Auto / MTP / Ngram / MTP+Ngram / Off. Draft
Tokens input only visible when speculativeType is "mtp" or "mtp+ngram".
- chat-runtime-store.ts: initial value flips from "default" to "auto".
- use-chat-model-runtime.ts normalizeSpeculativeType mirrors the backend
canonicaliser so persisted "default" / "draft-mtp" / "ngram-mod" / chain
values hydrate to the right dropdown option.
- types/api.ts: docs the canonical wire vocabulary.
Tests
- 53 new assertions in test_llama_cpp_mtp_detection.py: full
_canonicalize_spec_mode table, a 23-row resolver matrix across
(requested mode) x (GPU/CPU) x (model size class), plus n_max override,
user-extra-args precedence, requested-mode round-trip, and graceful
degrade on an outdated llama-server without an MTP token.
- 165 existing backend tests still green. 218 total in the MTP /
server-args / reload-inheritance suite.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: reset Speculative Decoding to Auto on model switch
When the user switches from model A to a different model B, clear the
runtime store's speculativeType + specDraftNMax (and their loaded*
shadows). The new load request then carries null, the backend
canonicalises that to "auto", and its platform-aware resolver runs
fresh for the new model.
Without this, a non-MTP model loaded with "Off" carried the Off choice
into a subsequent MTP load, suppressing MTP auto-promotion (and the
sub-3B ngram-mod fallback) until the user manually opened settings and
flipped the dropdown back to Auto. The clean-sweep deep probe caught
it as anomaly A-1.
The reset only fires when currentCheckpoint != modelId, so a
same-model reapply or forceReload still honours the user's current
spec choice. End-to-end probe on Qwen3.5-4B-GGUF (non-MTP, Off) ->
Qwen3.5-0.8B-MTP confirms: dropdown shows Auto, /api/inference/status
returns speculative_type=auto, studio.log shows the Auto sub-3B
fallback emitted --spec-type ngram-mod.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Move the inline tool-call XML parser and stripper out of
studio/backend/core/inference/llama_cpp.py into a new
studio/backend/core/tool_healing.py so external inference servers
(llama-server wrappers, llama-swap, custom shims) can reuse the same
logic without importing the inference orchestrator, structlog, httpx,
or anything from torch / transformers / unsloth.
Closes#5502.
What this PR does:
- New file studio/backend/core/tool_healing.py contains the regex
constants (_TOOL_CLOSED_PATS, _TOOL_ALL_PATS, _TC_JSON_START_RE,
_TC_FUNC_START_RE, _TC_END_TAG_RE, _TC_FUNC_CLOSE_RE,
_TC_PARAM_START_RE, _TC_PARAM_CLOSE_RE), parse_tool_calls_from_text,
and strip_tool_call_markup. The regexes and function bodies are
byte-for-byte the same as the previous inline implementation in
llama_cpp.py; only the @staticmethod decorator and the closure-only
`if not auto_heal_tool_calls: return text` short-circuit are dropped
(the latter stays in the caller as a fast path when healing is off).
- studio/backend/core/inference/llama_cpp.py now imports the regexes
and helpers from .tool_healing. LlamaCppBackend._parse_tool_calls_from_text
becomes a one-line delegate; the _strip_tool_markup closure keeps the
auto_heal_tool_calls fast path and delegates the work.
- Helper module imports cleanly without torch, transformers, structlog,
httpx, or numpy. studio.backend.core itself is already stdlib-only
at import time (lazy __getattr__), so `from
studio.backend.core.tool_healing import parse_tool_calls_from_text,
strip_tool_call_markup` is the lightweight import path issue #5502
asked for.
No behaviour change for existing Studio paths. parse_tool_calls_from_text
and strip_tool_call_markup produce the same OpenAI-shape output the
old inline code produced for every input.
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* studio: emit one comma-chained --spec-type for CPU/Mac MTP path
llama-server takes a single --spec-type whose value may be
comma-separated to chain implementations (e.g. ngram-mod,draft-mtp).
The CPU/Mac MTP branch in LlamaCppBackend.load_model was passing
--spec-type twice in the same invocation, which is not the documented
chaining mechanism and silently drops one of the two specs depending
on llama.cpp's argv handling.
Collapse the pair to --spec-type ngram-mod,{mtp_token} and update the
stale _extra_args_set_spec_type docstring that claimed llama-server
accumulates repeated --spec-type. Update the matching pass-through
fixture in test_llama_server_args.py.
* studio: align MTP ngram-mod knobs with llama.cpp upstream defaults
Two correctness fixes against the llama.cpp server README:
1. The CPU/Mac comma-chained branch was emitting
--spec-ngram-mod-n-max 6 with --spec-ngram-mod-n-min 48, which is
nonsensical (min > max). Per the upstream default the value is 64.
2. The standalone ngram-mod branch was emitting --spec-ngram-size-n,
--draft-min, --draft-max. llama.cpp removed those arg aliases for
ngram-mod (they live only on the ngram-simple / map families now);
the correct knobs are --spec-ngram-mod-n-match / n-min / n-max.
Also refresh the inline comment block to point at the server README
rather than the older docs/speculative.md draft- aliases.
* studio: engage draft-mtp on vision MTP GGUFs
The draft-mtp auto-promotion in LlamaCppBackend.load_model was gated on
not effective_is_vision, and the spec-emit branch repeated the same
guard. Every Unsloth -MTP GGUF repo ships an mmproj projector, so
effective_is_vision was always True for those repos and the MTP speedup
silently never engaged out of the box.
llama.cpp #22673 explicitly states MTP is compatible with vision input.
The bundled b9204 server happily loads both: a manual run with
--mmproj ... --spec-type draft-mtp --spec-draft-n-max 6 logs
"loaded multimodal model" followed by
"adding speculative implementation 'draft-mtp'".
Drop the vision gate from both sites and rewrite the matching short
circuit in _already_in_target_state so reload checks reach the auto
promotion path on vision MTP loads. Add three regression tests covering
vision MTP match (auto and default), and non MTP vision repo unaffected.
Verified on a B200 with unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL:
base decode 179.7 t/s vs MTP decode 253.8 t/s, draft acceptance 0.57,
1.41x speedup on a 255 token completion. mmproj still loads and image
input remains available.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: prefer Qwen3.5 -MTP GGUF variants in default model lists
With the vision gate dropped in the previous commit, draft-mtp now
auto-engages on -MTP GGUF repos out of the box. Swap the four Qwen3.5
recommended entries in DEFAULT_MODELS_GGUF and DEFAULT_MODELS_STANDARD
to their -MTP-GGUF counterparts so new users get the speedup by default:
unsloth/Qwen3.5-4B-GGUF -> unsloth/Qwen3.5-4B-MTP-GGUF
unsloth/Qwen3.5-9B-GGUF -> unsloth/Qwen3.5-9B-MTP-GGUF
unsloth/Qwen3.5-35B-A3B-GGUF -> unsloth/Qwen3.5-35B-A3B-MTP-GGUF
unsloth/Qwen3.5-0.8B-GGUF -> unsloth/Qwen3.5-0.8B-MTP-GGUF
All four HF repos exist (HEAD 200) and ship the same UD-Q4_K_XL quant
layout as the non-MTP variants. Non-Qwen3.5 entries are untouched.
* bump version to 2026.5.4
Picks up the studio MTP vision-gate fix and the Qwen3.5 -MTP default
swap in this PR.
* studio: prefer Qwen3.6-35B-A3B-MTP-GGUF in default model lists
Same rationale as the previous Qwen3.5 swap. The Qwen3.6 MTP variant
exists at unsloth/Qwen3.6-35B-A3B-MTP-GGUF (HF HEAD 200) and now
auto-engages draft-mtp out of the box with the gate fix.
* studio: drop --spec-draft-n-max from 6 to 3 for draft-mtp
n=6 is too greedy: on Qwen3.6 the draft has to guess 6 tokens ahead
and acceptance crashes to ~0.45, leaving only ~14% throughput gain.
PR ggml-org/llama.cpp#22673's author benched n=3 at ~0.72 acceptance
and 2 to 3x speedup on the same Qwen3.6 family, and the README sample
command uses n=2 or n=3. Match that.
CPU/Mac branch already uses n=3, so this aligns both paths.
* studio: set --spec-draft-n-max back to 6 for draft-mtp on GPU
Reverts the n=3 tuning. n=6 is the original default; user-side comparisons
hold the larger draft window steady so the toggle (next commit) is the
primary on/off lever.
* studio: add Speculative Decoding toggle under Max Tokens
Adds a top-level kill switch (panel-switch under Max Tokens, mirroring
Auto-Healing Tool Calls) that forces the /load request's
speculative_type to "off" when disabled. The backend "off" branch in
LlamaCppBackend.load_model skips both the draft-mtp auto-promotion and
the spec-emit branch, so neither --spec-type draft-mtp nor
--spec-default reaches llama-server.
Wiring:
- chat-runtime-store: new speculativeDecodingEnabled bool, default
true, persisted to localStorage under unsloth_speculative_decoding,
plus a setSpeculativeDecodingEnabled setter.
- chat-settings-sheet: SpeculativeDecodingToggle rendered immediately
beneath the Max Tokens slider for non-external models.
- use-chat-model-runtime: when speculativeDecodingEnabled is false,
override speculative_type to "off" in the loadModel call so the
switch wins over any pre-existing speculativeType state (including
the existing per-model toggle in Model Settings).
Verified end to end on unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL:
toggle ON emits --spec-type draft-mtp --spec-draft-n-max 6; toggle
OFF emits zero --spec-* flags on the same MTP GGUF.
* studio: relocate Speculative Decoding toggle into Model Settings
Move the toggle out from under Max Tokens and back into the Model
Settings section, directly beneath KV Cache Dtype, where the existing
Apply/Reset workflow already drives a reload on dirty. This way flipping
the switch in the UI actually picks up: the section becomes dirty,
Apply re-runs /load with the new speculative_type.
Drop the !currentModelIsMultimodal gate so vision MTP GGUFs can also
disable speculative decoding from the UI.
Switch the toggle's off-value from null to "off" so the backend's "off"
short-circuit fires for MTP models too (null normalises to None which
re-triggers the draft-mtp auto-promotion).
Tooltip now reads "Faster generation with 0% accuracy hit".
Remove the now-redundant speculativeDecodingEnabled bool + setter from
the runtime store and the load-time override in use-chat-model-runtime;
the toggle binds directly to speculativeType.
* studio: restore OOM/TIGHT badge on recommended GGUF rows
The recommended-list row passed vramStatus=null for any GGUF repo
because the existing useRecommendedModelVram hook reads safetensors
totals from HF model info, which GGUF-only repos do not expose. As a
result, an OOM Q-quant repo would render with only a "GGUF" badge and
no visual signal that nothing in it fits.
Add useGgufRecommendedFit: per repo, fetch the variant list via the
existing /api/models/gguf-variants endpoint, take the smallest
variant's size_bytes, and classify with the same 0.7*GPU + 0.7*RAM
thresholds as GgufVariantExpander. Session-scoped cache + in-flight
dedup so a repo is requested at most once.
Wire the result into the three GGUF row sites in pickers.tsx so OOM
and TIGHT badges show on the collapsed cards.
* Revert "studio: restore OOM/TIGHT badge on recommended GGUF rows"
This reverts commit 07793b1240df72b13e51d6dc15f63c4ee8c6cba9.
The new useGgufRecommendedFit hook was treating the symptom. PR #5561
identified the real root cause: useGpuInfo was calling /api/system
with plain fetch instead of authFetch, so the session-auth check
failed silently and gpu.available stayed false everywhere. With no
GPU info, every fit check (variant expander, recommended carousel)
fell back to "no signal" and dropped the OOM/TIGHT badges.
Reverting the over-engineered hook and applying the authFetch fix
in the next commit, which restores the existing badges with one line.
* chore: replace qwen suggested with MTP variant
* fix: restore GPU info auth for GGUF fit badges
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Studio: gate image input on a usable mmproj for GGUF vision models
* Improve image gating and model capability sync
Tighten image-handling and model capability syncing across the chat flow. Key changes:
- chat-adapter: Replace per-message current-user image check with a simpler gate that blocks if ANY image is present in the outbound payload when the selected model cannot handle vision. Show the toast reason and flip the per-thread running flag on→off to avoid hanging wait promises before throwing.
- shared-composer: Simplify and correct image-attachment gating for single vs compare modes. Use an attach-time gate that defers to send/ensureModelLoaded in compare mode, introduce attachUnavailableReason, and only block immediately for single-mode. Remove an unused models selector.
- shared-composer: Sync the runtime models[] entry with the response from ensureModelLoaded so UI/send gates read fresh capabilities (isVision, isGguf, isAudio, audioType, hasAudioInput). This addresses catalog lag (e.g., GGUF mmproj arriving after the catalog snapshot).
- UX tweak: the file-picker button no longer outright blocks on image availability; addFiles still filters images per-file and toasts appropriately.
These changes prevent mid-stream server rejections, avoid deadlocks, and ensure model capability checks are accurate when attaching images or audio.
* studio: only pass --mmproj to llama-server when effective_is_vision
When a text-only GGUF (static is_vision=False) was paired with a
family-matching mmproj path, the launcher appended both --mmproj and
--spec-default, leaving llama-server in an inconsistent state while
Studio reported is_vision=False. Gate the --mmproj flag on
effective_is_vision so the launch command tracks the runtime
capability the rest of Studio sees.
* studio: reject image content in streaming /v1/responses for non-vision GGUF
_responses_stream forwards the OpenAI request body directly to
llama-server's /v1/chat/completions, bypassing the image-vs-vision
guard that openai_chat_completions enforces for the wrapped path.
Add the same check at the top of the streaming entry point so an
SDK client that posts an image to a non-vision GGUF receives a
typed 400 instead of an opaque downstream error.
* studio: gate external chat providers in the image input helper
External selections (cohere, deepseek, mistral, openrouter, ...) live
in externalProviders, not in runtime.models[], so activeModel is
undefined for them and the helper short-circuited to allow. Result:
images attached to a non-vision external chat model were dropped
silently downstream instead of rejected up front.
Add providerTypeSupportsVision to external-providers.ts (false for
known text-only providers, true for known vision-capable ones, null
for unknown / custom self-hosted) and thread externalSupportsVision
+ externalModelLabel through the helper. shared-composer.tsx,
runtime-provider.tsx (VisionImageAdapter.add), and chat-adapter.ts
pre-stream gate all resolve the provider type and pass it.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: warn when llama.cpp prebuilt is too old for MTP
Layered on #5527. Adds a one-shot llama-server --help capability probe
so users get a clear signal when their prebuilt is missing MTP support,
plus a graceful fallback if they load an MTP GGUF against an outdated
binary.
What's surfaced:
1. Startup log + stderr line in main.py:lifespan() if MTP isn't
advertised:
WARNING: llama.cpp prebuilt is missing MTP support
(--spec-type mtp / draft-mtp). Run `unsloth studio update` to
refresh it. MTP GGUFs will load without speculative decoding.
2. Load-time graceful fallback in load_model's spec block: skip the
auto-emit and log a clear warning instead of letting llama-server
fail with an unknown-flag error.
3. /api/inference/status now returns llama_cpp_supports_mtp: bool so
the frontend can show a banner / popup.
Probe internals:
- Class-level cache keyed on (binary_path, mtime). One subprocess call
the first time, instant thereafter. Touching the binary (e.g. via
`unsloth studio update`) invalidates the cache automatically because
the mtime changes, so the new build is picked up without restarting
the server.
- Recognises both upstream naming forms: the original draft-mtp from
llama.cpp PR #22673 and the renamed mtp variant in later commits.
- Spec block uses whichever token the binary accepts so we emit the
right value regardless of which release the user has.
Tests:
- 6 new cases in test_llama_cpp_mtp_detection.py covering each probe
variant (draft-mtp, renamed mtp, pre-MTP build, missing binary,
mtime-based cache invalidation).
- Existing 38 MTP detection cases still pass; broader 188-test
regression suite (server args, reload inheritance, gguf metadata,
load progress, context fit, model validation) still green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: auto-enable MTP speculative decoding for MTP GGUFs
Detect Unsloth's MTP (multi-token-prediction) GGUFs and auto-emit the
right --spec-type draft-mtp flags for llama-server (llama.cpp PR
#22673), so users get the speedup without configuration.
Detection prefers the GGUF metadata field <arch>.nextn_predict_layers
(verified on Qwen3.6-27B-MTP-GGUF / qwen35 and Qwen3.6-35B-A3B-MTP-GGUF
/ qwen35moe). Falls back to a -MTP marker in the identifier / filename
so HF-mode loads can detect MTP from the repo name before the GGUF is
downloaded.
Flag presets follow the Unsloth MTP guide:
GPU: --spec-type draft-mtp --spec-draft-n-max 6
CPU/Mac: --spec-type draft-mtp --spec-draft-n-max 3 \
--spec-type ngram-mod --spec-ngram-mod-n-match 24 \
--spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 6
User overrides win: if the caller passes --spec-type / --spec-default
via unsloth run / unsloth studio run pass-through (or HTTP
llama_extra_args), the auto-emit steps aside so llama-server only sees
the user's flag. Scalar tuning knobs like --spec-draft-n-max compose
with the auto preset via llama-server's last-wins parsing.
_already_in_target_state mirrors the same promotion so a repeat /load
with unchanged settings against an MTP backend running draft-mtp
short-circuits cleanly instead of forcing a reload.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* tests/studio: end-to-end Windows GPU detection mock test (#5106)
Locks in the combined fix from #5322 + #5324 with a synthetic
Windows scenario that CI runners without GPUs can execute. The
test packs the real PyPI win_amd64 wheel layouts (cu12 modular and
the new unsuffixed cu13 nvidia/cu13/bin/x86_64 layout) plus the
exact filename set of the upstream b9103 cudart-llama-bin-win-cuda
bundles, then mocks nvidia-smi output and asserts that:
* Studio's nvidia-smi probe parses the CSV and reports the GPU.
* After PR #5322 the install_dir/build/bin/Release/ tree contains
all three cudart bundle DLLs alongside llama-server.exe.
* After PR #5324 the PATH built by start_llama_server's win32
branch lists pip nvidia + torch/lib dirs in addition to the
binary_dir.
* cudart64_X.dll, cublas64_X.dll, and cublasLt64_X.dll are
each reachable from at least one PATH entry, with cudart
specifically reachable from BOTH the install dir and a pip
nvidia dir (defence in depth).
* Bare venvs without pip nvidia wheels still work via #5322's
binary_dir drop; pre-#5322 installs still work via #5324's
PATH augmentation.
* A reconstructed pre-PR scenario (cudart absent from binary_dir
and pip dirs not on PATH) leaves cudart unreachable, confirming
the test would catch a future regression.
Bonus housekeeping in studio/install_llama_prebuilt.py: drop the
pointless f-prefix on the literal "llama-" in the
windows_cuda_attempts pairing guard (no behaviour change; lint
nit flagged in the post-merge review).
The mocks model real artifact contents I verified empirically:
* pip download nvidia-cuda-runtime --platform win_amd64
produces nvidia/cu13/bin/x86_64/cudart64_13.dll.
* unzip on the b9103 cudart-llama-bin-win-cuda-13.1-x64.zip
produces exactly cudart64_13.dll + cublas64_13.dll +
cublasLt64_13.dll, no executables.
* objdump -p on the b9103 ggml-cuda.dll shows a static PE
import on cublas64_13.dll (the root cause of #5106 when
cublas64_13.dll is unreachable).
Refs #5106#5322#5324
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test_5106_windows_gpu_detection_mock: don't shadow real httpx
This file's name sorts before every other file in studio/backend/tests/
(starts with the digit '5'), so pytest collects it first. The previous
``sys.modules.setdefault("httpx", _httpx_stub)`` ran before any other
test imported real httpx, which meant the stub permanently shadowed
the real module for the rest of the collection. Tests that did
``from httpx import HTTPError, Response`` (test_anthropic_messages,
test_browse_folders_route, test_training_*, etc) then failed at
collection with ``ImportError: cannot import name 'HTTPError'``
because the stub did not define those names. The existing
test_llama_cpp_windows_nvidia_path.py did not trigger the same issue
because it sorts after test_a* / test_b* / etc, by which point the
real httpx has already been imported and setdefault is a no-op.
Switch the stub installation to ``importlib.util.find_spec(name) is
None`` so we only fall back to the stub when the real module truly is
not installed. Backend CI installs httpx, structlog, and the
studio/backend/loggers package is reachable via the sys.path
augmentation a few lines above, so on CI all three find_spec calls
succeed and no stubs are installed at all.
Also add HTTPError and Response to the stub module for the offline
case, so anyone running this test outside CI with httpx absent still
gets a stub that satisfies the broader test suite's imports.
Refs #5106
* test_5106 + llama_cpp: extract win32 PATH helper and harden the regression test
Follow-up to PR #5376's review feedback. Three real findings from the
bot reviewers, plus one stale one.
1. (codex P2 line 201, gemini medium line 209) The regression test's
_build_path_dirs_like_start_llama_server hand-copied the win32
branch of LlamaCppBackend.start_llama_server, so a future drop or
reorder of _windows_pip_nvidia_dll_dirs(sys.prefix) in production
would have passed the test silently.
Extract a new staticmethod LlamaCppBackend._build_windows_path_dirs
(binary_dir, prefix, cuda_path). Production start_llama_server now
calls this helper. The test's wrapper is reduced to a one-line
delegate that forwards to the staticmethod, so the regression
asserts against the exact production logic instead of a parallel
copy of it.
2. (codex P2 line 245) test_nvidia_smi_probe_reports_synthetic_gpu did
not clear CUDA_VISIBLE_DEVICES. On a shared GPU runner with the
variable set in the parent shell, _get_gpu_free_memory() filters
the mocked CSV and returns [] or falls through to the torch
fallback. Cleared CUDA_VISIBLE_DEVICES and NVIDIA_VISIBLE_DEVICES
via monkeypatch.delenv(..., raising=False).
3. (codex P2 line 66) _maybe_stub gated on importlib.util.find_spec
("loggers"), which returns a spec because studio/backend/loggers/
is on sys.path. But the actual import chain loads
loggers/handlers.py which does `from fastapi import Request,
Response` at module load. In a lightweight env without fastapi
installed, the stub never lands and `from core.inference.llama_cpp
import LlamaCppBackend` raises during collection. Switched
_maybe_stub to a real import attempt under try / except ImportError
so the stub falls into place when the package is discoverable but
not importable. CI has fastapi so this is purely a developer-
machine ergonomics fix.
The fourth comment (codex P1 line 85 "Keep the httpx stub from leaking
across tests") was already addressed by 7437e735, which replaced the
unconditional sys.modules.setdefault with the find_spec-gated
_maybe_stub. No code change needed.
Production behaviour is unchanged: _build_windows_path_dirs returns
exactly the same ordering start_llama_server used inline
([binary_dir, *pip_dirs, cuda_bin?, cuda_bin_x64?]).
Verification (run inside studio/backend):
pytest tests/test_5106_windows_gpu_detection_mock.py -v
-> 10 passed
pytest tests/test_llama_cpp_*.py tests/test_llama_server_args.py
tests/test_5106_windows_gpu_detection_mock.py -q
-> 171 passed
CUDA_VISIBLE_DEVICES=1 pytest tests/test_5106_windows_gpu_detection_mock.py::TestWindowsGpuDetectionAfter5106Fix::test_nvidia_smi_probe_reports_synthetic_gpu
-> 1 passed
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Rename Windows GPU detection test to a generic filename and trim comments
- studio/backend/tests/test_5106_windows_gpu_detection_mock.py
-> studio/backend/tests/test_windows_gpu_detection_mock.py
The file is the generic regression suite for Windows GPU detection;
encoding the issue number in the filename is noise.
- Shorten module docstring, helper docstrings, per-test docstrings and
inline comments in the renamed test file. No behaviour change,
all 10 cases still pass.
- Shorten the _build_windows_path_dirs docstring in
studio/backend/core/inference/llama_cpp.py and update the test-path
reference; trim the win32 call-site comment to one line.
Local verification:
- pytest studio/backend/tests/test_windows_gpu_detection_mock.py -- 10 passed.
- pytest studio/backend/tests/test_llama_cpp_windows_nvidia_path.py
studio/backend/tests/test_llama_server_args.py
studio/backend/tests/test_windows_gpu_detection_mock.py -- 110 passed.
* Studio: harden _wait_for_health against transient httpx ReadError
The probe loop in LlamaCppBackend._wait_for_health only caught
ConnectError and TimeoutException. On Windows, when llama-server.exe
accepts the TCP probe and then dies before sending HTTP headers, the
peer process RST closes the socket. httpx maps this to ReadError
("WinError 10054 -- An existing connection was forcibly closed by the
remote host"), which fell through the except clause and bubbled out of
_wait_for_health, the routes/inference.py load_model handler, and back
to /api/inference/load as an opaque 500.
The crash diagnostic Studio actually wants to surface lives on the
self._process.poll() branch at the top of the loop body: "llama-server
exited with code X. Output: ...". We never reached that branch on the
WinError 10054 path because the very first probe blew up.
Expand the except to also swallow ReadError and RemoteProtocolError so
the next 0.5-second iteration runs the poll() branch. Outcomes:
* Process really died: structured exit-code + last-stdout log line.
* Single transient probe blip: silently retried; load succeeds.
Adds studio/backend/tests/test_llama_cpp_wait_for_health.py with five
cases covering happy-path 200, transient ReadError + dead process,
RemoteProtocolError + dead process, ConnectError cycling until success,
and dead process before the first probe. The new cases would have
failed against the old except clause -- ReadError / RemoteProtocolError
would have propagated instead of returning False.
Found while triaging the Windows Studio GGUF CI flake on this PR's
5a6ddc34 push: llama-server.exe (b9203 prebuilt) crashed within 2.2 s of
launch on the GPU-less runner, and Studio reported "WinError 10054"
instead of an upstream-tag-attributable exit-code line.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* studio: tighten sandbox blocklist precision (bash, hf upload, NOFILE)
Three precision fixes in core/inference/tools.py. Same security
boundary; fewer false positives that broke legitimate sandbox use.
bash blocklist:
The per-token loop introduced in #5375 fired on any blocklist word in
any token position, so the entirely benign `grep -r curl .`,
`echo source the data`, and `ls /usr/bin/curl` were rejected with
"blocked command 'curl'". The position-anchored regex already covers
real command-position invocations, including `;rm`, `&&wget`, `$(rm)`,
`<(rm)`, backticked subshells, and `/usr/bin/sudo`. The token loop is
re-scoped: it only fires when the previous shlex token is a shell
separator (or at start of line), so split-quoting obfuscations like
`r''m -rf /` are still caught (shlex collapses them to a single
command-position token) while argument-position blocklist words pass
through. Trailing meta-chars glued to a shlex token (`rm;`) are
stripped before basename matching.
hf upload AST gate:
`_method_call_is_hf_upload` previously matched any method named
`upload_file` / `upload_folder` / `upload_large_folder` / `create_commit`
on any receiver, so paramiko.SFTPClient.upload_file, boto3.create_commit,
and similar non-HF SDK methods were rejected. The fallback now requires
an `import huggingface_hub` / `import hf_api` / `from huggingface_hub
import ...` somewhere in the same module. Fully-qualified
huggingface_hub.upload_file(...) calls are unchanged.
NOFILE env knob:
`RLIMIT_NOFILE = (1024, 1024)` was the only sandbox rlimit without an
env override. 1024 is below Linux's typical soft default and below
what multi-shard safetensors mmap chains need on Llama-3 70B-class
loads. Default is now 16384 with UNSLOTH_STUDIO_SANDBOX_NOFILE, parity
with the other rlimits.
15 new bash-blocklist-position tests pin both the false-positive
fixes and the still-blocked invariants (semicolon, &&, subshell,
backtick, split-quote, /usr/bin/ prefix, nested bash -c).
4 new hf-upload-import-gate tests pin both the false-positive
allowances and that HF-imported uses are still blocked.
1 new pin asserts the NOFILE env var is wired.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: cover command wrappers, find -exec, dynamic HF imports, NOFILE clamp
Reviewer follow-ups to the sandbox blocklist precision change.
Command-position scanner missed Bash command-prefix wrappers and inline
shell assignments. shlex tokenised `env curl`, `time curl`, `nohup rm`,
`FOO=bar curl`, `sudo rm`, etc. with the prefix at command position and
the real command at argument position, so the position-anchored check
returned set() while pre-PR's per-token scan caught them. Likewise the
position-anchored regex requires `^` or a shell separator before the
command, so `env curl` slipped through.
Reworked the scanner to track an expect_command flag plus a
prefix_pending flag:
- assignments (FOO=bar) keep expect_command=True for the next token,
- flags ('-oL', '--') keep it intact while prefix_pending is set,
- numeric duration args ('timeout 1 cmd') skip without breaking
expect_command,
- known wrappers (env, command, builtin, exec, time, nohup, nice,
setsid, stdbuf, timeout, ionice, chroot, sudo, doas, su, xargs)
set prefix_pending so the wrapper's command is still checked,
- shell separators now include `{`, `}`, `)`, `then`, `do`,
`else`, `elif` so brace groups and if/then/while/do bodies are
recognised as command positions.
Also lex with `shlex.shlex(punctuation_chars=";&|()`")` so split-quote
forms like `echo done; r''m -rf /tmp/x` and `echo done;r''m` tokenise
as `[..., ';', 'rm', ...]` and the command position check fires.
Added a small `find -exec CMD ... ;` / `-execdir CMD ... ;` pass so
`find . -exec rm -f {} +` and friends are caught even though the
direct token is at argument position to `find`.
Dynamic Hugging Face imports were treated as no-HF-in-scope. The
upload-method gate now also resolves `__import__('huggingface_hub')`,
`importlib.import_module('huggingface_hub')`, and bare
`import_module('huggingface_hub')` (via `from importlib import
import_module`) as HF imports, so HfApi().upload_file via dynamic
import is still blocked.
RLIMIT_NOFILE: setrlimit(NOFILE, (16384, 16384)) silently failed if
the parent's hard cap is below the requested value; the broad
except swallowed the OSError and left the sandbox at the parent's
default. Clamp the requested value to the inherited hard limit
before calling setrlimit.
Test cleanup: the existing test_cat_with_word_source_allowed had
`assert ... or True` so it could not fail; rewrote it to assert the
actual return value plus the two membership checks. Added
parametrised coverage for shell prefix wrappers, find -exec / xargs,
brace groups, if/then, while/do, split-quote command-name forms, and
dynamic HF import upload patterns.
Test:
- python -m pytest studio/backend/tests/test_sandbox_tools.py -q
-> 90 passed (was 67 before this commit)
- full studio/backend/tests/ minus llama_cpp_load_progress_live and
GPU CUDA_VISIBLE_DEVICES tests (pre-existing isolation flake)
-> 1063 passed
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: catch bare-name HF upload calls in AST gate
`from huggingface_hub import upload_file; upload_file(...)` is a
canonical HF call shape that the previous Attribute-only check missed:
the bare-name call lands as ast.Name (not ast.Attribute), so the
fuzzy gate skipped it.
Extend _method_call_is_hf_upload to also match ast.Name when HF is in
scope. Same import-gating discipline as the Attribute branch, so
paramiko/boto3 and locally-defined `def upload_file(...)` helpers
without HF imports still pass.
Pins: 4 new TestHfUploadImportGate cases (upload_file/folder/create_commit
bare-name imports blocked; local upload_file without HF import allowed).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: scope HF uploads to sandbox-local literals; block env / token leaks
The previous gate dropped every HF upload call. Two refinements make it
precise enough to allow legitimate sandbox->HF uploads while still
catching credential / file exfil:
- path_or_fileobj / folder_path / create_commit operation paths must be
sandbox-local relative-path literals (no '/', '~', drive letter, or
'..' segments). Variable / dynamic paths are rejected.
- Any positional or keyword argument that statically resolves to
os.environ / os.environ.get / os.getenv / bare getenv / subprocess
shape readers is rejected (env-var exfil).
- token / hf_token / api_token / api_key / auth_token / access_token /
password / secret kwargs are always rejected; sandbox env strips all
parent credentials by construction, so any value here is hard-coded
or lifted.
Recursive subtree walk in _reads_env_or_secret catches wrapper shapes
(str(os.environ), json.dumps(os.environ.items()), etc.).
Add TestSandboxEnvIsolation: pin that _build_safe_env builds the env
from a whitelist, not by stripping. Cover Linux/macOS/WSL/Windows
secret shapes. The whitelist is PATH / HOME / TMPDIR / LANG / TERM /
PYTHONIOENCODING (+ VIRTUAL_ENV / SystemRoot when applicable); HOME
points at the sandbox workdir, so HF / wandb / aws SDKs cannot reach
the operator's ~/.cache credentials.
Test classes added:
- TestHfUploadSandboxLocalPaths (relative literals allowed; absolute,
drive-letter, '~', '..', mid-path traversal, dynamic vars, and
open() of unsafe paths blocked, including create_commit recursion).
- TestHfUploadEnvAndSecretLeakBlock (os.environ subscript/get/getenv,
bare getenv, subprocess.check_output, str(os.environ), token=,
hf_token=, api_key=, and create_commit operations referencing env).
- TestSandboxEnvIsolation (no parent secret leaks into sandbox env).
131 tests in test_sandbox_tools.py pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: load cached GGUF models when fully offline
When huggingface.co is unreachable, GGUF model loads fail in three distinct
places even though the bits are already in ~/.cache/huggingface/hub. Each
failure has a different surface symptom:
1. list_gguf_variants() raises straight through HTTPException(500), so the
variant dropdown shows 'Failed to list GGUF variants'.
2. detect_gguf_model_remote() silently returns None after retries fail. The
caller then treats a GGUF-only repo as non-GGUF and routes it through the
transformers/MLX path. On Apple Silicon this surfaces as 'Unsloth currently
only works on NVIDIA, AMD and Intel GPUs.'
3. _download_gguf() loses list_repo_files() to the network and falls back to a
filename heuristic ('{repo}-{variant}.gguf'). When the repo name does not
echo the filenames (e.g. repo 'Qwen3.6-27B-MTP-GGUF' contains a file
'Qwen3.6-27B-UD-Q4_K_XL.gguf' with no MTP), hf_hub_download cannot find
that invented filename in the cache and aborts.
Fix in three layers:
- list_gguf_variants / detect_gguf_model_remote: honor HF_HUB_OFFLINE and
fall back to scanning the local HF cache snapshot when the API throws.
detect_gguf_model_remote still keeps its retry loop for transient flakes;
the cache fallback only kicks in after every attempt fails.
- _download_gguf: when list_repo_files() fails, look up variant -> real
filename inside the cached snapshot before resorting to the heuristic.
- llama_cpp.load_model / inference worker startup: when DNS for
huggingface.co fails (2s probe), set HF_HUB_OFFLINE=1 for the process so
every hf_hub_download call below resolves from cache instantly instead of
spending ~25s on five exponential retries.
Online behavior is unchanged: the API is tried first and only used to fail
over. The cache scan is a strict subset of what list_local_gguf_variants
already does today for local paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten inline comments on offline GGUF fallback
* studio: address review feedback on offline GGUF fallback
Fixes from the review pass on #5505:
* ruff F823 (lint CI red): the late `import os` at the bottom of
LlamaCppBackend.load_model made `os` a function-local name, so my
new `os.environ` reference at the top of the same method was a
use-before-bind. Surfaces at runtime as
'cannot access local variable os where it is not associated with a value'
and is why the Mac/Windows Studio API jobs were failing too. The
env-var mutation has been moved into a module-level contextmanager,
so load_model no longer touches `os` directly.
* Codex P1: cache variant match now uses the relative path, not the
basename. Layouts like `BF16/foo.gguf` (variant token only in
parent dir) were silently skipped, falling through to the bogus
`{repo}-{variant}.gguf` heuristic and failing offline loads of
models stored under quant-named subdirs.
* Codex P1: HF_HUB_OFFLINE no longer persists past one model load.
llama_cpp.load_model now uses a contextmanager that probes DNS,
sets HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE only when DNS is dead,
and pops them in finally (preserving any prior user setting of
TRANSFORMERS_OFFLINE). Pre-existing user-set HF_HUB_OFFLINE is
respected as a no-op. worker.py keeps the startup probe because the
orchestrator spawns a fresh worker per load -- comment updated to
make that lifecycle explicit, and a warning is now logged.
* Gemini: cache-dir lookup centralized in `_iter_hf_cache_snapshots`.
Three near-identical copies (in list/detect helpers and the
llama_cpp offline scan) now go through one helper.
* Gemini: `huggingface_hub.utils.is_offline_mode` does not exist in
1.x (verified locally); `huggingface_hub.constants.HF_HUB_OFFLINE`
is snapshot-at-import-time and does not reflect runtime mutations.
Manual env-var parsing kept.
* socket probe now saves and restores the prior default timeout
instead of unconditionally setting None on exit, so it composes
with caller code that already configured a timeout.
* worker.py probe now logs a warning when offline mode is auto-enabled
so debugging the case isn't blind.
* studio: regression tests for offline GGUF cache fallback
Lock in the offline fallback path from #5505 so future refactors can't
silently regress either bug. 26 tests, 0.55 s, no network/GPU/subprocess.
Covers:
* _iter_hf_cache_snapshots: missing cache, missing repo, missing
snapshots/, newest-mtime ordering, case-insensitive repo match.
* _list_gguf_variants_from_hf_cache and the list_gguf_variants
online/offline-env/API-exception/reraise paths.
* _detect_gguf_from_hf_cache and detect_gguf_model_remote 3x-fail
fallback. Pre-existing RepositoryNotFoundError early-return preserved.
* Codex P1 #1 regression: BF16/foo.gguf (quant only in subdir name)
must resolve via _detect_gguf_from_hf_cache, which now matches the
snapshot-relative path rather than the basename.
* _probe_dns_dead: returns True/False, restores prior socket timeout.
* Codex P1 #2 regression: _hf_offline_if_dns_dead sets env only inside
the block, restores on exit (including on exception), re-probes DNS
on the next call so a transient hiccup cannot lock the long-lived
LlamaCppBackend singleton offline. Honors a user-set HF_HUB_OFFLINE
as a no-op. Preserves a user-set TRANSFORMERS_OFFLINE across exit.
Follows the existing studio backend test stub pattern (loggers /
structlog / httpx stubs + backend dir on sys.path).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: extend offline cache fallback to _download_mmproj and quant label
Two follow-up fixes from the review pass on #5505:
* _download_mmproj() now mirrors _download_gguf()'s offline path:
when list_repo_files() fails, scan the local HF cache snapshot for
any GGUF whose basename starts with mmproj-. Without this, offline
vision GGUF loads succeed at the main weight (the existing PR fix)
but the mmproj returns None and llama-server starts without vision
support. Same _iter_hf_cache_snapshots helper, F16 preference and
fallback to the first match are preserved.
* _extract_quant_label() now considers parent directory segments when
the basename has no quant token. Layouts like BF16/foo.gguf are
already documented in this file and are returned by the new
snapshot-relative-path filter in _download_gguf; before this fix
their variant label collapsed to "foo" (the last hyphen segment of
the basename). Regex is the same; the search just walks parent
segments innermost-first if the basename misses.
Tests (studio/backend/tests/test_offline_gguf_cache_fallback.py):
* TestExtractQuantLabelSubdir: basename quant unchanged, quant-only-
in-parent, UD- prefix in parent, deeper nesting picks the
innermost matching segment.
* TestDownloadMmprojOfflineCacheFallback: cache fallback returns the
mmproj when list_repo_files fails, F16 preference holds when both
variants are in cache, no-mmproj cache returns None.
* httpx stub now prefers the real package when installed (the CI
install list already includes it) and falls back to the stub only
when httpx is genuinely missing. Newer huggingface_hub imports
HTTPError/Response/Request at module load, so the previous
fixed-set stub broke when those names were added upstream.
26 existing cases plus 7 new = 33 pass in 0.74s.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust offline cache + DNS probe per PR #5505 review
Four review findings tightened, with regression tests:
- list_local_gguf_variants subdir collapse (P1 codex 10:08): pass the
snapshot-relative path to _extract_quant_label so BF16/foo.gguf and
Q4_K_M/foo.gguf produce distinct labels instead of folding to the same
basename pseudo-quant.
- list_gguf_variants cache fallback (P2 codex 12:10): surface
RepositoryNotFoundError / GatedRepoError / RevisionNotFoundError /
EntryNotFoundError to the caller instead of masking with stale cache,
matching detect_gguf_model_remote.
- _detect_gguf_from_hf_cache mmproj (P2 codex 12:10): exclude mmproj
files from the candidate list so a partial cache with only a vision
projector cannot route the projector as the main model.
- _probe_dns_dead global timeout (P2 codex 13:06): run the gethostbyname
on a daemon thread with join timeout so concurrent sockets in the same
interpreter never inherit a process-wide socket.setdefaulttimeout
mutation. Same shape applied in worker.py's startup probe.
* Make llama-server health check tolerant of warmup races
Two layered fixes for the Windows GGUF smoke CI Tool calling Tests
flake that exit-22'd on a single httpx.ReadError during llama-server
warmup. The 'windows-latest -> windows-2025-vs2026' image rollout is
hitting main with the identical symptom.
A. _wait_for_health: catch httpx.ReadError, RemoteProtocolError,
WriteError alongside ConnectError and TimeoutException. A TCP RST
mid-read while llama-server is still binding the port (WinError
10054) is a 'still warming up' signal, not fatal. The existing
_process.poll() check still wins for real crashes.
B. _drain_stdout + spawn: tee llama-server stdout/stderr to a
per-launch log file at ~/.unsloth/studio/logs/llama-server/
<port>.log. Any future subprocess crash leaves a forensic trace
on disk even when Studio's traceback only captures the symptom
(ReadError) and not the cause. Best-effort: a logging-side OSError
never blocks the load.
Regression coverage: TestWaitForHealthRetriesOnReadError pins the
retry behaviour for the three new exception types and verifies that a
real process exit still short-circuits the loop.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci(windows): retry inference/load + collect llama-server logs
Composite fix for the Tool calling Tests flake that exit-22'd on a
single httpx.ReadError during llama-server warm-up. The
windows-latest -> windows-2025-vs2026 runner image rollout has been
hitting main with the identical symptom.
- All three jobs (openai-anthropic, tool-calling, json-images) now
retry POST /api/inference/load up to 3 times with 10s backoff and
preserve the response body for post-mortem. One transient 500 no
longer fails the whole job.
- A new "Collect llama-server logs" step copies the per-launch
llama-server stdout teed by Studio under ~/.unsloth/studio/logs/
llama-server/ into the workspace, and the upload-artifact step
now includes logs/llama-server/*.log so any future subprocess
crash leaves a forensic trace.
---------
Co-authored-by: shimmyshimmer <shimmyshimmer@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* studio/chat: reuse Anthropic code_execution container across turns
Mirror the OpenAI shell-tool reuse path for Anthropic. Backend latches
`message.container.id` off the message_start SSE event, emits a synthetic
container_ready _toolEvent, and forwards a stored id back on the next
turn via the top-level `container` request field. Stale-id 4xx surfaces
as container_invalidated so the next turn falls back to auto-create.
* studio/chat: temp diag log of Anthropic SSE events when code_execution is on
To locate where the API actually emits container.id on the stream.
* studio/chat: latch Anthropic container id from message_delta, drop diag
Anthropic surfaces container.id on `message_delta.delta.container`, not
on `message_start` (at start the container is not provisioned yet).
Move the latch + container_ready emit to message_delta and remove the
temporary raw-event log.
* Studio: serialise GGUF reload and inherit unsloth-run extra args
Closes#5401.
Three related GGUF reload bugs reproduced against `unsloth studio run -m unsloth/Qwen3-0.6B-GGUF --gguf-variant Q4_K_M --top-k 20 --seed 42`:
1. The `POST /api/inference/load` already-loaded short-circuit only compared `model_identifier` and `hf_variant`. A same-(model, variant) Apply that flipped `cache_type_kv` / `speculative_type` / `chat_template_override` / `max_seq_length` / `llama_extra_args` returned `status="already_loaded"` and the new setting silently never reached llama-server.
2. The frontend chat-settings Apply path POSTs `/unload` then `/load` without round-tripping `llama_extra_args`. Every reload after `unsloth run --some-flag X` quietly dropped `--some-flag X` from the spawned `llama-server` command line.
3. `LlamaCppBackend.load_model` released `_lock` between Phase 1 (kill) and Phase 3 (spawn) so two concurrent loads each passed Phase 1 with `self._process is None`. Both ran Phase 2 (download), both reached Phase 3, and the Phase 3 defensive `_kill_process()` from #5171 collapsed them to one survivor only after both `subprocess.Popen` calls had landed. For the 86 GB MoE in #5161 / the model in #5401 the overlap window was tens of seconds, long enough to OOM the host. With a 0.6B model the pgrep timeline showed two simultaneous PIDs for 3.3 s on `main`.
Fix:
`studio/backend/core/inference/llama_cpp.py`
* Add `self._serial_load_lock = threading.Lock()`. The whole body of `load_model` runs under this lock so two concurrent `/api/inference/load` requests are strictly sequential. The fine-grained `_lock` and the Phase 3 defensive `_kill_process()` from #5171 are kept as a second layer. `/unload`, `/status`, and `/load-progress` are unaffected because they only touch the fine-grained lock or read properties.
* Add `self._extra_args` plus an `extra_args` property, written inside `load_model` whenever the caller supplies a non-`None` value. `unload_model()` deliberately does not reset it so the route layer can inherit the args across the frontend's `/unload` + `/load` gap.
`studio/backend/routes/inference.py`
* Add `_request_matches_loaded_settings(request, llama_backend)` that compares `max_seq_length`, `cache_type_kv`, `speculative_type`, `chat_template_override`, and `llama_extra_args` between the incoming request and the live backend. Same-(model, variant) requests whose runtime settings differ now fall through to a real reload instead of returning `already_loaded`. A missing `llama_extra_args` field on the request is treated as "inherit current", so the short-circuit still fires when the only difference is the frontend not echoing the CLI flags back.
* GGUF load branch inherits `llama_extra_args` from `llama_backend.extra_args` when the request omits the field, re-validates through `validate_extra_args`, and forwards the result to `load_model(...)`. An explicit `[]` from the caller is still honoured as "clear".
Verified end to end against a live `unsloth studio run` instance:
| Scenario | Before | After |
| --------------------------------------------------------------- | --------- | ------------------------------------------------------------------------ |
| `/load` same (model, variant, settings) | 1 PID, `already_loaded` | unchanged |
| `/load` same model, variant, new `cache_type_kv=q8_0` ctx=8192 | `already_loaded`, settings dropped | `loaded`, `/status` reports the new settings, new server has `-c 8192 --cache-type-k q8_0 --top-k 20 --seed 42` |
| Frontend Apply `/unload` + `/load`, new settings, no `llama_extra_args` field | Drops `--top-k 20 --seed 42` | Preserves `--top-k 20 --seed 42` |
| `/unload` + two parallel `/load` | Two PIDs for 3.3 s | Max simultaneous count = 1 across the full pgrep timeline |
| `/load` with `llama_extra_args=[]` (explicit clear) | n/a | `loaded`, new server has no `--top-k` / `--seed` |
| `/load` with `llama_extra_args=["--top-k","30","--seed","7"]` (override) | n/a | `loaded`, new server has the supplied flags |
`pytest studio/backend/tests` is green except for one pre-existing terminal-width-sensitive assertion (`test_studio_api.py::test_help_output`) and the pre-existing `test_studio_api.py` fixture errors that fail on unmodified main too. No new regressions.
* Studio: track requested n_ctx so Auto-slider flips trigger a reload
Review feedback on PR #5427 from gemini-code-assist.
The original short-circuit compared ``request.max_seq_length`` against
``llama_backend.context_length`` (the effective context). VRAM-fit
logic can cap the running server below what the caller asked for, so
this comparison incorrectly returns ``already_loaded`` when the user
flips the slider from an explicit length (e.g. 8192) back to "Auto"
(0): the explicit request was capped to, say, 4096, and the new "Auto"
request reads ``backend.context_length == 4096`` and decides nothing
changed.
Track the originally requested ``n_ctx`` on the backend instead and
compare against that. ``requested_n_ctx == 0`` means the last load
asked for the model's native length; ``request.max_seq_length == 0``
matches it.
Verified in the sandbox suite (now 90 tests):
- ``test_explicit_to_auto_triggers_reload`` -- loaded with explicit
8192, then Apply with ``max_seq_length=0`` falls through to a real
reload and the new server runs at the native 40960.
- ``test_auto_to_explicit_triggers_reload`` -- inverse direction.
- ``test_explicit_to_same_explicit_short_circuits`` -- re-Apply with
the same explicit value still short-circuits (no needless reload).
- Existing scenarios (kv change, spec change, template change, extra
args inherit, parallel-load stress, frontend Apply flow) unchanged.
``pytest studio/backend/tests`` still green on the same set of tests;
the pre-existing ``test_help_output`` failure and ``test_studio_api``
fixture errors are unaffected.
* Studio: tighten comments in the 5401 fix
Trim the verbose explanatory comments and docstrings introduced in
f9cbec3b and dd0b1d58 down to one-line summaries. The "why" still
points at issue #5401; the multi-paragraph rationale belonged in the
PR body, not the source. No behaviour change.
* ci: retrigger after zoo drift + IPython fixes landed in main
* ci: retrigger Mac Studio UI CI after transient fetch flake
* Studio: address six P2 followups on the 5401 reload PR
Tightens the inheritance and serial-load paths to close the six P2
findings raised by codex-connector on PR #5427 against `f9cbec3b` /
`dd0b1d58`.
1. Re-check loaded state before killing queued loads. Two duplicate
`/api/inference/load` requests both pass the route-level
`is_loaded` gate before the first publishes `_healthy = True`. The
second waits on `_serial_load_lock`, enters Phase 1, and tears down
the just-spawned llama-server for a redundant full reload. Added
`LlamaCppBackend._already_in_target_state(...)` and a short-circuit
at the top of the serial-lock block: if the live server already
satisfies the kwargs, return True without killing.
2. Don't inherit CLI overrides that shadow new first-class settings.
`unsloth run -c 4096` is a permitted pass-through; the validator
docs explicitly call out `-c`/`--ctx-size`. Stored in `_extra_args`
and appended after Studio's own flags, the inherited `-c 4096`
silently won the last-wins parse against a new
`max_seq_length=8192`. Added `strip_shadowing_flags` in
`llama_server_args.py` (covers `-c`, `--cache-type-k/v`, `--spec-*`,
`--chat-template*`, `--jinja`/`--no-jinja`) and the route runs the
inherited list through it before validate + forward.
3. Restrict inherited llama args to the same GGUF model. `_extra_args`
is deliberately preserved across `unload_model()` for the chat-
settings Apply flow (`/unload` + `/load` with no `llama_extra_args`
field). Now also track `_extra_args_source = (model_identifier,
hf_variant)` so the route can refuse cross-model inheritance.
`LlamaCppBackend.extra_args_source` exposes the tuple.
4. Persist extras only after a successful load. `_extra_args` was
written at the top of `load_model` before Popen + health check, so
a failed startup left bad args in place to poison the next UI
retry. The write (along with `_requested_n_ctx`) is now deferred
until after `_healthy = True`.
5. Ignore speculative diffs for vision loads. `load_model` silently
gates speculative decoding on `not is_vision`, so the backend's
`_speculative_type` stays `None` for vision models. The route's
comparator now normalises the request's value to `"off"` when
`llama_backend.is_vision` to avoid a no-op reload of a vision
server every time the dropdown defaults to `default`. The
`_already_in_target_state` helper applies the same rule.
6. Wait for the replacement server before short-circuiting. `_kill_process`
did not clear `_healthy`; the new first-class settings
(`_cache_type_kv`, `_speculative_type`, `_chat_template_override`)
are written under `_lock` BEFORE Popen + `_wait_for_health`. A
duplicate `/load` arriving during the new server's warm-up window
could short-circuit against the not-yet-healthy replacement and the
caller would start inference against a server that was still
loading. `_kill_process` now sets `_healthy = False` in its
`finally` block so `is_loaded` returns False from the moment the
old server is killed until the new one finishes warm-up.
Tests:
- Sandbox suite under `./temp/sim_5401/` extended to 136 tests (was
90): new unit coverage for `strip_shadowing_flags` (12 cases),
`_kill_process` clears `_healthy`, `extra_args_source` lifecycle and
cross-model behaviour, failed-load preserving prior extras, and the
duplicate-load short-circuit at `load_model` level. New live
integration cases verify shadow-strip via `pgrep` on the live
llama-server cmdline, cross-model refusal, and PID stability across
a duplicate-load race. All 136 pass.
- `pytest studio/backend/tests --deselect test_studio_api.py`:
1079 passed, 46 skipped, identical to the pre-change count. The
pre-existing `test_studio_api.py` fixture errors and the
terminal-width-sensitive `test_help_output` are unaffected.
- Ruff: clean on the three modified files.
* Studio: tighten GGUF reload inheritance and duplicate-load guard
Re-narrow llama_extra_args to None after validate_extra_args when the
incoming request omitted the field, so the backend can distinguish
"caller omitted, inherit prior load" from "caller explicitly cleared
to []". Without this a queued duplicate /load reaches the backend as
[] and fails _already_in_target_state's exact-equality check, killing
the just-started llama-server. The pass-through validate call from
the original "forward llama-server args from unsloth studio run /
unsloth run" change is preserved as-is; only the post-pass narrowing
is new. Cross-source loads now explicitly clear extras so a model
switch can't accidentally inherit via the backend's "no opinion"
semantics.
Store the caller's hf_variant kwarg (None for local GGUF files) in
_extra_args_source instead of the derived self._hf_variant
(an extracted filename quant label like "Q4_K_M"). Same-source check
in the route is now symmetric for HF and direct-file loads.
Add gguf_path to _already_in_target_state and prefer on-disk path
identity when both backend and caller have a path. This stops the
duplicate-load guard from killing a healthy server on repeat local
loads (where hf_variant is None on the caller side but extracted on
the backend side).
Split shadow-flag stripping into per-group toggles (context / cache /
spec / template). The route now opts into stripping only the groups
whose first-class field was actually set on the incoming request, so
an inherited --chat-template-file survives an Apply that omits
chat_template_override. _request_matches_loaded_settings detects
shadowing extras on the inherit path and falls through to a real
reload so the strip can run.
Mark --spec-default, --jinja, --no-jinja as boolean inside the
shadow stripper so the value-consuming heuristic no longer eats the
following positional token.
* Studio: trim comments around GGUF reload inheritance
* Studio: cover GGUF reload inheritance and shadow-flag stripping
* Studio: drop redundant issue refs from inheritance comments
* Studio: drop redundant issue refs from inheritance comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: key inheritance source off resolved gguf_variant
codex-connector P2 on PR #5427cd14cae1: the inheritance gate at
``routes/inference.py:696`` compared the stored ``source[1]`` against
``request.gguf_variant``, but the HF branch loaded with
``hf_variant = config.gguf_variant`` (the *resolved* variant after
ModelConfig auto-pick). When the caller omitted ``gguf_variant`` on a
follow-up Apply, ``source[1] == "Q4_K_M"`` but
``(request.gguf_variant or "") == ""``, ``same_source`` returned False,
and the chat-settings Apply silently dropped CLI pass-through flags
for every auto-pick / local-file load.
Fix both sides of the comparison to key off ``config.gguf_variant``:
* The route compares ``source[1]`` to ``config.gguf_variant`` (the
resolved label) rather than the request field.
* The local-mode load_model call now passes
``hf_variant = config.gguf_variant`` so ``_extra_args_source``
stores the same string the route reads back. The HF branch already
did this.
Sandbox: added test_source_records_caller_variant_not_extracted_label
to lock the storage key contract.
``pytest studio/backend/tests --deselect test_studio_api.py``:
1100 passed, identical to pre-change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: deny upstream --ui family on llama-server pass-through
The validator's web-UI block named only ``--webui`` / ``--no-webui``,
which is llama.cpp's pre-rename spelling. Current upstream
(``tools/server/README.md``) uses ``--ui`` / ``--no-ui`` plus
``--ui-config``, ``--ui-config-file``, and ``--ui-mcp-proxy`` /
``--no-ui-mcp-proxy``. Without these in the denylist a user could
``unsloth run --ui`` and enable llama-server's built-in web UI on
the port Studio's reverse proxy targets, breaking the UI surface.
Keep the legacy ``--webui`` group so the validator still rejects
old binaries that haven't been re-spelled.
Cross-referenced against the README's full flag list; this was the
only gap for the post-#5401 inheritance / shadow-strip work. Pass-
through flags from every other README category (sampling, jinja,
ctx, cache, threads, GPU, reasoning, grammar, chat-template-kwargs)
already validate cleanly; sandbox suite exercises ~60 of them in
the new ``test_08_llama_server_pass_through.py``.
``pytest studio/backend/tests --deselect test_studio_api.py``:
1100 passed, identical to pre-change.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio/chat: fix OpenAI container delete UX (expired filter, TTL cap, idempotent 404, refresh-on-error)
- Filter status="expired" from /containers/list so the picker only
shows usable containers. OpenAI keeps expired entries in the list
indefinitely, which made delete look broken.
- Cap ttl_minutes at 20 (backend Field + frontend TTL_MAX + persistence
clamp). OpenAI's actual hard limit is 20; the prior 10080 cap caused
integer_above_max_value rejections on create.
- Treat 404 on delete as idempotent success in the frontend client so
already-gone containers don't surface a scary error toast.
- Run refresh() in finally for onCreate/onDelete so the picker stays
in sync with OpenAI even when the call errors.
- Add route-level test for the expired filter.
* studio/chat: add diagnostic logging for OpenAI /containers DELETE
Trace what arrives at /external/openai/containers/delete (subject,
container_id, base_url) and what we send to OpenAI (URL, presence
of Authorization, value of OpenAI-Beta) plus the full response
status + body (capped at 300 chars). Helps confirm whether the
beta header is on the wire and whether OpenAI's response actually
reports deleted=true, when users report the delete "not taking".
No secrets are logged — Authorization is reported as a boolean.
* studio/chat: log raw /containers list response from OpenAI
Sibling to the delete diagnostics. After a confirmed delete
(deleted=true on the wire), we want to see whether the very next
list call returns the just-deleted id — that distinguishes
"OpenAI eventually-consistent list" from "frontend stale state".
Logs each entry's id + status only; no names, no timestamps.
* studio/chat: fingerprint decrypted API key for container CRUD
Logs kind (sk-proj-/sk-/other), length, and last-4 chars only —
never the full secret. Lets us compare what the backend actually
uses against the key the user expects, since the same DELETE
request shape can produce different results across keys
(project-scoped containers: list is permissive but delete requires
the owning project's key).
* studio/chat: use fresh httpx client for /v1/containers DELETE
Same key, same headers, same URL via the shared _http_client
returned deleted=true but the container persisted in subsequent
list calls. A fresh httpx.AsyncClient with the identical request
shape (verified with a standalone reproducer) deleted the same
container cleanly. Suspect connection-pool state from earlier
chat-completion streams interferes at the edge — switching to a
per-call client side-steps it entirely. Scoped to delete only;
list/create keep using the shared pool until we can confirm the
same fix is needed there.
* studio/chat: log OpenAI response headers on container DELETE
Adds cf-ray / x-request-id / openai-organization / openai-project /
openai-processing-ms to the delete-response diagnostic line. Lets
us cross-reference a failing delete against OpenAI support (or
against a working standalone reproducer) using the unique
request-id and edge node.
* studio/chat: client-side tombstone for just-deleted OpenAI containers
OpenAI's /v1/containers DELETE returns {"deleted": true} but the
list endpoint can keep returning the same container for several
minutes (replica lag or in-use silent no-op — undocumented per
developers.openai.com/api/docs/guides/tools-shell). Our backend
sends the correct DELETE with OpenAI-Beta: containers=v1 and a
standalone reproducer shows the same behavior, so the right fix
is UI-side rather than waiting on OpenAI.
After a successful delete, the id goes into a per-component
tombstone map with a 5-minute expiry. visibleContainers (now the
single chokepoint feeding sortedContainers, auto-bind, and the
all-containers list) filters those ids out. A 30s sweep clears
expired tombstones so the picker recovers automatically if OpenAI
eventually catches up (or the container's TTL elapses).
* studio/chat: tombstones live for the page lifetime; drop API key fingerprint log
- Tombstones change from Map<id, expiry> to Set<id>: once tombstoned,
the id stays hidden from the picker until page reload. OpenAI's list
can keep returning a deleted id for an undocumented and variable
amount of time; automatically un-tombstoning after a fixed window
surfaces it again and creates more confusion than it solves. The
container's own TTL eventually expires the entry on OpenAI's side,
and the expired-status filter at the backend list route hides it
anyway.
- Remove the periodic sweep effect (dead code without expiries).
- Remove the api-key fingerprint log added during debugging — it
served its purpose (confirmed parity) and isn't needed long-term.
* studio/chat: built-in code execution for Anthropic Claude 4.x
Wire Anthropic's server-side code_execution_20250825 tool to the
existing Code pill in the composer. Pill lights up only for Claude
Opus/Sonnet/Haiku 4.x models that the docs list as compatible; pairs
independently with Search. Backend appends the tool entry plus the
code-execution-2025-08-25 beta header, and translates the SSE
server_tool_use / *_tool_result blocks (bash + text_editor sub-tools)
into the _toolEvent shape the frontend renderer consumes. File
uploads via the Files API are a deliberate follow-up.
* studio/chat: enable code execution pill in in-thread composer too
thread.tsx renders its own composer with a separate CodeToolsToggle
that was still gated on supportsTools only, so the pill stayed
disabled inside an active thread even after picking Anthropic 4.x.
Surface the capability through the runtime store
(supportsBuiltinCodeExecution, set from chat-page alongside
supportsBuiltinWebSearch) and read it in the toggle.
* studio/chat: built-in code execution for OpenAI cloud gpt-5.5
Extend the Code pill to OpenAI cloud's gpt-5.5 / gpt-5.5-pro via the
shell tool on /v1/responses. Per-thread container reuse: capture the
container_id from each response on a synthetic container_ready event,
persist it onto the ThreadRecord, and pass it back as
environment.type="container_reference" on follow-up turns so the
model sees filesystem state from prior turns until OpenAI's idle
expiry. Stale ids surface a container_invalidated event that clears
the thread record so the next turn falls back to container_auto.
Gated strictly on OpenAI cloud (api.openai.com base URL) — Ollama,
llama.cpp, vLLM, and custom OpenAI-compat presets won't see the
shell tool entry even when their providerType collapses to "openai".
* studio/chat: OpenAI shell-tool container management UI
Side-panel section (settings sheet → Code Execution) for managing
OpenAI's shell-tool containers per thread. Three controls:
- New-container idle timeout (provider-level default, pre-fills the
create dialog and is used by the lazy-create path on a thread's
first turn when set to a non-default value).
- Active container picker for the active thread — pick any existing
container or stay on "Auto-create per thread".
- Inline create form (name + idle TTL) and per-row delete actions.
Three new backend endpoints under /api/inference/external/openai/
containers/{list,create,delete} proxy to OpenAI /v1/containers using
the encrypted API key. All three reject non-cloud base URLs up front
so the picker stays scoped to api.openai.com.
Deleting a container clears all thread bindings pointing at it; the
next turn falls back to auto-create.
* studio/chat: inherit container across threads + styled active picker
New threads on the same OpenAI provider now default to the most
recently used container instead of "Auto-create per thread" — both
in the chat-adapter (so a send works even if the side panel was
never opened) and in the side panel itself (auto-binds the active
thread when the dropdown loads on a thread that has no container).
Picker is visually emphasized with an accent panel and the
currently-active row in the list below is highlighted with the same
accent so the two views stay in sync.
* studio/chat: friendly English-word names for auto-created containers
Replaces the "chat-<thread-id-slug>" auto-name with a random
English-word + short hex suffix (e.g. "kestrel-3f9c"). Applies only
to the chat-adapter's lazy-create path; the OpenAI container_auto
path stays unnamed (only fires when no custom TTL is set).
* studio/chat: always pre-create OpenAI containers via frontend
Drops the TTL-based gate on the chat-adapter's lazy-create path so
every code-execution container the user ever sees in the picker has
a friendly English-word name. The backend's container_auto fallback
stays as a safety net (used only if the POST /v1/containers call
fails); in practice that branch should be rare.
* studio/chat: send OpenAI-Beta header for /v1/containers CRUD
Without OpenAI-Beta: containers=v1, OpenAI returns 200
{"deleted": true} for DELETE /v1/containers/{id} but does not
actually remove the container. The list call then keeps returning it,
making it look like Studio's "Delete container" button is broken.
Verified 2026-05-15 against api.openai.com: DELETE with the beta
header returns 200 and removes the container; the same DELETE without
the header returns the same 200 deleted:true body but the container
stays alive.
- Add _container_headers() that merges OpenAI-Beta on top of the
shared auth headers; route list / create / delete through it.
- Verify the DELETE response body reports {"deleted": true}; raise
httpx.HTTPError otherwise so the route surfaces a 5xx instead of
silently reporting success on a silent no-op.
- Add tests covering header propagation and the deleted-flag guard
(true, false, missing key, non-JSON body, 4xx passthrough).
* studio/chat: surface unpersisted-thread picker no-op as a toast
The "Active for this thread" container picker uses
db.threads.update(activeThreadId, ...), which silently returns 0 rows
affected when the thread record isn't yet in IndexedDB. That happens
on a brand-new thread where the user toggles code execution on and
opens settings before sending the first message — the chat adapter
only materializes the thread row on first send. The picker would
appear to ignore the user's selection and snap back to "Auto-create
per thread".
- onPick now awaits the update and toasts an actionable hint
("Send a message first to pin a container to this thread.") when
the update affected zero rows.
- Auto-bind effect comment clarifies why it stays best-effort silent.
The auto-bind effect itself is unchanged: it's a heuristic that
should not nag the user when it can't apply.
* studio/chat: let user pick OpenAI container before first send
Previously the picker silently no-op'd until the user sent the first
message, because Dexie's ThreadRecord is only materialized inside the
runtime-provider's `initialize` hook (assistant-ui's first-message
callback). That kept users from binding a thread to an existing
OpenAI container up front; they had to either send a message and
risk the chat adapter auto-creating one, or accept the cross-thread
inheritance default.
- Export `ensureThreadRecord` from runtime-provider so other surfaces
can materialize the row idempotently.
- In OpenAICodeExecSection.onPick, await ensureThreadRecord before
the update, with modelType="base" (the settings sheet that hosts
this section is only rendered in single-thread mode).
Behaviour after this commit:
- New thread + user picks a container in the sidebar → thread row is
created with that container_id; first send uses it, no auto-create.
- New thread + user does nothing → row still absent; first send goes
through the existing inherit/lazy-create path as before.
- The auto-bind effect remains silent best-effort: it does not
eagerly create the thread row, so it cannot pre-empt the user's
pick on a fresh thread.
* studio/chat: drop "Auto-create per thread" option, default to latest
The dropdown previously offered "Auto-create per thread" as an
explicit value (null in storage), with the chat-adapter then
inheriting from the most recent container at send-time. That made
the picker display disagree with what the backend would actually do:
the picker said "auto", but the backend was reusing an existing
container.
Behaviour after this commit, when code execution is enabled on an
OpenAI cloud provider:
- Containers list non-empty: dropdown defaults to the container with
the latest lastActiveAt, eagerly bound via ensureThreadRecord +
db.threads.update so the bind survives even when the thread row
has not been materialized by the chat adapter yet. User can pick
any other container in the list.
- Containers list empty: render a disabled placeholder "(none yet —
will be created on first send)". The chat-adapter's lazy-create
path (chat-adapter.ts:1040-1082) mints the first container on
first send and writes it back to the thread; the next refresh
surfaces it in the picker.
Expiration mid-operation is unchanged: the existing
container_invalidated _toolEvent clears the thread's stored id and
the next turn re-creates.
* studio/chat: fix picker stuck on "Selecting most recent…" + manual-create binding
Two follow-up fixes to the picker rework in d0cbeb99b.
1) The dropdown was getting stuck on the "Selecting most recent…"
placeholder option even after the auto-bind write completed,
because the select was controlled by `activeContainerId` (whatever
sits in Dexie) and there's a brief window between the auto-bind
firing and useLiveQuery propagating the new row back. Decoupled
the rendered value from the Dexie state: compute the displayed id
locally as `activeContainerId ?? sortedContainers[0]?.id`, so the
most-recent container's name shows up immediately. The auto-bind
effect still writes the bind to Dexie so the chat adapter sees it
on send. Dropped the placeholder option entirely.
2) The manual "Create container" flow (`onCreate`) bound the new
container to the active thread with a bare `db.threads.update`.
On a brand-new thread that hadn't been materialized yet, the
update affected 0 rows; the user's next send then went through
cross-thread inheritance / lazy-create and could land on a stale
container, surfacing as "container does not exist". Same fix as
`onPick`: ensureThreadRecord before update so the bind lands.
* make API key optional for local providers (llama.cpp/vLLM/Ollama)D
* chore: reduce comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* polish: update provider dropdown and rename cloud
* fix: tighten custom provider fallback handling
* fix: external provider fallback typing
* studio: wire the chat Search button to OpenAI's built-in web_search tool
When the active model is an OpenAI external provider and the user
clicks the existing Search pill in the composer, the chat-completion
request now carries the unified enable_tools shorthand:
enable_tools: true
enabled_tools: ["web_search"]
The backend's stream_chat_completion threads enabled_tools through
to _stream_openai_responses, which translates it into the Responses
API tool schema:
body["tools"] = [{"type": "web_search"}]
per the OpenAI Responses tool spec
(https://developers.openai.com/api/docs/guides/tools). OpenAI then
runs the search server-side before the model replies; the search-
informed answer streams back through the existing
response.output_text.delta path. web_search_call lifecycle events
are silently ignored for now — sources / status indicators are
follow-up scope.
Frontend:
- provider-capabilities.ts: new providerSupportsBuiltinWebSearch()
helper. Returns true only for `openai` today; Anthropic
(web_search_20250305), Gemini grounded-search, and OpenRouter
variants can be added later with matching backend translation.
- chat-page.tsx: both model-switch paths (the onChange handler and
the inferenceParams.checkpoint useEffect) set supportsTools to
match the new helper, and force toolsEnabled=false on every
external switch so the Search toggle is opt-in by default.
- chat-adapter.ts: external branch adds enable_tools +
enabled_tools=["web_search"] to the request body when the
toggle is on AND the active provider supports built-in
web-search. Local-model branch is unchanged — it continues to
route the same shorthand through our local tool runtime.
Backend:
- routes/inference.py: forwards payload.enabled_tools to
stream_chat_completion at the proxy site (line 1599).
- external_provider.py: stream_chat_completion gains an
enabled_tools parameter; _stream_openai_responses appends
{"type": "web_search"} to body["tools"] when the list contains
"web_search". Other tools (file_search, code_interpreter,
image_generation, computer_use_preview) are easy follow-ups in
the same block.
Reuses the existing pydantic ChatCompletionRequest.enabled_tools
field, so no schema migrations.
* studio/backend: surface OpenAI server-side web_search in the chat UI
When the user has the chat Search button toggled on and OpenAI's
/v1/responses invokes the built-in web_search tool, _stream_openai_responses
now translates the tool's lifecycle events and citation annotations
into the same _toolEvent shape that local-tool calls use. The result:
the chat UI shows a web_search tool-call card mid-stream, then lists
the cited sources at the end of the message — identical to how local
web_search renders.
SSE event translation:
- response.output_item.added with item.type=web_search_call ->
emit _toolEvent tool_start. Carries item.action.query as args
when OpenAI ships it on the added event.
- response.output_item.done with item.type=web_search_call ->
backfill the query if it only arrives on the done variant. The
existing reasoning branch on the same event is preserved as an
if/elif under a shared isinstance guard.
- response.output_text.annotation.added with type=url_citation ->
collect into the most-recent web_search_call.citations list.
- response.output_text.delta with inline annotations[] (older
API variant) -> same collection path, so both wire shapes work.
- response.completed -> emit _toolEvent tool_end per call with
citations formatted as
Title: <title>\nURL: <url>\nSnippet: <snippet>
blocks joined by `\n---\n`. The frontend's
parseSourcesFromResult already lifts this format into source
content parts at end-of-stream.
- response.incomplete -> close out web_search cards with whatever
citations had landed, so a truncated response does not leave a
perpetually "running" tool card in the UI.
Both reasoning and web_search work simultaneously on the same turn —
the body sends `reasoning: {effort, summary}` and `tools: [{type:
"web_search"}]` independently, and the SSE handler tracks them
through separate channels.
Diagnostic: finally-block logger now reports per stream
web_search_requested - whether the client asked for it
web_search_invocations - how many calls OpenAI actually made
citations - total URLs cited
queries - the search queries the model issued
reasoning_emitted - whether <think> content was streamed
so reports of "I clicked Search and nothing happened" can be triaged
from the backend log without browser devtools.
* studio/backend: fix empty query + per-card '(no sources cited)' on OpenAI web_search
Two display bugs on the OpenAI Responses web_search → chat-UI bridge:
1. Tool cards showed "Searching for ''" — query missing.
OpenAI's response.output_item.added for web_search_call does not
reliably populate action.query across API versions; the canonical
place is output_item.done. The previous code emitted tool_start
at added with empty args and tried to backfill at done, but the
frontend's _toolEvent: tool_start is a one-shot push (no update
mechanism), so the args stayed empty.
Fix: defer both tool_start *and* a placeholder tool_end emission
to output_item.done, where action.query is guaranteed populated.
added now just initialises tracking. Frontend then renders one
card per call with the right "Searching for: <query>" label.
2. Every card showed "(no sources cited)".
The previous code tried to attribute url_citation annotations
to individual web_search_call invocations, but OpenAI's
annotations carry no link back to a specific search call —
they're just URLs the model cited from the aggregated search
pool. With N invocations and M annotations, the previous logic
bucketed all M into the last call and stamped "(no sources
cited)" on the rest.
Fix: collect citations into a single shared all_url_citations
list, dedup by URL. At response.completed (and
response.incomplete) overwrite the *last* web_search_call's
tool_end result with the aggregated Title:/URL:/Snippet:
blocks. The frontend's parseSourcesFromResult already flatMaps
every web_search result, so one non-empty result is enough to
surface the full source-pill set at the message tail. Other
tool cards get an empty result string (no '(no sources)' text).
Diagnostic log unchanged in shape; total_citations now reads
len(all_url_citations) directly.
* studio/chat: split Code and Search pill gates so external models cannot enable Code
The previous wire-up set supportsTools=true for OpenAI external
models to light up the Search pill, but supportsTools also gates the
Code pill, so Code became clickable for OpenAI even though external
providers have no local code execution.
Separate the two gates so each pill reflects what's actually
available:
- chat-runtime-store: new `supportsBuiltinWebSearch: boolean` flag.
Distinct from supportsTools — that one still means "runtime has a
local tool sandbox" (Code, python, our DuckDuckGo web_search).
This one means "the active external provider exposes a server-side
web_search tool we can opt into" (OpenAI's /v1/responses today).
- chat-page model-switch (both code paths): for external models,
supportsTools is now forced to false (no local Code path) and
supportsBuiltinWebSearch follows providerSupportsBuiltinWebSearch.
Local-model paths are unaffected — they only set supportsTools.
- shared-composer: Search pill gates on
`searchDisabled = !modelLoaded || !(supportsTools ||
supportsBuiltinWebSearch)`. Code pill gates on
`codeDisabled = !modelLoaded || !supportsTools` — strictly the
local runtime, so external models keep Code greyed out.
A `toolsDisabled = codeDisabled` alias is left in place for any
later-touched call site that may still reference the old name.
No backend changes — chat-adapter already calls
providerSupportsBuiltinWebSearch directly, independent of the store
flags, so the request shape and the backend translation are
unchanged.
* studio/chat: default external reasoning effort to medium, not the carry-over
When switching to an external model with reasoning support, the effort
dropdown was inheriting whatever value the user had set on a prior
model — frequently "xhigh" left over from a previous Opus/gpt-5
session. That meant every fresh OpenAI/Anthropic selection started at
Extra High, burning tokens unintentionally.
Both model-switch sites in chat-page (the useEffect on
inferenceParams.checkpoint and the onChange callback) now pick
"medium" whenever the new model's level list contains it, instead of
the clamped carry-over. The clamp still fires as a fallback for the
narrow case where a model doesn't expose medium (e.g. gpt-5.3-chat-
latest which only has medium anyway — no change there). Users can
still pick another level explicitly via the Think dropdown.
* studio/chat: also light the Search pill in the welcome-screen composer
There are two composers in the chat feature. shared-composer.tsx
renders inside an active thread, and assistant-ui/thread.tsx has its
own WebSearchToggle / CodeToolsToggle that ship the welcome-screen
"Send a message…" composer (visible before the first user message).
The previous fix split supportsTools and supportsBuiltinWebSearch in
shared-composer but never touched the welcome-screen toggles in
thread.tsx — they both still gated on supportsTools alone, so the
Search pill stayed greyed on the welcome screen even for OpenAI
external models that legitimately support web_search server-side.
Mirror the shared-composer rule in WebSearchToggle:
disabled = !modelLoaded || !(supportsTools || supportsBuiltinWebSearch)
CodeToolsToggle is left as-is — its current
`disabled = !(modelLoaded && supportsTools)` is correct: external
models have no local code-execution sandbox, so Code stays greyed
when supportsTools=false (which is what chat-page now writes for
external selections).
* studio/backend: wire Anthropic server-side web_search end-to-end
Mirrors the OpenAI web_search integration for Anthropic's
web_search_20250305 tool. When the user toggles Search on with an
Anthropic model selected, the request now carries the documented
tool entry:
tools: [{type: "web_search_20250305", name: "web_search",
max_uses: 5}]
on /v1/messages, and the SSE translation surfaces tool cards +
source pills in the chat UI exactly the same way as OpenAI.
stream_chat_completion now forwards enabled_tools into the
Anthropic branch (was only doing this for the OpenAI Responses
branch). _stream_anthropic gains an enabled_tools parameter and
the web_search request-body block plus three additional event
handlers:
- content_block_start with type=server_tool_use, name=web_search:
start tracking a new call. id becomes the tool_call_id.
- content_block_delta with type=input_json_delta inside a
server_tool_use block: buffer the partial_json so we can read
out the search query when the block closes.
- content_block_start with type=web_search_tool_result: capture
the per-call result list (urls + titles) that Anthropic ships
inline.
- content_block_stop: closes whichever block we're inside —
* server_tool_use -> emit _toolEvent: tool_start with the
parsed query as args.
* web_search_tool_result -> emit _toolEvent: tool_end with
Title:/URL: blocks the frontend's parseSourcesFromResult
lifts into source pills.
* thinking block -> existing </think> close.
Unlike OpenAI we get per-call results directly, so no aggregated-
last-call fallback is needed — each tool card carries its own
citations.
Diagnostic log on stream completion now reports
web_search_requested / invocations / total_results / queries,
matching the OpenAI shape.
Frontend providerSupportsBuiltinWebSearch returns true for
'anthropic' as well, so the Search pill lights up on Claude
models the same way it does on OpenAI. The existing chat-adapter
external branch already sends enabled_tools=['web_search'] based
on this helper — no adapter changes needed.
* studio: wire OpenRouter built-in web search via :online model suffix
OpenRouter exposes a universal "add web search to any model" shortcut:
append `:online` to the model id and the gateway runs the search
server-side, streaming citations back as annotations on text deltas.
Documented at https://openrouter.ai/docs/features/web-search
Hook the existing Search toggle into that path:
Backend (external_provider.py, default OAI-compat branch):
- When provider_type == 'openrouter' and enabled_tools contains
'web_search', rewrite body['model']:
openai/gpt-4o -> openai/gpt-4o:online
anthropic/claude-sonnet-4-5:free -> anthropic/claude-sonnet-4-5:online
Any existing `:variant` (`:free`, `:nitro`, etc.) is replaced —
OpenRouter variants are mutually exclusive.
- `openrouter/free` is skipped: it's a meta-router and `:online` is
not a valid suffix on it (the gateway 400s).
- A one-line INFO log fires whenever the rewrite happens so the
diagnostic backend log shows exactly which model id the request
was promoted to.
Frontend (provider-capabilities.ts):
- providerSupportsBuiltinWebSearch now returns true for 'openrouter'
alongside 'openai' and 'anthropic'. The Search pill lights up and
the existing chat-adapter external branch already forwards
enabled_tools=['web_search'] based on this helper — no adapter
changes needed.
No new SSE event handling: OpenRouter does not emit a separate
web_search_call event the way OpenAI/Anthropic do. Citations come
back as text annotations via the existing reasoning_details path
the adapter already parses, so source data flows through without
extra translation. A per-call tool-card UX ("Searching for: …")
would require synthesizing one client-side; deferred to a follow-up
if the bare-citation flow feels too minimal.
* studio: wire Mistral built-in web search connector
Same shape as OpenAI's web_search tool, lives on
/v1/chat/completions instead of /v1/responses. When the chat
Search pill is toggled on with a Mistral model selected, the
backend now appends
{"type": "web_search"}
to body["tools"] before the request goes out. Idempotent —
won't double-append if a future call site adds it first. Models
in the registry allowlist that don't support the connector
(codestral, devstral, ministral, mistral-tiny) will surface a
400 from upstream; the existing default-path error log captures
it. Mistral's docs:
https://docs.mistral.ai/capabilities/agents/connectors/websearch
Frontend providerSupportsBuiltinWebSearch returns true for
'mistral' now, alongside openai / anthropic / openrouter. The
Search pill lights up for Mistral models and the existing
adapter branch already sends enabled_tools=['web_search'] off
this helper — no adapter changes.
No SSE translation yet — Mistral streams citations inline as
text annotations or `references` in the final assistant content,
not as a separate web_search_call event. Citations flow through
to the message body as text; a per-call tool-card UX with
"Searching for: …" indicators is a follow-up if needed.
* studio/backend: fix OpenRouter web_search to use plugins shape + synthesize tool card
Two changes against the actual OpenRouter docs at
https://openrouter.ai/docs/guides/features/plugins/web-search:
Request shape:
The previous commit appended :online to the model id, which works on
concrete model ids but rejects on meta-routers like openrouter/free —
and that's exactly the model the user was testing with, so neither
the request rewrite nor the diagnostic log fired. Switch to the
universal plugins shape:
body["plugins"] = [{"id": "web"}]
Per the docs this is "exactly equivalent" to :online but works on
every model id including openrouter/free and openrouter/auto. No
model suffix manipulation, idempotent if added twice.
Tool-card synthesis:
OpenRouter doesn't emit a structured web_search_call event the way
OpenAI/Anthropic do — citations come back only as `annotations` of
type=url_citation on delta/message objects. To match the chat-UI
tool-card UX the user expects ("Searching for: …" indicator,
source pills at message tail), synthesize the events client-side
in the default OAI-compat stream loop:
- On stream open (after the 200 status check): yield a synthetic
_toolEvent: tool_start with tool_name=web_search, fixed id
"openrouter_web_search". The chat-UI then renders the running
tool card before any text streams.
- During the SSE loop: scan every chunk's choices[].delta and
choices[].message for `annotations: [{type: "url_citation",
url_citation: {url, title, content}}]` entries. Dedup by URL
into a citations list. Handles both the nested-url_citation
shape OpenRouter documents and the flat-on-annotation shape
some upstreams ship.
- On [DONE] (or stream-close without [DONE]): emit synthetic
tool_end carrying the citations as
Title: …\nURL: …\nSnippet: …\n---\n…
blocks the existing parseSourcesFromResult lifts into source
pills at message tail.
Diagnostic log on completion now also reports
web_search_requested + citation count alongside the existing
chosen-model / event-count telemetry.
* studio: drop Mistral built-in web_search — connector lives on Agents API only
Mistral's web_search is exclusively on /v1/agents + /v1/conversations;
sending it on /v1/chat/completions returns
"WebSearchTool connector is not supported". Wiring it would require a
dedicated Agents streaming path. Remove from the frontend capability map
and revert the chat-completions tool injection.
* studio: wire Kimi $web_search builtin via two-call round-trip
Kimi's $web_search lives on /v1/chat/completions but requires a client
round-trip per https://platform.kimi.ai/docs/guide/use-web-search:
the first call returns tool_calls with function.arguments populated;
the caller echoes those arguments back as a role=tool message; the
second call streams the final answer with search results incorporated.
The docs also mandate thinking=disabled while the builtin is active.
Backend: new _stream_kimi_web_search helper dispatched from
stream_chat_completion when provider_type=='kimi' and 'web_search' in
enabled_tools. Buffers tool_calls across deltas, falls back to a plain
stream if the model declines to search, and synthesizes tool_start
(with parsed query) / tool_end (with any url_citation annotations) so
the chat UI's web-search card behaves the same as other providers.
Frontend: kimi added to providerSupportsBuiltinWebSearch so the Search
pill lights up in the composer.
* studio/chat: mutual exclusion of Think + Search on Kimi composer
Kimi's $web_search builtin requires thinking=disabled per
https://platform.kimi.ai/docs/guide/use-web-search, so the two states
cannot coexist. Make the pills mutually exclusive in both composers
(shared and welcome-screen): clicking Search turns Think off; clicking
Think back on turns Search off. Default Think to on when a Kimi model
is selected — k2.6/k2.5 ship with thinking enabled out of the box.
* studio/chat: fix wrong provider var name in onChange branch
selectedProvider, not provider — TS2304 in tsc -b.
* studio/backend: add diagnostics to Kimi $web_search round-trip
Log the actual function.arguments from the first call (so we can see
the model's search query) and the second call's usage.prompt_tokens +
any annotation type names that came through. prompt_tokens spiking
above the input message length is direct proof the server injected
search results into context. annotation_types lets us learn the shape
Kimi uses for citations if/when they emit any.
* studio: per-provider defaults — Anthropic xhigh + Search on, OpenAI high + Search on, Opus 4.7 gains max
Anthropic: Think effort defaults to the highest level the model
supports (xhigh on 4.6/4.7, high on 4.5) and Search starts on, since
the web_search_20250305 tool returns structured citations end-to-end.
OpenAI: Think effort defaults to 'high' (the gpt-5.x reasoning sweet
spot for /v1/responses + web_search) and Search starts on.
Opus 4.7: 'max' added as an effort level above 'xhigh' in both
backend (_ANTHROPIC_THINKING_SPECS) and frontend (ANTHROPIC_REASONING_MODELS).
Kimi diagnostics: emit tool_end immediately after tool_start so the
web-search card transitions to 'complete' before the second-call
answer streams, log first-call args + second-call usage/prompt_tokens
+ any annotation type names, request stream_options.include_usage so
the second call exposes usage in SSE.
* studio/backend: harden Kimi fallback path with HTTPError handler + manual aiter_lines loop
Addresses PR review feedback (#5443): the no-search fallback streaming
path was using `async for response.aiter_lines()` and had no
`httpx.HTTPError` guard around the POST. Switch to the manual
__anext__ loop pattern used elsewhere in this module (avoids the
Python 3.13 + httpcore 1.0.x GeneratorExit propagation issue) and wrap
the whole request in a try/except so network failures surface as a
proper SSE error frame instead of a raw traceback.
* feat: prompt caching frontend for openai/anthropic
* studio/chat: route vLLM provider to /v1/chat/completions, not /v1/responses
vLLM's /v1/responses rebuilds messages through the loaded model's chat
template, which 400s on strict-alternation templates like Gemma 3
("Conversation roles must alternate user/assistant/..."). Stop collapsing
vllm -> openai in the frontend so the backend sees the real provider type
and falls through to the standard chat-completions path. Register vllm as
a hidden entry in PROVIDER_REGISTRY so supports_vision and provider-create
validation work without surfacing it in the cloud-provider dropdown.
* studio/chat: wire prompt caching for OpenAI and Anthropic external providers
Backend half of the prompt_caching toggle that already exists in the chat
settings panel. Scoped to OpenAI cloud (/v1/responses) and Anthropic
(/v1/messages); every other provider plumbs the flag as a no-op.
- Anthropic: attach cache_control={type:ephemeral} to the system block so
the static prefix is reused across turns. Without the marker Anthropic
caches nothing, so this is the only way to make the toggle do real work
on /v1/messages.
- OpenAI: opt into prompt_cache_retention="24h" — same price as the
default in_memory policy per the OpenAI docs, but the cache survives
~24 hours of idle instead of ~5-10 minutes. The model picker is
registry-scoped to gpt-5.x / o3 / gpt-4.5, all of which accept the
parameter (gpt-5.5+ already defaults to "24h" so it's a no-op there).
- Treats `enable_prompt_caching=None` as enabled to match the frontend
default for both providers; pass `false` explicitly to opt out.
* studio/chat: log cache token counts on OpenAI and Anthropic stream completion
Surface cache usage in the existing "stream complete" info logs so
prompt-caching behavior can be verified by tailing the studio backend
log instead of opening the provider dashboard.
- Anthropic: latch usage from message_start (input + cache_creation +
cache_read counts) and message_delta (output_tokens), then include in
the per-request summary. cache_read_input_tokens > 0 confirms the
cache_control marker on the system block is doing its job.
- OpenAI Responses: latch usage from response.completed and
response.incomplete, extract usage.input_tokens_details.cached_tokens
(the /v1/responses field name, not prompt_tokens_details). A non-zero
value on turn N proves prompt_cache_retention="24h" let the prefix
hit the cache instead of being recomputed.
* studio/backend: strip temperature/top_p for Claude 4.7 family
Anthropic Opus 4.7 removed temperature, top_p, and top_k as a launch
breaking change ("Sampling parameters removed" in the 4.7 release notes
at https://platform.claude.com/docs/en/about-claude/models/whats-new-claude-4-7).
Setting any of them to a non-default value returns 400
"<param> is deprecated for this model". The existing guard only handled
top_k; temperature was still being sent unconditionally and is now
breaking opus-4-7 requests.
Rename _ANTHROPIC_TOP_K_DEPRECATED to _ANTHROPIC_4_7_SAMPLING_REMOVED to
reflect the broader scope, omit temperature from the base body on 4.7,
and skip the thinking-mode temperature=1 override on 4.7 (still applied
on 4.5/4.6 where it's required). Existing thinking_translation tests
target 4.5/4.6 / mock the wire so they're unaffected.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/chat: anchor Anthropic prompt cache on the latest message too
A system-only cache_control marker is a no-op when the system prompt is
empty or shorter than Anthropic's ~1024-token cache floor — caching
silently does nothing (both cache_creation and cache_read return 0).
Add a second cache_control breakpoint on the final block of the latest
conversation message so the entire prefix (system + prior turns + new
user turn) becomes eligible for caching. On turn N+1, Anthropic
rehydrates everything up through turn N's marker instead of recomputing
it. Up to 4 breakpoints are allowed per request; we use at most 2
(system + tail). Tail rebuild avoids mutating the caller's content list
so an image-bearing turn still slots cleanly into the cached prefix.
* studio/chat: gate vLLM reasoning toggle on provider config
Add a "This server runs a reasoning model" checkbox on the vLLM
provider config. When off (default), the chat Think pill stays
hidden and no enable_thinking ever reaches vLLM. When on, the
pill renders, per-turn state flows through the existing
enable_thinking plumbing, and the backend proxy lifts it onto
chat_template_kwargs.enable_thinking so vLLM's Jinja template
honours it.
* chore: clean vLLM reasoning-toggle comments
* studio/chat: gate prompt_cache_retention to actual OpenAI cloud requests
Addresses Codex P1 review on _stream_openai_responses. The frontend
only sends enable_prompt_caching for the openai/anthropic UI provider
types, so ollama/llama.cpp/"custom" requests reach this helper with
the flag as None. The previous `is not False` check treated None as
enabled and injected prompt_cache_retention="24h" into every request
including those bound for non-OpenAI servers, which would 400 on
servers that implement /v1/responses but not the retention parameter.
Match the public OpenAI host (api.openai.com) on the client base_url
before adding the field so it only lands on actual OpenAI cloud
requests. Studio's openai picker is already registry-scoped to
gpt-5.x / o3 / gpt-4.5, all of which accept the parameter.
---------
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: wire the chat Search button to OpenAI's built-in web_search tool
When the active model is an OpenAI external provider and the user
clicks the existing Search pill in the composer, the chat-completion
request now carries the unified enable_tools shorthand:
enable_tools: true
enabled_tools: ["web_search"]
The backend's stream_chat_completion threads enabled_tools through
to _stream_openai_responses, which translates it into the Responses
API tool schema:
body["tools"] = [{"type": "web_search"}]
per the OpenAI Responses tool spec
(https://developers.openai.com/api/docs/guides/tools). OpenAI then
runs the search server-side before the model replies; the search-
informed answer streams back through the existing
response.output_text.delta path. web_search_call lifecycle events
are silently ignored for now — sources / status indicators are
follow-up scope.
Frontend:
- provider-capabilities.ts: new providerSupportsBuiltinWebSearch()
helper. Returns true only for `openai` today; Anthropic
(web_search_20250305), Gemini grounded-search, and OpenRouter
variants can be added later with matching backend translation.
- chat-page.tsx: both model-switch paths (the onChange handler and
the inferenceParams.checkpoint useEffect) set supportsTools to
match the new helper, and force toolsEnabled=false on every
external switch so the Search toggle is opt-in by default.
- chat-adapter.ts: external branch adds enable_tools +
enabled_tools=["web_search"] to the request body when the
toggle is on AND the active provider supports built-in
web-search. Local-model branch is unchanged — it continues to
route the same shorthand through our local tool runtime.
Backend:
- routes/inference.py: forwards payload.enabled_tools to
stream_chat_completion at the proxy site (line 1599).
- external_provider.py: stream_chat_completion gains an
enabled_tools parameter; _stream_openai_responses appends
{"type": "web_search"} to body["tools"] when the list contains
"web_search". Other tools (file_search, code_interpreter,
image_generation, computer_use_preview) are easy follow-ups in
the same block.
Reuses the existing pydantic ChatCompletionRequest.enabled_tools
field, so no schema migrations.
* studio/backend: surface OpenAI server-side web_search in the chat UI
When the user has the chat Search button toggled on and OpenAI's
/v1/responses invokes the built-in web_search tool, _stream_openai_responses
now translates the tool's lifecycle events and citation annotations
into the same _toolEvent shape that local-tool calls use. The result:
the chat UI shows a web_search tool-call card mid-stream, then lists
the cited sources at the end of the message — identical to how local
web_search renders.
SSE event translation:
- response.output_item.added with item.type=web_search_call ->
emit _toolEvent tool_start. Carries item.action.query as args
when OpenAI ships it on the added event.
- response.output_item.done with item.type=web_search_call ->
backfill the query if it only arrives on the done variant. The
existing reasoning branch on the same event is preserved as an
if/elif under a shared isinstance guard.
- response.output_text.annotation.added with type=url_citation ->
collect into the most-recent web_search_call.citations list.
- response.output_text.delta with inline annotations[] (older
API variant) -> same collection path, so both wire shapes work.
- response.completed -> emit _toolEvent tool_end per call with
citations formatted as
Title: <title>\nURL: <url>\nSnippet: <snippet>
blocks joined by `\n---\n`. The frontend's
parseSourcesFromResult already lifts this format into source
content parts at end-of-stream.
- response.incomplete -> close out web_search cards with whatever
citations had landed, so a truncated response does not leave a
perpetually "running" tool card in the UI.
Both reasoning and web_search work simultaneously on the same turn —
the body sends `reasoning: {effort, summary}` and `tools: [{type:
"web_search"}]` independently, and the SSE handler tracks them
through separate channels.
Diagnostic: finally-block logger now reports per stream
web_search_requested - whether the client asked for it
web_search_invocations - how many calls OpenAI actually made
citations - total URLs cited
queries - the search queries the model issued
reasoning_emitted - whether <think> content was streamed
so reports of "I clicked Search and nothing happened" can be triaged
from the backend log without browser devtools.
* studio/backend: fix empty query + per-card '(no sources cited)' on OpenAI web_search
Two display bugs on the OpenAI Responses web_search → chat-UI bridge:
1. Tool cards showed "Searching for ''" — query missing.
OpenAI's response.output_item.added for web_search_call does not
reliably populate action.query across API versions; the canonical
place is output_item.done. The previous code emitted tool_start
at added with empty args and tried to backfill at done, but the
frontend's _toolEvent: tool_start is a one-shot push (no update
mechanism), so the args stayed empty.
Fix: defer both tool_start *and* a placeholder tool_end emission
to output_item.done, where action.query is guaranteed populated.
added now just initialises tracking. Frontend then renders one
card per call with the right "Searching for: <query>" label.
2. Every card showed "(no sources cited)".
The previous code tried to attribute url_citation annotations
to individual web_search_call invocations, but OpenAI's
annotations carry no link back to a specific search call —
they're just URLs the model cited from the aggregated search
pool. With N invocations and M annotations, the previous logic
bucketed all M into the last call and stamped "(no sources
cited)" on the rest.
Fix: collect citations into a single shared all_url_citations
list, dedup by URL. At response.completed (and
response.incomplete) overwrite the *last* web_search_call's
tool_end result with the aggregated Title:/URL:/Snippet:
blocks. The frontend's parseSourcesFromResult already flatMaps
every web_search result, so one non-empty result is enough to
surface the full source-pill set at the message tail. Other
tool cards get an empty result string (no '(no sources)' text).
Diagnostic log unchanged in shape; total_citations now reads
len(all_url_citations) directly.
* studio/chat: split Code and Search pill gates so external models cannot enable Code
The previous wire-up set supportsTools=true for OpenAI external
models to light up the Search pill, but supportsTools also gates the
Code pill, so Code became clickable for OpenAI even though external
providers have no local code execution.
Separate the two gates so each pill reflects what's actually
available:
- chat-runtime-store: new `supportsBuiltinWebSearch: boolean` flag.
Distinct from supportsTools — that one still means "runtime has a
local tool sandbox" (Code, python, our DuckDuckGo web_search).
This one means "the active external provider exposes a server-side
web_search tool we can opt into" (OpenAI's /v1/responses today).
- chat-page model-switch (both code paths): for external models,
supportsTools is now forced to false (no local Code path) and
supportsBuiltinWebSearch follows providerSupportsBuiltinWebSearch.
Local-model paths are unaffected — they only set supportsTools.
- shared-composer: Search pill gates on
`searchDisabled = !modelLoaded || !(supportsTools ||
supportsBuiltinWebSearch)`. Code pill gates on
`codeDisabled = !modelLoaded || !supportsTools` — strictly the
local runtime, so external models keep Code greyed out.
A `toolsDisabled = codeDisabled` alias is left in place for any
later-touched call site that may still reference the old name.
No backend changes — chat-adapter already calls
providerSupportsBuiltinWebSearch directly, independent of the store
flags, so the request shape and the backend translation are
unchanged.
* studio/chat: default external reasoning effort to medium, not the carry-over
When switching to an external model with reasoning support, the effort
dropdown was inheriting whatever value the user had set on a prior
model — frequently "xhigh" left over from a previous Opus/gpt-5
session. That meant every fresh OpenAI/Anthropic selection started at
Extra High, burning tokens unintentionally.
Both model-switch sites in chat-page (the useEffect on
inferenceParams.checkpoint and the onChange callback) now pick
"medium" whenever the new model's level list contains it, instead of
the clamped carry-over. The clamp still fires as a fallback for the
narrow case where a model doesn't expose medium (e.g. gpt-5.3-chat-
latest which only has medium anyway — no change there). Users can
still pick another level explicitly via the Think dropdown.
* studio/chat: also light the Search pill in the welcome-screen composer
There are two composers in the chat feature. shared-composer.tsx
renders inside an active thread, and assistant-ui/thread.tsx has its
own WebSearchToggle / CodeToolsToggle that ship the welcome-screen
"Send a message…" composer (visible before the first user message).
The previous fix split supportsTools and supportsBuiltinWebSearch in
shared-composer but never touched the welcome-screen toggles in
thread.tsx — they both still gated on supportsTools alone, so the
Search pill stayed greyed on the welcome screen even for OpenAI
external models that legitimately support web_search server-side.
Mirror the shared-composer rule in WebSearchToggle:
disabled = !modelLoaded || !(supportsTools || supportsBuiltinWebSearch)
CodeToolsToggle is left as-is — its current
`disabled = !(modelLoaded && supportsTools)` is correct: external
models have no local code-execution sandbox, so Code stays greyed
when supportsTools=false (which is what chat-page now writes for
external selections).
* studio/backend: wire Anthropic server-side web_search end-to-end
Mirrors the OpenAI web_search integration for Anthropic's
web_search_20250305 tool. When the user toggles Search on with an
Anthropic model selected, the request now carries the documented
tool entry:
tools: [{type: "web_search_20250305", name: "web_search",
max_uses: 5}]
on /v1/messages, and the SSE translation surfaces tool cards +
source pills in the chat UI exactly the same way as OpenAI.
stream_chat_completion now forwards enabled_tools into the
Anthropic branch (was only doing this for the OpenAI Responses
branch). _stream_anthropic gains an enabled_tools parameter and
the web_search request-body block plus three additional event
handlers:
- content_block_start with type=server_tool_use, name=web_search:
start tracking a new call. id becomes the tool_call_id.
- content_block_delta with type=input_json_delta inside a
server_tool_use block: buffer the partial_json so we can read
out the search query when the block closes.
- content_block_start with type=web_search_tool_result: capture
the per-call result list (urls + titles) that Anthropic ships
inline.
- content_block_stop: closes whichever block we're inside —
* server_tool_use -> emit _toolEvent: tool_start with the
parsed query as args.
* web_search_tool_result -> emit _toolEvent: tool_end with
Title:/URL: blocks the frontend's parseSourcesFromResult
lifts into source pills.
* thinking block -> existing </think> close.
Unlike OpenAI we get per-call results directly, so no aggregated-
last-call fallback is needed — each tool card carries its own
citations.
Diagnostic log on stream completion now reports
web_search_requested / invocations / total_results / queries,
matching the OpenAI shape.
Frontend providerSupportsBuiltinWebSearch returns true for
'anthropic' as well, so the Search pill lights up on Claude
models the same way it does on OpenAI. The existing chat-adapter
external branch already sends enabled_tools=['web_search'] based
on this helper — no adapter changes needed.
* studio: wire OpenRouter built-in web search via :online model suffix
OpenRouter exposes a universal "add web search to any model" shortcut:
append `:online` to the model id and the gateway runs the search
server-side, streaming citations back as annotations on text deltas.
Documented at https://openrouter.ai/docs/features/web-search
Hook the existing Search toggle into that path:
Backend (external_provider.py, default OAI-compat branch):
- When provider_type == 'openrouter' and enabled_tools contains
'web_search', rewrite body['model']:
openai/gpt-4o -> openai/gpt-4o:online
anthropic/claude-sonnet-4-5:free -> anthropic/claude-sonnet-4-5:online
Any existing `:variant` (`:free`, `:nitro`, etc.) is replaced —
OpenRouter variants are mutually exclusive.
- `openrouter/free` is skipped: it's a meta-router and `:online` is
not a valid suffix on it (the gateway 400s).
- A one-line INFO log fires whenever the rewrite happens so the
diagnostic backend log shows exactly which model id the request
was promoted to.
Frontend (provider-capabilities.ts):
- providerSupportsBuiltinWebSearch now returns true for 'openrouter'
alongside 'openai' and 'anthropic'. The Search pill lights up and
the existing chat-adapter external branch already forwards
enabled_tools=['web_search'] based on this helper — no adapter
changes needed.
No new SSE event handling: OpenRouter does not emit a separate
web_search_call event the way OpenAI/Anthropic do. Citations come
back as text annotations via the existing reasoning_details path
the adapter already parses, so source data flows through without
extra translation. A per-call tool-card UX ("Searching for: …")
would require synthesizing one client-side; deferred to a follow-up
if the bare-citation flow feels too minimal.
* studio: wire Mistral built-in web search connector
Same shape as OpenAI's web_search tool, lives on
/v1/chat/completions instead of /v1/responses. When the chat
Search pill is toggled on with a Mistral model selected, the
backend now appends
{"type": "web_search"}
to body["tools"] before the request goes out. Idempotent —
won't double-append if a future call site adds it first. Models
in the registry allowlist that don't support the connector
(codestral, devstral, ministral, mistral-tiny) will surface a
400 from upstream; the existing default-path error log captures
it. Mistral's docs:
https://docs.mistral.ai/capabilities/agents/connectors/websearch
Frontend providerSupportsBuiltinWebSearch returns true for
'mistral' now, alongside openai / anthropic / openrouter. The
Search pill lights up for Mistral models and the existing
adapter branch already sends enabled_tools=['web_search'] off
this helper — no adapter changes.
No SSE translation yet — Mistral streams citations inline as
text annotations or `references` in the final assistant content,
not as a separate web_search_call event. Citations flow through
to the message body as text; a per-call tool-card UX with
"Searching for: …" indicators is a follow-up if needed.
* studio/backend: fix OpenRouter web_search to use plugins shape + synthesize tool card
Two changes against the actual OpenRouter docs at
https://openrouter.ai/docs/guides/features/plugins/web-search:
Request shape:
The previous commit appended :online to the model id, which works on
concrete model ids but rejects on meta-routers like openrouter/free —
and that's exactly the model the user was testing with, so neither
the request rewrite nor the diagnostic log fired. Switch to the
universal plugins shape:
body["plugins"] = [{"id": "web"}]
Per the docs this is "exactly equivalent" to :online but works on
every model id including openrouter/free and openrouter/auto. No
model suffix manipulation, idempotent if added twice.
Tool-card synthesis:
OpenRouter doesn't emit a structured web_search_call event the way
OpenAI/Anthropic do — citations come back only as `annotations` of
type=url_citation on delta/message objects. To match the chat-UI
tool-card UX the user expects ("Searching for: …" indicator,
source pills at message tail), synthesize the events client-side
in the default OAI-compat stream loop:
- On stream open (after the 200 status check): yield a synthetic
_toolEvent: tool_start with tool_name=web_search, fixed id
"openrouter_web_search". The chat-UI then renders the running
tool card before any text streams.
- During the SSE loop: scan every chunk's choices[].delta and
choices[].message for `annotations: [{type: "url_citation",
url_citation: {url, title, content}}]` entries. Dedup by URL
into a citations list. Handles both the nested-url_citation
shape OpenRouter documents and the flat-on-annotation shape
some upstreams ship.
- On [DONE] (or stream-close without [DONE]): emit synthetic
tool_end carrying the citations as
Title: …\nURL: …\nSnippet: …\n---\n…
blocks the existing parseSourcesFromResult lifts into source
pills at message tail.
Diagnostic log on completion now also reports
web_search_requested + citation count alongside the existing
chosen-model / event-count telemetry.
* studio: drop Mistral built-in web_search — connector lives on Agents API only
Mistral's web_search is exclusively on /v1/agents + /v1/conversations;
sending it on /v1/chat/completions returns
"WebSearchTool connector is not supported". Wiring it would require a
dedicated Agents streaming path. Remove from the frontend capability map
and revert the chat-completions tool injection.
* studio: wire Kimi $web_search builtin via two-call round-trip
Kimi's $web_search lives on /v1/chat/completions but requires a client
round-trip per https://platform.kimi.ai/docs/guide/use-web-search:
the first call returns tool_calls with function.arguments populated;
the caller echoes those arguments back as a role=tool message; the
second call streams the final answer with search results incorporated.
The docs also mandate thinking=disabled while the builtin is active.
Backend: new _stream_kimi_web_search helper dispatched from
stream_chat_completion when provider_type=='kimi' and 'web_search' in
enabled_tools. Buffers tool_calls across deltas, falls back to a plain
stream if the model declines to search, and synthesizes tool_start
(with parsed query) / tool_end (with any url_citation annotations) so
the chat UI's web-search card behaves the same as other providers.
Frontend: kimi added to providerSupportsBuiltinWebSearch so the Search
pill lights up in the composer.
* studio/chat: mutual exclusion of Think + Search on Kimi composer
Kimi's $web_search builtin requires thinking=disabled per
https://platform.kimi.ai/docs/guide/use-web-search, so the two states
cannot coexist. Make the pills mutually exclusive in both composers
(shared and welcome-screen): clicking Search turns Think off; clicking
Think back on turns Search off. Default Think to on when a Kimi model
is selected — k2.6/k2.5 ship with thinking enabled out of the box.
* studio/chat: fix wrong provider var name in onChange branch
selectedProvider, not provider — TS2304 in tsc -b.
* studio/backend: add diagnostics to Kimi $web_search round-trip
Log the actual function.arguments from the first call (so we can see
the model's search query) and the second call's usage.prompt_tokens +
any annotation type names that came through. prompt_tokens spiking
above the input message length is direct proof the server injected
search results into context. annotation_types lets us learn the shape
Kimi uses for citations if/when they emit any.
* studio: per-provider defaults — Anthropic xhigh + Search on, OpenAI high + Search on, Opus 4.7 gains max
Anthropic: Think effort defaults to the highest level the model
supports (xhigh on 4.6/4.7, high on 4.5) and Search starts on, since
the web_search_20250305 tool returns structured citations end-to-end.
OpenAI: Think effort defaults to 'high' (the gpt-5.x reasoning sweet
spot for /v1/responses + web_search) and Search starts on.
Opus 4.7: 'max' added as an effort level above 'xhigh' in both
backend (_ANTHROPIC_THINKING_SPECS) and frontend (ANTHROPIC_REASONING_MODELS).
Kimi diagnostics: emit tool_end immediately after tool_start so the
web-search card transitions to 'complete' before the second-call
answer streams, log first-call args + second-call usage/prompt_tokens
+ any annotation type names, request stream_options.include_usage so
the second call exposes usage in SSE.
* studio/backend: harden Kimi fallback path with HTTPError handler + manual aiter_lines loop
Addresses PR review feedback (#5443): the no-search fallback streaming
path was using `async for response.aiter_lines()` and had no
`httpx.HTTPError` guard around the POST. Switch to the manual
__anext__ loop pattern used elsewhere in this module (avoids the
Python 3.13 + httpcore 1.0.x GeneratorExit propagation issue) and wrap
the whole request in a try/except so network failures surface as a
proper SSE error frame instead of a raw traceback.
* fix(studio/mmproj): block cross-family projectors in flat local GGUF dirs (#5347)
When a flat local GGUF directory holds several unrelated models with their
own mmproj siblings, detect_mmproj_file() returned the first projector it
walked into. For the layout reported in #5347 (Qwen weights + a Gemma
mmproj in the same dir) that meant llama-server was launched with
--mmproj pointing at the Gemma projector, which fails to load and surfaces
as a confusing crash.
Disambiguation rules:
- Drop candidates whose family token (qwen/gemma/llama/mistral/phi/...)
disagrees with the model's family. Candidates with no recognised
family token (e.g. the HF-convention 'mmproj-F16.gguf') are kept.
- Among same-family candidates, prefer the one whose stem shares the
longest prefix with the model (Qwen3.5-9B mmproj beats Qwen3.5-35B
mmproj for a Qwen3.5-9B model).
- If every candidate is dropped, return None — better than attaching
a wrong projector and getting a server-launch failure.
Tests cover the cross-family block, multi-candidate prefix tie-break,
HF-convention 'mmproj-F16.gguf', unrecognised families, and the
existing search_root walk.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/mmproj: word-bounded family match, expanded token list, launcher guard
Tighten the family-token detector to match only on word boundaries so
substring collisions stop tagging false families: phi no longer matches
sapphire, yi no longer matches yip, mimo no longer matches mimosa, and
mistral does not bleed into ministral/magistral/devstral. Pick the token
whose first occurrence is leftmost in the filename rather than the first
hit in tuple order, so merge models disambiguate predictably (llama-phi
tags llama; phi-llama tags phi).
Expand _MODEL_FAMILY_TOKENS with the families an audit of the unsloth
HF org turned up that the previous list missed: devstral, ministral,
magistral (Mistral-derivative naming), nemotron, kimi, nanonets, cosmos,
mimo, apriel, lfm. Without these, a flat local GGUF directory containing
one of these weights plus an unrelated renamed projector still hit the
original #5347 failure.
Add mmproj_matches_model_family() and call it at the llama-server launch
site in core/inference/llama_cpp.py. detect_mmproj_file already drops
cross-family candidates at discovery time, but mmproj_path can also reach
the launcher via config injection or future overrides; this guard keeps
those paths from silently loading a known-wrong projector.
Tests: 12 new cases covering substring rejection, leftmost-position
selection, new family tokens, a new flat-dir Nemotron + Gemma rejection
case, and the launcher-level guard. All 21 detect_mmproj_file tests and
the existing 106 llama_cpp tests pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/mmproj: pair via GGUF general.* metadata, not just filenames
Real Unsloth vision GGUFs carry rich identity metadata that has been
ignored by the discovery path. Every projector under the unsloth org
has general.type='mmproj' plus general.base_model.0.repo_url pointing
at the same upstream HF repo as its weight, and the equivalent
basename, base_model.0.name, and base_model.0.organization fields. A
flat-dir mismatch is therefore decidable from the headers alone, no
matter how the user has renamed the files.
Add utils/models/gguf_metadata.py with read_gguf_general_metadata():
a fast (~30 ms) header walk that pulls only the general.* string
fields and skips everything else, cached by (resolved path, mtime_ns,
size). Mirrors the parser shape already used by
LlamaCppBackend._read_gguf_metadata so the format handling is
consistent.
is_mmproj_by_metadata() returns True/False/None from general.type,
and pairing_score() returns 100 for an exact base_model URL match,
80 for basename plus organization match, 60 for basename only, -1
for definitive metadata disagreement, and 0 when neither side has
enough metadata to decide.
Rewire detect_mmproj_file() to a two-stage selector:
1. Detect projectors via metadata (general.type) when present, else
fall back to the filename substring heuristic. This recovers
headerless projectors AND projectors whose name does not contain
'mmproj' but whose header advertises one.
2. Score each candidate against the weight via pairing_score. Drop
candidates with score -1 (definitive metadata disagreement). For
candidates with score 0 (no usable metadata) fall back to the
existing filename family-token check, dropping recognised-family
mismatches. Pick the survivor with the highest (score,
longest_prefix, -len(stem)) tuple, so a metadata URL match
always wins over a filename-prefix match.
Tests: 16 new cases. tests/test_gguf_metadata.py covers the parser
(missing file, non-GGUF, string extraction, walking past arrays and
uint32s, cache invalidation by mtime/size) and the score helpers.
tests/test_detect_mmproj_file.py adds end-to-end cases that synthesise
real on-disk GGUF headers: URL match wins over a longer-prefix
sibling, URL mismatch returns None even when filenames match, a
projector named 'vision-projector.gguf' is still discovered via
general.type, and a 100-score header match outranks a near-perfect
filename prefix on a headerless candidate.
All 75 tests across detect_mmproj_file, gguf_metadata, llama_cpp
load progress, cached gguf routes, trained model scan, and vision
cache pass.
* studio/mmproj: shorten comments and docstrings across the #5347 changes
Trim verbose explanations to one-line statements of intent. The
behaviour is unchanged: 161 tests across detect_mmproj_file,
gguf_metadata, llama_cpp_load_progress (+ matrix), llama_server_args,
llama_cpp_cache_aware_disk_check, trained_model_scan, and vision_cache
all pass.
* studio/mmproj: shorten remaining detect_mmproj_file body comments
Trim the docstring and the dir-walking block comments inside
detect_mmproj_file to one-liners. Behaviour unchanged; 44 mmproj +
gguf_metadata + llama_cpp_load_progress tests pass.
* studio/mmproj: cap gguf_metadata cache below ceiling on every insert
The eviction branch popped exactly one entry when len >= max, so the
cache size could only converge to the cap when entries were added
slowly enough for natural growth. After a sandbox sim that reduced
the cap mid-run, len stayed above the cap because each insert popped
one and added one. Switch to a while loop so we evict until len is
strictly below the cap before inserting. Steady-state behaviour at
the default 4096 ceiling is unchanged.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* mlx fixes
* Fix studio integration, local dataset files, chat templates without the torch gpu imports
* pass grad norm in mlx worker
* fix(studio): pass MLX grad clipping settings
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* mlx: update grad value
* fix(mlx): address ci and clipping review
* fix backward compatibility and CI tests
* unsloth local is mlx function
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* dont reference runtime
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio mlx: hardcode value clipping, drop max_grad_value from frontend
Simplifies the MLX grad-clipping plumbing now that we are standardising on
elementwise value clipping at [-5, 5] for the compiled MLX path and norm
clipping disabled. The MLX worker no longer reads max_grad_norm /
max_grad_value from the request; both are pinned in one place. Frontend
stops sending the field at all, and the TypeScript request type drops it
to match. Non-MLX (CUDA/AMD/Intel) is untouched and continues to pick up
HF TrainingArguments' default max_grad_norm = 1.0.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* studio: add external provider support for chat inference
Adds the ability to connect to OpenAI, Mistral, Google, Cohere, Together,
Fireworks, and Perplexity from the Studio chat interface.
- Provider configs stored in SQLite (no API keys persisted)
- RSA-2048 key pair generated at startup for client-side key encryption
- httpx proxy client streams SSE responses in OpenAI-compatible format
- New /api/providers routes: registry, CRUD, test, models
- /v1/chat/completions routes to external provider when provider fields present
- Integration test suite covering CRUD, connection, model listing, and inference
- Frontend spec doc with full API contract
* remove frontend spec doc from branch
* fix auth fixture: handle forced password change on fresh install
* fix tests: default port 8000, allow 400 for no-model-loaded
* fix: update Cohere models to current (command-r retired Sept 2025)
* feat: add OpenRouter as 8th provider
* feat: add native Anthropic provider with Messages API translation
* fix: correct Anthropic base URL and drop top_p (conflicts with temperature)
* feat: add DeepSeek provider (deepseek-chat, deepseek-reasoner)
* feat: rename google -> gemini, refresh model list to 2.5 series
* feat: remove together, fireworks, perplexity providers
* feat: multimodal image support for external providers
- Add _build_external_messages() that preserves image_url parts for
vision-capable providers instead of stripping them
- Update _proxy_to_external_provider() to use new helper
- Translate image_url content parts to Anthropic native image format
in _stream_anthropic()
- Add TestVisionInference pytest class (1x1 PNG smoke test)
* test: use sloth photo URL for vision test, add Anthropic remote URL support
* fix: update Mistral model to mistral-small-2506
* update mistral default model to mistral-large-2512
* fix gemini vision test: download image as base64 data URI instead of remote URL
* add gemini-3-flash-preview as default gemini model
* fix gemini truncated reply (max_tokens 16->64) and suppress GeneratorExit on client disconnect
* increase vision test max_tokens to 215
* fix GeneratorExit: aclose stream generator before closing httpx client
* fix httpcore GeneratorExit: explicitly aclose aiter_lines before response closes
* fix duplicate [DONE] and suppress httpcore RuntimeError on Python 3.13 asyncgen cleanup
* fix: call response.aclose() before lines_gen.aclose() to prevent httpcore RuntimeError on Python 3.13
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Potential fix for code scanning alert no. 36: Clear-text logging of sensitive information
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* review: add comments for manual iteration rationale, mask password in test print, clarify Anthropic URL/models support
* perf: use shared module-level httpx client for connection pooling across requests
* studio: add API provider UI and integrate wiring (#4737)
* feat: expose external models in selector and chat settings
* feat(chat): wire external providers to backend + RSA key flow
- Fetch registry/configs; create/update/delete saved providers
- Encrypt API keys (Web Crypto RSA-OAEP) for test/models/chat
- External model selection + chat payload (provider_id/type, external_model, encrypted key, optional base URL)
- Local storage for keys + provider list; small UX/copy and guardrails
* add missing providers-api.ts file by Imagineer99
* fix: address PR review comments — system prompt visibility, retry loop, test logging
* feat(studio): encrypt external provider API keys at rest in localStorage
API keys for external providers (OpenAI, Mistral, etc.) were stored as
plaintext in localStorage, vulnerable to browser extensions and XSS.
Add password-derived AES-256-GCM encryption: on login the user's password
is used via PBKDF2 (100k iterations, SHA-256) to derive an in-memory
encryption key. API keys are encrypted before writing to localStorage and
decrypted on read. The derived key is never persisted — cleared on logout,
re-derived on next login.
Legacy plaintext keys are transparently migrated on first access. Password
changes re-encrypt all stored keys. No backend changes required — the
existing RSA-OAEP transit encryption is unaffected.
* fix: cast PBKDF2 salt to BufferSource for strict TypeScript lib types
* fix: persist session password in sessionStorage to survive page refreshes
* feat(studio): preserve image parts in external provider chat requests
toOpenAIMessage() now returns multimodal content arrays (OpenAI vision
format) when messages contain images, instead of always flattening to
plain text. This enables vision-capable external providers (OpenAI,
Gemini, Anthropic, etc.) to receive user images. The backend already
handles image_url content parts in _build_external_messages().
* studio: fix external models selectable in chat-only mode (#4779)
* fix: external models selectable in chat-only mode
* fix: model selector tabs default to active model kind
* Studio: API external provider registry + curated catalogs (HF/OpenRouter) and chat UX (#4787)
* fix: external models selectable in chat-only mode
* fix: model selector tabs default to active model kind
* feat(studio): expand provider registry, curated catalogs, and chat UX
- Add Hugging Face, Kimi, Qwen; remove Cohere; reorder registry
- model_list_mode curated for HF/OpenRouter; lightweight /models check
- API returns default models for curated providers; expose model_list_mode
- Frontend: provider logos in model picker, providerType on external models
- Chat providers dialog: curated vs remote flows, motion polish
- Thread: LayoutGroup + composer motion alignment with app easing
* fix(studio): disable Anthropic tool-calling flag and preselect curated defaults
* feat(studio): add external provider logos and ApiProviderLogo helper
* Studio: Polish API Providers dialog (#4899)
* fix: lower verbage in API providers page
* fix: fix(studio): tune API Providers dialog width with rem-based responsive caps
* feat: add custom provider support (#4902)
* fix: replace crypto.subtle with node-forge for HTTP compatibility
crypto.subtle is only available in secure contexts (HTTPS/localhost),
which breaks provider API key encryption when Studio is accessed over
plain HTTP on remote GPU VMs. Switch to node-forge for RSA-OAEP and
AES-256-GCM operations — same algorithms, works on any origin.
* fix: store provider API keys as plaintext in localStorage
Drop AES-256-GCM at-rest encryption for provider API keys. The
session-password-derived encryption broke on auto-login via refresh
token (password never captured), causing keys to silently vanish.
API keys are still RSA-encrypted in transit via node-forge. At-rest
encryption in localStorage added no real security since the
decryption key also had to live client-side.
Removes crypto-storage.ts, session password plumbing, and
reEncryptAllKeys.
* fix: use max_completion_tokens for OpenAI provider
Newer OpenAI models (gpt-4o, gpt-5.x) reject the max_tokens param
and require max_completion_tokens instead. Other providers still use
max_tokens.
* fix: skip empty assistant messages in external provider requests
Some providers (Mistral) reject assistant messages with empty content.
Filter them out when building the message list for external providers.
* Update model-selector.tsx
* Update model-selector.tsx
* Update model-selector.tsx
* Update chat-adapter.ts
* Update chat-adapter.ts
* Update chat-page.tsx
* Update chat-settings-sheet.tsx
* Update chat-settings-sheet.tsx
* Update chat-settings-sheet.tsx
* Update chat-providers-dialog.tsx
* feat: polish providers settings form UI
* style: polish provider row icon sizing and alignment
* style: stabilize provider layout
* style: add provider API key visibility toggle
* fix: add provider render on empty list
* studio/frontend: sync package-lock.json with package.json
npm ci was failing because node-forge and @types/node-forge were
declared in package.json but missing from the lockfile. Ran
npm install to regenerate.
* studio/backend: fix backend CI failures for providers router
- test_desktop_auth: include providers_router in the routes stub so
studio.backend.main imports cleanly under the monkeypatched module
- test_providers_api: skip the whole module when STUDIO_TEST_PASSWORD
is unset (it is an integration test against a live Studio server,
same shape as the already-ignored test_studio_api.py)
* studio/chat: drive ChatSettingsPanel from a per-provider capability map
Replace the binary isExternalModel toggle in the sampling section with a
provider-aware capability map. Each external provider type advertises
which of top_k / min_p / repetition_penalty / presence_penalty its
chat-completions API actually accepts, so the panel only renders the
knobs that map onto the active provider's request body.
Anthropic now exposes top_k; DeepSeek hides presence_penalty (deprecated
in their docs); OpenRouter and custom providers continue to show every
knob (OpenRouter drops unsupported server-side, custom assumes
OpenAI-compat or a permissive vLLM/Ollama backend). Local models are
unaffected — null capabilities means 'show everything'.
chat-adapter.ts now forwards top_k / presence_penalty to the external
proxy only when the active provider's capabilities permit it, so the
request body matches what the UI shows.
* studio/backend: forward top_k to Anthropic; filter OpenAI model list
Two paired changes so the frontend capability map has matching backend
behaviour:
1. ExternalProviderClient.stream_chat_completion now accepts top_k and
forwards it to the Anthropic Messages body. OpenAI-compat providers
(which all reject unknown sampling params) still receive only the
fields they document. The proxy route in routes/inference.py passes
payload.top_k through, so a UI request with top_k actually reaches
Anthropic instead of being silently dropped at the boundary.
2. PROVIDER_REGISTRY['openai'] gains a model_id_allowlist regex that
scopes the /models picker to current-gen ids (gpt-5.5 / gpt-5.4 /
gpt-5.3 / gpt-4.5 / o3 families). The remote /v1/models listing
otherwise returns dozens of historical snapshots, fine-tunes and
non-chat models (embeddings, TTS, image, moderation) that we never
want in the chat UI. default_models is refreshed to match.
* studio/chat: relax presence_penalty to optional on OpenAIChatCompletionsRequest
Followup to 1fbf445a — chat-adapter now omits presence_penalty for
providers that do not accept it (Anthropic / DeepSeek), but the
request type still required it as a non-optional number, breaking
tsc. The backend pydantic model already defaults presence_penalty
to 0, so making it optional client-side matches reality.
* studio/backend: route OpenAI traffic through /v1/responses
OpenAI's new flagship models (gpt-5.x) return 404 'This is not a chat
model' on /v1/chat/completions and are only reachable via /v1/responses.
Add a dedicated _stream_openai_responses path in ExternalProviderClient
that:
- Translates outbound messages into the Responses shape: system messages
are folded into the top-level 'instructions' field, user/assistant
messages become {role, content} items with input_text / input_image
content parts (data URLs and https URLs both pass through).
- Drops presence_penalty / top_k / frequency_penalty, none of which the
Responses contract accepts.
- Translates inbound SSE events back into OpenAI Chat Completions
chunks so the frontend keeps a single SSE shape:
response.output_text.delta -> delta chunk with content
response.completed -> chunk with finish_reason='stop'
response.incomplete -> chunk with finish_reason='length'
response.failed / error -> propagated error SSE line
Stream terminates with data: [DONE] (Responses emits this verbatim).
stream_chat_completion dispatches all provider_type='openai' calls to
this path; other OpenAI-compatible providers (mistral, gemini, etc.)
continue to use /v1/chat/completions.
Frontend provider-capabilities map updated to hide presence_penalty for
OpenAI in the chat settings panel, matching the new request contract.
Includes unit coverage in tests/test_openai_responses_translation.py
exercising the request body translation, image-part rewriting, and
SSE-to-chat-completions translation via httpx.MockTransport.
* studio/chat: clamp external max_tokens to 32k to stay within provider caps
The chat settings slider already capped maxTokens at 32768 for external
models, but a value persisted from a prior local-model session (where
the cap can be 128k+) was sent verbatim to the provider — Claude Opus
returns 'max_tokens: 131072 > 128000' on requests like that, and other
providers have stricter limits still.
Expose EXTERNAL_MAX_OUTPUT_TOKENS from provider-capabilities (32k) and
use it both for the slider max and as the clamp inside chat-adapter's
external-request body. 32k sits below the tightest declared output
limit across the providers we ship and well above what a typical chat
reply needs; the local-model path is unaffected.
* studio: drop temperature/top_p for OpenAI reasoning models
gpt-5.x / o3 / gpt-4.5 are reasoning-class models served via
/v1/responses, and reject temperature and top_p with
'Unsupported parameter' 400s. The OpenAI registry allowlist already
scopes the picker to those families, so neither knob ever applies on
this branch.
- external_provider._stream_openai_responses no longer puts
temperature or top_p in the request body (kept on the method
signature for API symmetry with the other stream methods).
- ProviderCapabilities gains temperature/topP flags; OpenAI sets both
to false. ChatSettingsPanel hides the sliders for OpenAI so the user
does not see inert controls.
- chat-adapter omits temperature/top_p from the external request body
when the active provider does not advertise them.
- OpenAIChatCompletionsRequest type marks both as optional, matching
the new chat-adapter shape.
- test_responses_request_body_uses_input_and_instructions: assertions
flipped to confirm temperature / top_p are absent from the body.
* studio: stop forwarding top_k to Anthropic
Claude 4.x (Opus / Sonnet / Haiku 4.x) returns 400 'top_k is
deprecated for this model' on any request that includes top_k. It
was always optional on the older 3.x line, so dropping it
unconditionally for every Anthropic call is the simplest path —
no per-model gate to maintain.
- external_provider._stream_anthropic no longer adds top_k to the
Messages body (kept on the method signature for API symmetry).
- provider-capabilities sets anthropic.topK = false so the chat
settings panel hides the Top K slider for Anthropic providers
and chat-adapter does not send top_k in the external request.
* studio: gate Anthropic top_k drop to Claude 4.7 only
Previous commit (b5aa6ffd) dropped top_k for every Anthropic call,
but only Claude 4.7 (Opus/Sonnet/Haiku) actually rejects it. 4.6, 4.5,
and the 3.x line still accept top_k and use it as documented.
Backend: _stream_anthropic matches the model id against
^claude-(opus|sonnet|haiku)-4-7(-|.|$) and only strips top_k when it
hits. Every other Claude generation continues to receive the value
from the chat settings panel.
Frontend: anthropic.topK is restored to true so the Top K slider is
visible again — the backend handles the per-model drop, and the
4.7 case is silent (request still succeeds without top_k).
* chore: hide dated openai models in provider select
* studio/providers: apply model_id_denylist when listing remote models
The OpenAI registry entry gained a model_id_denylist regex matching
dated snapshot ids (-YYYY-MM-DD) in 048d73bf, but the list-models
route was never consulting it, so the snapshots still showed up
alongside their canonical ids (gpt-5.5 and gpt-5.5-2026-04-23 both
listed). Apply the denylist with .search() right after the allowlist
filter so dated entries are dropped before the response is built.
* studio/chat: seed registry default_models for remote providers in picker
The Anthropic provider runs in remote model-list mode, so the picker
started with an empty availableModels until the user clicked
'Load Models'. If that /api/providers/models call fails (e.g. the
known transient decryption error during key rotation), the user sees
no models at all — claude-haiku-4-5 in particular was missing from
the dialog even though it is seeded in the registry.
Always pre-populate availableModels with the registry's default_models
when a provider type is selected (curated and remote alike), and have
loadModels() return the union of defaults + the live /models response
so registry-seeded ids are reachable regardless of what the provider's
endpoint returns or whether the call succeeds at all.
* studio/backend: diagnostic logging on provider key decryption
Decryption failures currently log just 'Failed to decrypt API key:
Decryption failed', which leaves no way to tell whether the cause is
a stale public key in the browser, a corrupted ciphertext, an
unexpected exception class, or a server-side keypair rotation. That's
the gap the next reproduction needs to close.
- key_exchange now publishes a short SHA256 fingerprint of the public
key PEM. init_key_pair logs the fingerprint on generation and warns
if it is ever called a second time (re-init silently invalidates
every browser that cached the previous public key).
- decrypt_api_key wraps both the base64 decode and the RSA decrypt
in dedicated try/excepts that log exception type, ciphertext byte
length (RSA-2048 should be exactly 256), input string length, and
the current public-key fingerprint.
- GET /api/providers/public-key returns the fingerprint alongside the
PEM so the frontend can correlate a future encrypt-time fingerprint
against the decrypt-time fingerprint and prove or rule out a
keypair rotation as the cause.
- The /test and /models route-level decrypt warnings now include the
exception class name (alongside the existing message).
* studio/providers: hide dated Anthropic snapshots from the model picker
Anthropic's /v1/models returns dated snapshot ids (e.g.
claude-3-5-sonnet-20241022, claude-3-5-haiku-20241022) alongside
the canonical names users actually want to pick. Same intent as
the OpenAI denylist added in 048d73bf, just a different date
format — Anthropic uses -YYYYMMDD (no dashes) while OpenAI uses
-YYYY-MM-DD.
- Add model_id_denylist = re.compile(r'-\d{8}$') to the anthropic
registry entry. The /api/providers/models route already applies
any denylist after fetching, so dated ids drop out automatically.
- Strip the dated 3.5 ids from default_models so the seeded picker
no longer surfaces them; keep claude-opus-4-7 and the 4.5 family
as the curated set.
Net effect: the picker shows opus-4-7 / opus-4-5 / sonnet-4-5 /
haiku-4-5 only, regardless of whether the remote /models call
succeeds or fails.
* fix: provider dialog and mistral short list
* style: fix provider dialog curated list styling
* fix: provider dialog curated model ids placeholder reference
* style: rename Providers to Cloud and tighten dialog header spacing
* UX: rename Providers to Cloud, remove header shortcut
* studio/chat: normalize structured delta.content from reasoning providers
Mistral's magistral (and similarly-shaped reasoning models) stream
chat-completion deltas where choices[0].delta.content is an array of
structured parts rather than a plain string, e.g.
[{ type: 'text', text: '...' }, { type: 'thinking', thinking: '...' }]
The accumulator did 'cumulativeText += delta', which coerced each
part to '[object Object]' and produced output like
'[object Object][object Object]...Hey there!'.
Add extractDeltaText() to normalize delta.content before append:
- string → returned as-is
- array of parts → text/output_text parts contribute their .text or
.content; thinking/reasoning parts are re-wrapped inline as
<think>...</think> so the downstream parseAssistantContent lifts
them into a reasoning part the same way it does for providers that
emit thinking inline. magistral keeps its thinking panel; no other
provider's output shape changes.
- unknown shapes → dropped rather than stringified, so a stray field
cannot pollute the rendered chat with '[object Object]'.
* Studio: restore Cloud icon shortcut in chat header
Brings back the header chip that opens Settings -> Cloud (external
providers) directly from the chat view. Same button as before the
bf24e604 removal: single-mode only, opens useSettingsDialogStore on
the 'connections' tab, tooltip 'API providers'.
* studio/chat: strip trailing template literal from external provider streams
Mistral's magistral occasionally appends a literal '${response}' token
after its actual answer — likely a training-format artifact, since it
keeps happening with an empty system prompt and only on that model.
Apply a tight strip in the chat-adapter SSE accumulator: when the
active provider is external, drop a trailing '${...}' template literal
(with optional whitespace) from cumulativeText after each chunk. The
regex anchors to end-of-string, so mid-stream fragments ('${re')
remain untouched and only collapse once the closing brace arrives.
Local-model output is unaffected.
* studio/providers: scope Kimi picker to kimi-k2.6 / kimi-k2.5
Mirror what the live Kimi docs surface as the current models
(https://platform.kimi.ai/docs/models). Everything else the
remote /v1/models call returns — moonshot-v1-* legacy ids and
dated k2 previews like kimi-k2-0711-preview — is filtered out.
- default_models: ['kimi-k2.6', 'kimi-k2.5'] (was four
legacy moonshot-v1 ids plus the dated k2 preview)
- model_id_allowlist: ^kimi-k2\.[56]$ applied in the
/api/providers/models route after the live fetch
- doc-link comments point at platform.kimi.ai overview /
models / list-models for the next refresh
* studio: drop temperature/top_p for Kimi reasoning models
Kimi k2.5/k2.6 are reasoning-class. The API locks temperature and
top_p to fixed defaults and 400s on any other value with
'invalid temperature: only 1 is allowed for this model'.
The frontend capability map already gated these knobs out of the
external request body, but the OpenAI-compat path on the backend
unconditionally re-adds them from the pydantic ChatCompletionRequest
defaults (temperature=0.7 etc), so the gate was bypassed end-to-end.
Add a generic body_omit hook on the provider registry that
stream_chat_completion consults after building the body, and use it
to strip temperature/top_p for Kimi. Frontend provider-capabilities
flips kimi.temperature and kimi.topP to false so the sliders are
hidden in the chat settings panel as well.
* studio/providers: scope Gemini picker to current 3.x + *-latest aliases
Google's /v1beta/openai/models returns dozens of historical,
experimental, and non-chat ids that we never want in the chat UI.
Cap the picker to the current curated set:
- gemini-3.1-pro-preview
- gemini-3.1-flash-lite
- gemini-3-flash-preview
- gemini-pro-latest
- gemini-flash-latest
- gemini-flash-lite-latest
Default_models seeded with these, model_id_allowlist applied in
the /api/providers/models route to drop anything else the live
fetch returns.
* studio/providers: switch Hugging Face to remote model listing
Per the Inference Providers docs
(https://huggingface.co/docs/inference-providers/index),
GET https://router.huggingface.co/v1/models returns the full
chat-model catalog across all providers, including per-provider
metadata. The OpenAI-compatible endpoint we already use for
chat completions accepts the same Bearer token, so flipping
model_list_mode from 'curated' to 'remote' lets users discover
models via the existing list_models() path without any new
wiring.
- model_list_mode: 'remote' (was 'curated')
- default_models refreshed with current popular ids
(gpt-oss-120b, DeepSeek-V3, Llama-3.3-70B, Qwen2.5-72B) so the
picker still has a sensible seed if /v1/models fails
- notes updated to reference the docs page and clarify the
endpoint is chat-only
* UX: chat cloud icon changed to model select signifier
* studio/providers: org allowlist + count cap for HF Inference picker
The HF /v1/models response is the full cross-provider catalog (hundreds
of ids — community fine-tunes, mirrors, fp8 variants, dated snapshots).
Scope the picker to the first-party org repos worth surfacing and cap
the post-filter list.
- model_id_allowlist matches the org prefixes openai/, deepseek-ai/,
google/, meta-llama/, Qwen/, moonshotai/, mistralai/, zai-org/.
Anything outside those orgs is dropped.
- model_id_limit (new registry field) caps the post-filter list. The
list-models route now slices [:limit] after allowlist/denylist; set
to 15 for HF Inference. Other providers leave it unset and behave
exactly as before.
- default_models stays as the seed so the flagship ids users care
about (gpt-oss-120b, DeepSeek-V3, Llama-3.3-70B, Qwen2.5-72B) are
always reachable regardless of the API's response order.
Dedup is already handled in loadModels() via Set, so no additional
work needed there.
* style: adjust cloud icon right margin with rem spacing
* Studio: cloud openai reasoning level toggle (#5402)
* feat: cloud openai reasoning level toggle
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: honor enable_thinking=false
* fix: prevent local reasoning toggle regressions and align OpenAI effort levels
* fix: isolate external OpenAI reasoning toggle state
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
* fix: clamp reasoning effort
* fix: align OpenAI reasoning effort
* fix: clear stale GGUF badge state
* ui: new badge on cloud setting
* fix: separate selected models from cached provider model list
* Studio: anthropic effort by model family (#5412)
* feat: external thinking control and Anthropic effort mapping
* fix: anthropic thinking constraints and 4.6 max effort mapping
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: harden Anthropic thinking params and effort mapping
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio/backend: drop top_p from Anthropic body when thinking is enabled
PR 5412 added body['top_p'] = max(0.95, min(top_p, 1.0)) inside the
thinking branch of _stream_anthropic, but Anthropic returns 400 on
extended/adaptive thinking when both temperature and top_p are set:
invalid_request_error: temperature and top_p cannot both be
specified for this model. Please use only one.
(Observed on Claude Opus 4.6.) The contract for thinking-enabled
requests is temperature=1 with neither top_p nor top_k allowed.
Replace the body['top_p'] = ... line with body.pop('top_p', None).
Defensive pop rather than a bare delete: the base body construction
above does not currently set top_p, but a future edit that adds it
would silently reintroduce the regression.
* studio/chat: force reasoningEnabled=true on local reasoning-effort models
Followup to PR 5402 / 5412. The model-status refresh path in
use-chat-model-runtime carried reasoningEnabled forward verbatim for
every reasoning-capable model. That left one observable edge case:
1. user picks an external model that supports Off (gpt-5.x, Claude
4.x), clicks Off — store sets reasoningEnabled=false
2. user switches back to a local reasoning-effort model
(gpt-oss / Harmony-style) which does NOT support Off
3. composer's effectiveReasoningEnabled override paints the UI as
'Think: <level>' (on)
4. chat-adapter sees reasoningEnabled=false on the local branch
and sends '{}', so the backend's _request_reasoning_kwargs
returns None and the Harmony template falls back to its own
default effort instead of the displayed level
Mirror the composer's override in the store on load: for local
reasoning-effort models (where supportsReasoningOff is false), force
reasoningEnabled=true so the store and the UI agree on every send.
Other reasoning styles still inherit prior state — only the
reasoning-effort family changes.
* studio/backend: align Anthropic thinking with the extended-thinking docs
Two compliance fixes against
https://platform.claude.com/docs/en/build-with-claude/extended-thinking
1. Adaptive-mode effort field shape
The docs spell adaptive thinking as:
{'thinking': {'type': 'adaptive'}, 'effort': {'type': '<level>'}}
We had been sending the legacy 'output_config: {effort: <level>}'
shape, which Anthropic appears to silently ignore — adaptive ran
at the server default effort regardless of the user's selection.
Rename to 'effort: {type: <level>}'.
2. thinking_delta event translation
The Messages-API streams reasoning content as
content_block_delta events with delta.type == 'thinking_delta',
which our SSE loop was dropping entirely. On Claude 4.5/4.6 with
display=summarized (the default), the user would see the answer
text but never the reasoning panel. Wrap thinking_delta.thinking
as inline <think>...</think> chunks (same pattern as the OpenAI
Responses path) so the frontend's parseAssistantContent lifts it
into the reasoning channel. The </think> closer fires on the
first text_delta transition, on content_block_stop for the
thinking block, on message_delta, and on message_stop —
whichever arrives first — so no model path can leak an
unclosed <think> into chat output.
signature_delta events are left as no-ops; they carry
verification metadata, not user-visible content.
Adds test_anthropic_thinking_translation.py with httpx.MockTransport
coverage of: effort shape on adaptive (Claude 4.6), budget_tokens
shape on manual (Claude 4.5), thinking_delta wrapping with signature
suppression, and thinking-only turns (display=omitted on Opus 4.7).
* studio/backend: revert Anthropic adaptive effort to output_config nesting
The previous commit (0a664df4) moved the adaptive-thinking effort
field to a top-level 'effort: {type: <level>}' based on a misread of
the docs page. The actual Messages API schema nests it under
output_config:
thinking: optional ThinkingConfigParam ({type: 'adaptive'})
output_config: optional OutputConfig
effort: optional 'low' | 'medium' | 'high' | 'xhigh' | 'max'
Sending the top-level field produced:
400 invalid_request_error: effort: Extra inputs are not permitted
Restore the body to:
body['thinking'] = {'type': 'adaptive'}
body['output_config'] = {'effort': effort}
This was the shape PR 5412 originally shipped (and the author
validated against live APIs). My 'compliance fix' was a regression.
The companion thinking_delta SSE translation added in 0a664df4 stays
— that part WAS missing from the previous shape and is unchanged
by this revert. Test pinning the body shape flipped to assert
output_config.effort, top-level effort is asserted absent.
* studio/backend: opt in to summarized thinking display on adaptive
Per the adaptive-thinking docs, the 'display' field on the thinking
config defaults to 'omitted' on Claude Opus 4.7 (and Mythos Preview).
With 'omitted' the API still emits a thinking content block, but its
'thinking' field is empty — only the signature_delta arrives.
Our SSE handler would then surface a stray '<think></think>' for the
empty block and the reasoning panel would stay blank for the entire
response. Set 'display': 'summarized' explicitly on the adaptive
thinking config so Opus 4.7 emits thinking_delta events the same way
Opus 4.6 / Sonnet 4.6 do (where 'summarized' is the default, making
the explicit setting a no-op there).
The manual-thinking branch (Claude 4.5) is unaffected — its default
is also 'summarized', and we have no reason to override it.
* studio/backend: log Anthropic SSE event counts for thinking diagnostics
Reports of 'no reasoning panel content on Anthropic' have two
distinct causes that produce the same symptom:
1. Anthropic streamed thinking_delta events but our frontend
dropped them somewhere on the rendering side.
2. Anthropic did not emit thinking_delta at all (adaptive mode
can skip thinking for simple prompts even with effort=high,
and display=summarized only re-enables the *content* — it
does not force thinking to happen).
Tally each event type for the duration of one stream and log the
counts in the finally branch, so the next 'no reasoning content'
report shows immediately whether thinking_delta was even on the
wire. Zero counts → upstream (model/effort/prompt choice).
Non-zero counts → triage moves to chat-adapter / parse-assistant
-content / the reasoning component.
* studio/backend: route external_provider logs through structlog
The studio backend wires structlog as the active logger (via
LogConfig.setup_logging at main.py:262), but external_provider.py
was using stdlib logging.getLogger(__name__) for every diagnostic.
The stdlib root logger defaults to WARNING with no handlers
attached, so plain logger.info('...') and logger.debug('...') from
this module were being silently dropped — including the
'Proxying chat completion to <url>' and the new
'Anthropic stream event counts' lines. Only WARNING/ERROR survived
(via the implicit fallthrough that the user actually observed
when an Anthropic call 400'd).
Switch the module-level logger to structlog.get_logger(__name__),
matching the routes/providers.py and routes/inference.py pattern.
All existing call sites use printf-style positional args, which
structlog accepts unchanged — no other edits needed.
* studio/backend: disable read timeout on SSE streams to external providers
Anthropic Opus 4.7 (adaptive thinking) and OpenAI gpt-5.x (/v1/responses)
can pause for tens of seconds between bytes while the model is
internally reasoning. httpx's read timeout is the *gap* between
successive reads, not a wall clock on the whole request — so the
shared 120s default was cutting streams mid-response:
log: Anthropic stream event counts (... text_delta: 11)
Read timeout from anthropic
(eleven text deltas in, no content_block_stop, no message_stop)
Add a separate _stream_timeout on ExternalProviderClient with
read = None (no gap timeout) and the same 10s / 120s connect/write/
pool bounds, then use it at the three SSE streaming call sites:
default OpenAI-compat chat completions, _stream_anthropic, and
_stream_openai_responses. Non-streaming call sites (chat_completion,
list_models, verify_models_endpoint_lightweight) keep self._timeout
because a stuck non-streaming response should still fail fast.
* studio/backend: log outbound Anthropic request shape for thinking debug
After bumping to Xhigh effort the user still saw zero thinking_delta
events and only one content_block_start, meaning Anthropic Opus 4.7
opened no thinking block at all. Per the effort docs that should be
impossible — Xhigh always thinks. Two open hypotheses:
1. Our adaptive branch is not wiring output_config.effort onto the
outbound body for this code path (regex miss, frontend never
propagated reasoning_effort, etc).
2. Anthropic is silently accepting output_config as an unknown
field and falling back to high default effort regardless.
Add a single-line structlog INFO right before the stream POST that
echoes the keys actually present on the body (thinking, output_config,
temperature, presence of top_p / top_k, max_tokens). Messages are
deliberately excluded to keep PII out of the log. With this in place
the next 'no thinking on 4.7 at Xhigh' report shows immediately
whether we sent the effort knob — separating client bug from
provider behaviour.
* studio/chat: surface delta.reasoning_content from Kimi / DeepSeek thinking
Kimi (kimi-k2.6, kimi-k2-thinking) and DeepSeek's reasoner stream
their thinking content via a separate top-level field on the
chat-completion delta — choices[0].delta.reasoning_content — rather
than as a structured part inside delta.content. Per Kimi docs:
In streaming output (stream=True), the reasoning_content field
will always appear before the content field.
Our chat-adapter SSE loop only read delta.content (via
extractDeltaText), so the entire reasoning channel from these
providers was being silently dropped — kimi-k2.6 thinks by default
yet the chat UI showed no reasoning panel.
In the adapter:
- Read both delta.content and delta.reasoning_content per chunk
- When reasoning_content arrives, open a <think> block in
cumulativeText (mirrors how the backend wraps Anthropic
thinking_delta and OpenAI Responses reasoning summaries)
- When content arrives after reasoning, close </think> first
- On stream end, force-close any still-open <think> so
parseAssistantContent can lift it into a reasoning part cleanly
Anthropic and OpenAI Responses paths are unaffected — they already
wrap as <think> on the backend and never set reasoning_content.
* studio: Kimi thinking toggle + 16k max_tokens floor
Two coordinated changes so Kimi's thinking is user-controllable and
the response budget meets the docs' floor.
Toggle (frontend + backend):
- getExternalReasoningCapabilities now handles provider=='kimi':
kimi-k2.6 -> reasoning_style=enable_thinking, reasoningOff allowed
kimi-k2-thinking -> always on (reasoningAlwaysOn=true, no off)
kimi-k2.5 (and anything else) -> no reasoning controls
- chat-adapter already forwards enable_thinking on the
enable_thinking-style branch, so the user toggle reaches the
backend without additional wiring there.
- external_provider stream_chat_completion now translates the
boolean into Kimi's wire shape on the default OAI-compat path:
enable_thinking=True -> body['thinking'] = {type: enabled, keep: all}
enable_thinking=False -> body['thinking'] = {type: disabled}
kimi-k2-thinking ignores the toggle so the API never gets a
disabled value it would reject. Other providers on the same
path are unaffected (gated on provider_type == 'kimi').
Max tokens floor:
- New EXTERNAL_MIN_OUTPUT_TOKENS_BY_PROVIDER table and
getExternalMinOutputTokens helper. Kimi entry = 16000 per docs:
'Set max_tokens >= 16,000 to ensure the full reasoning_content
and final content can be returned without truncation.'
- chat-adapter clamps the outbound max_tokens to
min(max(stored, providerMin), EXTERNAL_MAX_OUTPUT_TOKENS),
so a stored value of 4096 still becomes 16000 when sending to
Kimi (other providers unaffected, min stays effectively 64).
- chat-settings-sheet's Max Tokens slider min mirrors the same
floor when an external Kimi model is selected, so the slider
cannot show a value lower than what we'd actually send.
- chat-page threads activeExternalProviderType down to the panel.
* fix: stabilize external reasoning controls for Anthropic 4.6 and OpenAI o3
normalize Anthropic 4.6 reasoning effort handling by accepting max as an alias and mapping it to xhigh, while keeping Sonnet/Opus 4.6 in default model suggestions.
broaden reasoning effort typing across backend/frontend and migrate persisted max selections to xhigh for compatibility.
remove reasoning.summary=\"auto\" from OpenAI /v1/responses payloads to avoid o3 eligibility/gating errors.
tighten provider model filtering to hide retired gpt-5.3 IDs and add exact/prefix filtering support in provider routes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: add openrouter/free + full reasoning passthrough on OpenRouter
Four-layer wire-up so the OpenRouter free-router model (which picks
a free model at random per request, filtered by needed capabilities)
shows up in the picker and its reasoning channel surfaces in the
chat UI.
Registry:
- providers.py: openrouter/free seeded at the top of openrouter
default_models. Curated list, so picker shows it immediately.
Frontend capability map:
- provider-capabilities.ts: getExternalReasoningCapabilities now
treats openrouter as enable_thinking style with off support. The
Think dropdown appears for every OpenRouter model; the gateway
silently no-ops the parameter for models that do not reason, so
surfacing one toggle on every model is safe.
Backend reasoning passthrough:
- external_provider.py stream_chat_completion (default OAI-compat
branch): for provider_type=='openrouter', translate the request:
reasoning_effort in {low,medium,high} -> body['reasoning'] =
{'effort': <level>}
enable_thinking=True -> body['reasoning'] = {'enabled': True}
enable_thinking=False -> body['reasoning'] = {'enabled': False}
Matches the documented shape at
https://openrouter.ai/docs/guides/best-practices/reasoning-tokens
with effort and max_tokens mutually exclusive.
Frontend SSE reader:
- chat-adapter.ts: OpenRouter streams reasoning as a third shape we
did not handle yet: delta.reasoning_details is an array of parts
like {type: 'reasoning.text', text: '...'}. Pull text from every
part, merge with the existing delta.reasoning_content channel
used by Kimi/DeepSeek, and feed the combined string through the
same <think>...</think> wrap path so parseAssistantContent lifts
it into the reasoning panel. Anthropic/OpenAI Responses paths
already wrap on the backend, so they never set this field — no
cross-provider interference.
* studio/backend: surface OpenRouter SSE errors and router-chosen model in logs
The frontend showed 'Provider returned error' for some openrouter/free
requests with nothing on the backend side to triage from — the
existing 4xx error log only fires when the upstream returns a non-200
status code, but OpenRouter (and most OAI-compat providers) return
200 OK and emit the actual failure as an SSE error event mid-stream,
which our default-path stream loop forwarded verbatim without
logging.
Best-effort diagnostics on the default OpenAI-compat stream path:
- Peek at every `data:` line in the inner forward loop, parse JSON
best-effort (silently skip on failure so nothing is dropped).
- Count event types: delta / error / done.
- On any chunk containing an `error` field, emit a structlog WARNING
with the provider type and the error payload — same trail the
user would otherwise have to dig out of browser devtools.
- Latch the first non-empty `chunk.model` field. OpenRouter reports
the router-picked underlying model there per request, so the
finally-block summary log shows which free model handled the call.
In the finally block:
'openrouter stream complete (model=openrouter/free,
chosen=google/gemini-2.5-flash, events={delta: 47, done: 1})'
Zero overhead for non-error streams (a json.loads per chunk +
dict-key lookups). The structlog logger is already configured at
INFO; ERROR and WARNING surface in JSON logs without further setup.
Hoists `import json as _json` to module top so the default path can
reuse it; the existing in-function imports in _stream_anthropic and
_stream_openai_responses are now redundant but harmless.
* studio/chat: show router-picked model after 'openrouter/free:' in chip
When the user picks openrouter/free, the gateway routes each request
to a different underlying free model. Until now there was no way to
tell which one actually replied without reading the backend logs.
Surface the picked model in the active-model chip:
- chat-runtime-store gains lastOpenRouterChosenModel: string|null
plus a setter. Reset on every model switch unless the user stays
on openrouter/free.
- chat-adapter SSE loop latches chunk.model into the store on
every chunk whose top-level model differs from
openrouter/free, gated on the active checkpoint being
openrouter/free under an OpenRouter provider.
- chat-page externalModels useMemo appends :<chosen> to the display
name for the openrouter/free option when the store has a value,
so ModelSelector renders e.g.
'openrouter/free:google/gemini-2.5-flash'
in the chip. Other models unaffected.
- Model-switch callback in chat-page clears the cached value when
the user moves to any model other than openrouter/free, so the
chip never shows a stale suffix from a previous session.
* studio/chat: shorten openrouter/free chip to openrouter:<short-chosen>
The full display name in use was:
openrouter/free:inclusionai/ring-2.6-1t-20260508:free
The `:free` suffix on the underlying id already conveys 'free model',
which made the leading `/free` on the router id redundant, and the
`inclusionai/` org prefix was just noise crowding the chip.
Trim both. Now the chip renders as:
openrouter:ring-2.6-1t-20260508:free
Strictly a display change in chat-page externalModels useMemo — the
backend wire id stays `openrouter/free`, the runtime store still
caches the full `inclusionai/...:free` value, and the model-switch
clearing logic is unchanged.
* studio/providers: switch OpenRouter to remote listing with org allowlist + cap
Same shape as Hugging Face Inference. The curated list had only four
entries; remote listing fetches OpenRouter's full ~300-model
catalog via /v1/models and the new allowlist + limit scope it back
down to a usable picker.
- model_list_mode: remote (was curated)
- model_id_allowlist matches the prefixes:
openrouter | openai | anthropic | google | meta-llama | qwen
| mistralai | deepseek | moonshotai | inclusionai | zai-org
| z-ai
Anything outside drops out.
- model_id_limit: 20 — first 20 post-filter matches from the live
fetch; default_models stays seeded so the most useful canonical
ids are always visible regardless of API response order.
- default_models seed extended from 4 to 6 (openrouter/free,
openai/gpt-4o, anthropic/claude-sonnet-4-5, google/gemini-2.5-flash,
mistralai/mistral-large-2411, deepseek/deepseek-r1).
openrouter/free remains the first entry, so the dialog's
loadModels() union-merge (registryDefaults first, then remote,
deduped via Set) keeps it at the top of the picker.
* feat: external mistral thinking toggle
* studio/chat: fix TS2540 by replacing readonly ContentPart instead of mutating
The ContentPart type from @assistant-ui/react marks `text` as readonly,
so the coalesce-adjacent-same-type-part optimization in
parseAssistantContent failed the tsc build with:
parse-assistant-content.ts(15,10): error TS2540: Cannot assign to
'text' because it is a read-only property.
parse-assistant-content.ts(25,10): error TS2540: ...
This broke npm run build, the Studio installer's `building frontend...`
step, and every downstream CI job that runs against an installed
Studio (Mac/Windows/Linux variants of Studio API CI, GGUF CI, UI CI,
Tauri CI, Wheel CI).
Replace the last element with a fresh merged object instead of
mutating its `text` field. Same allocation profile as the previous
path (one object swap per merge), type-safe under the readonly
declaration. Behaviour unchanged.
* studio/backend: restore summary='auto' on OpenAI Responses reasoning body
A recent refactor dropped the `summary: 'auto'` field from the
reasoning config we send to /v1/responses. Without it OpenAI does
not emit reasoning summary events on most reasoning models, which
means our SSE handler has no <think>…</think> to wrap and the chat
reasoning panel stays blank for any gpt-5.x / o3 response.
The expected wire shape is:
body['reasoning'] = {'effort': '<level>', 'summary': 'auto'}
Two backend tests pin this:
- test_responses_reasoning_effort_included_when_requested (high)
- test_responses_reasoning_effort_xhigh_passthrough (xhigh)
Both were failing with AssertionError because the produced body
omitted `summary: auto`.
Restore the field. Skip it only for the explicit "off" case
(effort: 'none'), where summaries serve no purpose. The
enable_thinking=True fallback (no explicit effort) also pairs
medium effort with summary='auto' so that branch produces
reasoning text too.
* chat: external reasoning, OpenRouter curation, Think toggle fixes
* fix: opus and sonnet 4.6 xhigh --> max
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* studio: contain export and dataset paths under their configured roots
resolve_under_root and resolve_dataset_path previously returned absolute
paths unchanged, so an authenticated client could supply
save_directory="/tmp/escape" (or any other absolute path) and have the
exporter drop adapter files anywhere the server user could write. This
turned up during a recent audit pass where an authenticated POST to
/api/export/export/lora with save_directory="/tmp/lora_escape_test"
returned 200 and wrote adapter_model.safetensors, adapter_config.json,
and tokenizer files under /tmp.
The fix is two-layered:
storage_roots.py adds an _assert_contained(resolved, root) helper that
runs after path resolution and rejects any result whose realpath does
not sit under realpath(root). resolve_under_root now rejects '..'
segments and null bytes outright, and only accepts absolute inputs when
they are already inside the configured root (internal call sites that
re-resolve a stored absolute path stay idempotent;
worker.py:resolve_output_dir(output_dir) etc. continue to work).
resolve_dataset_path picks up the same containment rule, scoped to the
three dataset roots.
models/export.py adds field_validator("save_directory", mode="before")
to ExportCommonOptions and ExportGGUFRequest so bad input fails fast at
422 with a clear message rather than a 500 deep inside the resolver.
The validator rejects empty/whitespace, null bytes, control chars,
strings longer than 255 chars, absolute paths, and '..' segments.
routes/export.py:_export_details now returns os.path.relpath(output_path,
exports_root()) so the Export Complete dialog and /api/models/loras no
longer leak the absolute install prefix to the UI; the basename is
used as a last-resort fallback.
Verified end to end:
- POST /api/export/export/lora {"save_directory":"/tmp/foo"} -> 422
"save_directory must be a name or relative path under the export
root; absolute paths are rejected". /tmp/foo is not created.
- "../../etc/escape" -> 422 "may not contain '..' segments".
- save_directory="my_subdir" -> still accepted (400 only because the
test had no checkpoint loaded yet, not because of validation).
- Internal idempotent re-resolve via resolve_export_dir(absolute path
that is already under exports_root) returns the same path unchanged.
* studio/sandbox: harden bash + python tool execution
The sandboxed Bash and Python tool channels in Chat ran with a thin
preexec hook (PR_SET_NO_NEW_PRIVS + RLIMIT_FSIZE only). Bash had a
small word blocklist; Python had an AST safety pass aimed at
signal-tampering and shell-escape primitives. An audit pass showed
several gaps that a tool-calling model could trigger inadvertently:
- bash curl/wget/nc reached AWS IMDSv2 and returned live STS
credentials for the instance role.
- python "import socket; s.connect((169.254.169.254, 80))"
reached the same endpoint regardless of the bash blocklist.
- "cat /etc/passwd" was blocked at the bash side (because "passwd"
is in the blocklist), but "open('/etc/passwd').read()" in Python
happily returned its contents.
- "chr(115)+chr(117)+chr(100)+chr(111)" style dynamic-arg
construction slipped through the AST shell-escape check.
- The supervisor used proc.kill() on timeout, which only signals
the immediate pid; bash-backgrounded children survived. A fork
bomb could spawn for the full 300s timeout window.
- Session work directories under ~/studio_sandbox/<id>/ were
created with default umask (0o755), so any other UID on the host
could enumerate them.
- session_id sanitisation used a one-shot str.replace("..",""),
which is non-iterative and a small footgun.
This commit takes a conservative middle path: the sandbox still
runs as the Studio UID with no namespace tricks where the kernel
disallows them, but every chokepoint is tightened.
_sandbox_preexec now:
- calls os.setsid() so children share a process group; the
supervisor uses os.killpg(SIGKILL) on timeout/cancel so
backgrounded children die with the parent (new _kill_process_tree
helper, wired into _cancel_watcher and both _bash_exec /
_python_exec timeout branches).
- calls os.umask(0o077) so files the child writes default to 0o600.
- applies PR_SET_PDEATHSIG=SIGKILL so an orphaned child dies if
Studio exits.
- best-effort unshare(CLONE_NEWNET) for a private network namespace
(failure is logged and swallowed; defense-in-depth is still in
place via the bash blocklist and the AST checker below).
- sets RLIMIT_NPROC=10000 (tunable via UNSLOTH_STUDIO_SANDBOX_NPROC),
RLIMIT_AS=8GB, RLIMIT_CPU=300, RLIMIT_NOFILE=1024. The 10k NPROC
figure is chosen to sit well above the ~500 LWPs a healthy Studio
+ llama-server combination already uses while still capping a
runaway fork bomb. NPROC counts LWPs per real UID, so a lower
figure (e.g. 256) starves legitimate bash forks
("bash: fork: retry: Resource temporarily unavailable").
_get_workdir:
- rejects session_id that doesn't match [A-Za-z0-9_-]{1,64};
non-matching values bucket into a shared "_invalid" dir.
- chmod 0o700 on both the workdir and on ~/studio_sandbox/ so
other UIDs cannot read another session's contents.
_BLOCKED_COMMANDS_COMMON gains: doas, pkexec, halt, poweroff, curl,
wget, nc, ncat, netcat, socat, ssh, scp, sftp, rsync, eval, source.
The intent is to keep general bash usage working (echo, ls, pipes,
loops, for, head, etc.) while denying the obvious egress and
escalation paths.
The AST checker (_check_signal_escape_patterns) is split into the
existing shell/signal/loop checks plus a new narrow IO denylist:
- Always flag non-literal args to anything in _SHELL_EXEC_FUNCS,
not just _STRING_SHELL_FUNCS. Closes the dynamic-arg bypass.
- Reject calls to socket.create_connection, socket.socket().connect,
urllib.request.urlopen, http.client.HTTP*Connection, requests.*,
httpx.* whose literal host argument is in a cloud-metadata
denylist (169.254.169.254 + 169.254.* + 100.64.*, plus the
GCP/Alibaba/ECS metadata hostnames and IPv6 link-local). Public
hosts (example.com, huggingface.co, ...) still work. Dynamic
hosts cannot be statically blocked; mitigated by the bash
blocklist + the netns where the kernel allows it.
- Reject literal open("/etc/passwd"), /etc/shadow, /etc/sudoers,
/etc/ssh/*, and /proc/<pid>/environ. Other files
(/etc/os-release, /etc/hostname, /tmp/*, user dirs) still work.
The _check_code_safety summariser is updated to include the new
network_calls and sensitive_file_reads buckets in its error string.
Regression-checked: echo, sleep, ls /tmp, for loops, piped helpers
(echo a | tr a A), urllib.request.urlopen("http://example.com"),
socket.getaddrinfo("example.com",80), open("/etc/os-release"),
open("/tmp/...","w") all still succeed. curl, wget, nc, ssh, rm,
socket.create_connection(("169.254.169.254",80)),
open("/etc/passwd"), open("/proc/self/environ") all correctly
blocked.
* studio: rate-limit login, rotate refresh tokens, add logout, security headers, gate bootstrap injection
A pass over the auth surface found a cluster of related issues that this
commit closes together.
Login (routes/auth.py):
- Add an in-memory per-IP login rate limiter. Five failed POSTs to
/api/auth/login inside a 60s window produce 429 with Retry-After.
A successful login clears the bucket. Previously 30 wrong passwords
in under one second was accepted as 30x 401, which combined with
the (now fixed) admin-username leak from /api/auth/status made
brute-force trivial against a small password.
Logout (routes/auth.py):
- New POST /api/auth/logout returns 204 and calls
storage.revoke_user_refresh_tokens(subject) so the refresh token
is no longer valid. Previously POST /api/auth/logout returned 405
and there was no way to invalidate refresh tokens short of
changing the password. Frontend session.ts already calls
clearAuthTokens() to drop localStorage; the new endpoint lets the
client also tell the server to revoke server-side state.
Refresh-token rotation (routes/auth.py + auth/storage.py):
- New storage.consume_refresh_token(token) atomically validates +
deletes a refresh token, returning (username, is_desktop). The
/api/auth/refresh handler now mints both a new access AND a new
refresh token; the supplied token becomes invalid. Replaying a
consumed refresh returns 401 "Invalid or expired refresh token".
The previous refresh_access_token helper is left in place for
callers that intentionally want the non-rotating shape; nothing
in the route layer uses it now.
/api/auth/status no longer leaks default_username (models/auth.py +
routes/auth.py):
- AuthStatusResponse.default_username becomes Optional[str] with a
None default; the handler always returns None. The frontend already
hardcodes HIDDEN_LOGIN_USERNAME = "unsloth" (auth-form.tsx:82), so
no UI change is required.
window.__UNSLOTH_BOOTSTRAP__ no longer auto-injects (main.py):
- _inject_bootstrap is now opt-in via the
UNSLOTH_STUDIO_INJECT_BOOTSTRAP env var. The previous default
(inject whenever requires_password_change is true) embedded the
plaintext bootstrap password into the first-boot HTML for any
caller that hit /, /change-password, or any unknown SPA path.
Browser extensions and any XSS payload on the page could read it
trivially. With the new gate the bootstrap password lives only in
the auth/.bootstrap_password file (mode 0o600) where it has always
been; users typing it into a current-password field is the right
UX. routes/auth.py:change_password also clears
app.state.bootstrap_password defensively.
Security headers + server fingerprint (main.py + run.py):
- New SecurityHeadersMiddleware adds Content-Security-Policy,
X-Frame-Options: DENY, X-Content-Type-Options: nosniff,
Referrer-Policy: no-referrer,
Permissions-Policy: camera=(), microphone=(), geolocation=(),
interest-cohort=(), and stamps server: unsloth-studio so the
generic uvicorn banner no longer fingerprints the stack. The
uvicorn.Config gains server_header=False so it stops emitting its
own Server header.
/api/health minimisation (main.py):
- Unauthenticated GET /api/health returns just
{"status":"healthy","timestamp":...} so load-balancer liveness
probes keep working without leaking version, device_type,
chat_only, desktop_protocol_version, or studio_root_id to
arbitrary callers. A request that presents a valid Bearer token
still gets the full diagnostic payload so internal launchers and
sibling-Studio detection (which compares studio_root_id) keep
working.
Verification:
- 30 wrong-password POSTs to /api/auth/login -> first 5 = 401, 6th
through 30th = 429.
- POST /api/auth/logout with a fresh token -> 204. The matching
refresh token then fails 401.
- Login -> R1; /api/auth/refresh with R1 -> new access + R2 (R2 !=
R1); /api/auth/refresh with R1 again -> 401; /api/auth/refresh
with R2 -> still succeeds once and rotates again.
- curl /api/auth/status -> default_username: null.
- curl http://127.0.0.1/ does not contain __UNSLOTH_BOOTSTRAP__.
- curl -I / shows CSP, X-Frame-Options: DENY,
X-Content-Type-Options: nosniff, Referrer-Policy: no-referrer,
Permissions-Policy, and server: unsloth-studio.
- curl /api/health unauthenticated -> {status, timestamp} only.
curl with Authorization: Bearer <valid> -> full payload.
- Existing /api/system, /api/models/list, /api/train/status,
/api/inference/status, /api/auth/api-keys, login flow, SPA root
all still return 200 after the changes (regression smoke).
* studio: add SecurityHeadersMiddleware, MaxBodyMiddleware, /recipes redirect, gate _inject_bootstrap, minimise /api/health
This commit lands the main.py-side changes that share a single
middleware-registration spot. They are kept together because every
change here is either (a) a top-level middleware definition that has
to be added next to LoggingMiddleware, or (b) a route handler at the
same file-level.
SecurityHeadersMiddleware (Content-Security-Policy, X-Frame-Options:
DENY, X-Content-Type-Options: nosniff, Referrer-Policy: no-referrer,
Permissions-Policy, server: unsloth-studio). The previous responses
emitted no CSP, no XFO, no Referrer-Policy and were stamped
server: uvicorn.
MaxBodyMiddleware rejects POST/PUT/PATCH on the inference / dataset /
data-recipe / train / export prefixes when Content-Length exceeds
UNSLOTH_STUDIO_MAX_BODY_MB (default 100). The audit hit this by
attaching a 50 MB plain-text file to a chat message and watching
Studio base64-encode it into the JSON body; uvicorn has no enforced
cap so the only previous guard was the per-file 50 MB ceiling that
data-recipe upload routes already enforce. The new middleware extends
that ceiling to the OpenAI-compat path that the Chat attachments
flow through. Verified: a 200 MB JSON POST to /v1/chat/completions
returns HTTP 413 "Request body too large (209,715,264 bytes; max
104,857,600)". A small valid request continues to reach the handler.
_inject_bootstrap is gated behind UNSLOTH_STUDIO_INJECT_BOOTSTRAP.
The previous default was to inline window.__UNSLOTH_BOOTSTRAP__ =
{username, password} into the first-boot HTML whenever
requires_password_change was true, which exposed the plaintext
bootstrap password to any browser extension, page script, or LAN
caller on -H 0.0.0.0. The bootstrap password remains in the on-disk
.bootstrap_password file (mode 0o600) where it has always lived;
users typing it into a current-password field is the right UX.
/api/health unauthenticated returns {"status":"healthy","timestamp":
...} only; the previous payload (version, device_type, chat_only,
desktop_protocol_version, supports_desktop_auth, studio_root_id,
native_path_leases_supported) is preserved for callers that present
a valid Bearer token, so internal launchers and sibling-Studio
detection (which compares studio_root_id) keep working.
/recipes -> /data-recipes 308 redirect. The Data Recipes page lives
at /data-recipes; users typing /recipes hit the SPA catch-all and
saw "Not Found". The redirect also preserves any tail path, so
/recipes/<rest> -> /data-recipes/<rest>.
Verified end to end with curl: CSP / XFO / X-Content-Type-Options /
Referrer-Policy / Permissions-Policy all present on /, server header
is now unsloth-studio (uvicorn's own banner is suppressed via
server_header=False in run.py from the auth-batch commit). Followed
the /recipes redirect lands on the SPA HTML.
* studio: bound TrainingStartRequest hyperparameters at the schema level
POST /api/train/start accepted any value for learning_rate, batch_size,
max_steps, max_seq_length, warmup_steps, warmup_ratio, num_epochs,
save_steps, weight_decay, gradient_accumulation_steps, lora_r,
lora_alpha and lora_dropout, including -1, 0, 1e9, and non-numeric
strings like 'abc' or 'two' (which silently coerce to 0 in the
trainer). Probing showed the API returning 200 to learning_rate=-1
and batch_size=0; only max_steps had any partial clamping.
This commit adds field_validator on every numeric hyperparameter.
Bounds are chosen wide enough to span realistic single-host
configurations (B200 with 180 GB of memory comfortably fits the
upper end) while rejecting the values that always produce broken
training:
- learning_rate: parses str/float, requires 0 < lr < 1.0. Non-numeric
input raises with "learning_rate must be parseable as float (got
'abc')" instead of silently coercing to 0.
- batch_size: [1, 1024].
- gradient_accumulation_steps: [1, 4096].
- num_epochs: [1, 1000].
- max_steps: [1, 1_000_000].
- max_seq_length: [1, 131072].
- warmup_steps: [0, max_steps].
- warmup_ratio: [0.0, 1.0].
- save_steps: [0, 1_000_000].
- weight_decay: [0, 10] (typical 0..0.1).
- lora_r: [1, 512].
- lora_alpha: [1, 1024].
- lora_dropout: [0.0, 1.0).
Each validator names the offending field in its ValueError message
so the 422 response body identifies which input is bad. The
learning_rate validator returns its result as str (the schema field
type is str("2e-4") for backwards compatibility) so existing call
sites that float() the value continue to work.
Verified:
- learning_rate=-1 -> 422 "learning_rate must be > 0 (got -1.0);
typical range is 1e-6 .. 1e-3".
- learning_rate='abc' -> 422 "must be parseable as float".
- batch_size=-1 / 0 / 999999 -> 422 "batch_size must be in [1, 1024]".
- batch_size='two' -> 422 (pydantic int parser).
- max_steps=0 / -5 -> 422 "must be a positive int".
- max_seq_length=200000 -> 422 "must be in [1, 131072]".
- warmup_ratio=2.5 -> 422 "must be in [0.0, 1.0]".
- lora_dropout=1.5 -> 422 "must be in [0.0, 1.0)".
- Valid request with learning_rate='2e-4', batch_size=1, max_steps=5
passes validation and the training run starts as normal.
* studio: redact image-decode errors, clean checkpoint dirs on cancel, tolerate Stop-button + tool-result message shapes
Three small fixes that fall under "do not let the audit findings
become user-visible papercuts".
routes/inference.py - image-decode error redaction (the audit hit
this with a 0-byte / malformed / wrong-extension image upload). The
three image-normalise sites previously raised HTTPException(400,
detail=f"Failed to process image: {e}"). When PIL raised
UnidentifiedImageError(io.BytesIO(raw)) the message string included
"<_io.BytesIO object at 0x7e40a5d7bf60>", leaking both the Python
class name (confirming the PIL/io stack) and a heap address (mildly
useful for ASLR-bypass chaining if another memory-corruption bug is
ever found). Each site now catches UnidentifiedImageError and
returns the generic "Unsupported or corrupt image format"; the
fall-through generic except returns "Failed to process image". No
exception-repr is interpolated into a response body anywhere along
these paths.
core/training/training.py - checkpoint cleanup on cancel. When a
user clicks Cancel Training, the trainer flips _cancel_requested=True
and the supervisor force-terminates the subprocess. The trainer
writes checkpoint-<step> directories under output_dir every
save_steps; previously these survived the cancel and accumulated on
disk (the audit recorded ~67 MB stuck after a 200-step cancel with
save_steps=20). New helper _cleanup_cancelled_checkpoints(output_dir)
globs checkpoint-<int> entries and removes them. It is gated by a
realpath containment check against outputs_root() so it cannot
accidentally rmtree anything outside the configured outputs root.
force_terminate() invokes the helper after the subprocess join when
_cancel_requested is true. Stop-and-Save runs are unaffected because
that path keeps _cancel_requested=False.
models/inference.py - chat message shape tolerance. Two related
frontend interactions used to crash the request validator:
- After the Stop button truncates a generation, the frontend
retained {role:"assistant", content:""} in the conversation
history and replayed it on the next send. ChatMessage previously
required role="assistant" to have non-empty content or tool_calls,
so the next message returned 422 and the thread was permanently
broken. The validator now normalises empty assistant content to
None so the request round-trips and the trailing empty turn can
be ignored downstream.
- The frontend's second-round tool POST drops the streamed
tool_call_id, hitting the strict-spec check "role=tool requires
tool_call_id". The validator now synthesises an opaque id
(call_<8 hex>) when missing, so the request reaches the handler
and the model's final summarising response gets generated. The
proper fix lives in the frontend (carry the streamed id through
the second POST) and will follow.
Verified end to end with curl: HTTP 400 (model not loaded) on both
the empty-assistant history shape and the tool-result-without-id
shape, instead of HTTP 422 from the schema validator.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten code comments from security-hardening pass
Trim verbose docstrings and inline finding references added in the
previous commits in this branch. Functionality unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: await get_current_subject in /api/health and make refresh-token consumption atomic
The /api/health auth probe called get_current_subject(creds) without
awaiting it. The coroutine object is truthy, so any caller presenting a
Bearer header (valid or not) received the full diagnostic payload
including version, device_type, studio_root_id, etc. Await the coroutine
and treat HTTPException as 'fall back to the minimal liveness payload'.
consume_refresh_token did SELECT then DELETE WHERE id under default
autocommit isolation. Two concurrent POST /api/auth/refresh requests
could both win the SELECT before either DELETE ran, defeating
single-use refresh-token rotation. Replace with a single
DELETE ... WHERE token_hash = ? AND expires_at >= ? RETURNING ...
statement so the validate-and-delete lands as one atomic op under
SQLite's write lock (3.45.1 supports RETURNING; min was 3.35).
* studio: enforce body cap on chunked uploads and drop unsafe-inline from script-src
MaxBodyMiddleware previously only inspected the declared Content-Length
header; clients omitting it or sending Transfer-Encoding: chunked
bypassed the cap and could still drive an OOM via the downstream
JSON / file readers on /v1/chat/completions, /api/inference, /api/data-recipe,
/api/datasets, /api/train, /api/export. Rewrite as a raw ASGI middleware
that drains and counts http.request frames, replies 413 once the running
total exceeds UNSLOTH_STUDIO_MAX_BODY_MB before invoking the FastAPI
handler, and replays the buffered body to downstream so route code that
calls request.json() / await request.body() works unchanged.
CSP previously included 'unsafe-inline' on script-src, which defeats the
main XSS protection. The frontend bundle does not need inline scripts;
the only inline <script> the backend ever emits is _inject_bootstrap,
which is opt-in via UNSLOTH_STUDIO_INJECT_BOOTSTRAP. Drop 'unsafe-inline'
from script-src by default; when _inject_bootstrap fires, generate a
per-response nonce, embed it on the inlined <script>, and have
SecurityHeadersMiddleware splice 'nonce-XXX' into the CSP for that one
response (the internal x-internal-script-nonce header is popped before
the response leaves the server). 'unsafe-inline' stays on style-src for
Vite-injected styles.
* studio: drop empty assistant sentinel before passthrough
ChatMessage._validate_role_shape normalises role="assistant", content=""
(the post-Stop sentinel emitted by the frontend) to content=None so the
in-process path can drop it via _extract_content_parts. The passthrough
path then ran m.model_dump(exclude_none=True), which strips the now-None
content key entirely, sending {"role":"assistant"} to llama-server / the
OpenAI-compat backend. That fails upstream and leaves the user without a
recoverable Stop->resume.
Add _drop_empty_assistant_sentinels and call it at both passthrough
message origins: _openai_messages_for_passthrough (covers
/v1/chat/completions and the Responses API which routes through it) and
the anthropic_messages_to_openai output before
_anthropic_passthrough_*. Assistant messages that carry only tool_calls
(no content) are preserved.
* studio/tests: cover audit-fix surfaces and rebase pre-existing tests
Adds and updates pytest coverage for the four bot-flagged audit fixes
landed earlier in this branch and rebases two pre-existing tests that
were broken by the relaxed-validator and /api/health auth-gate changes.
studio/backend/tests/test_middleware.py (new)
MaxBodyMiddleware: small protected, large declared, unprotected
passthrough, chunked-upload-over-cap rejection (the regression for
the original Content-Length-only gap), and chunked-under-cap replay.
SecurityHeadersMiddleware: script-src no longer carries
'unsafe-inline', style-src still does, default headers
(XFO/XCTO/Referrer-Policy/Permissions-Policy/server), and the
internal x-internal-script-nonce header is consumed by the
middleware and converted to 'nonce-XXX' in the CSP.
/api/health: no auth -> minimal, invalid Bearer -> minimal
(the await regression), valid Bearer -> full diagnostic payload.
studio/backend/tests/test_desktop_auth.py
consume_refresh_token: second-call returns None, expired returns
None, and a 64-thread concurrent pile-up against the same hash
produces exactly one successful consumer (regression for the
SELECT-then-DELETE race).
test_health_response_reports_desktop_capability_fields: rebase
against the new health_check(request) signature by going through
TestClient with a real bearer instead of asyncio.run-ing the
handler directly.
studio/backend/tests/test_openai_tool_passthrough.py
Pin the new ChatMessage tolerance: assistant without content or
tool_calls is tolerated (normalises content -> None), empty-string
and empty-list assistant content normalise to None, and a missing
/ empty tool_call_id on role='tool' is synthesised as call_<hex>
rather than raising. Tests for _drop_empty_assistant_sentinels
cover the three drop shapes (empty string, empty list, missing
content key), preservation of assistant text and tool_calls-only
messages, and end-to-end through
_openai_messages_for_passthrough.
studio/backend/main.py
SecurityHeadersMiddleware.dispatch used response.headers.pop(...)
for the nonce-header handoff; Starlette's MutableHeaders has no
pop. Read-then-del so the internal handoff header is still
stripped before the response leaves the server.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/tests: rebase three more pre-existing CI tests against this branch
CI on PR #5375 was red on three tests that were tuned for behaviour
predating this branch. Updates each so the assertions match what the
audit fixes intentionally changed; no production code touched.
studio/backend/tests/test_trained_model_scan.py
test_scan_trained_models_includes_lora_and_full_finetune_outputs
passed an absolute tmp_path through scan_trained_models, which now
runs resolve_output_dir / _assert_contained against outputs_root().
Repoint outputs_root() at tmp_path via monkeypatch so the fixture
dirs land under the configured root and the realpath containment
check passes.
tests/test_studio_install_workspace_guard.py
test_health_endpoint_exposes_studio_root_id_not_raw_path read
the first 1500 bytes after @app.get("/api/health") and asserted on
the studio_root_id literal. The handler grew (unauth short-circuit
+ await dependency gate) and the literal slid past the byte window.
Replace the fixed window with a slice up to the next top-level
@app.* decorator so the test surveys the whole handler regardless
of size.
tests/studio/studio_api_smoke.py
The "login burst (5x wrong pw) -> 401 each" assertion was tagged
"When/if we add one, this assertion updates in the same PR." We
added the per-IP rate-limit in routes/auth.py
(_LOGIN_MAX_FAILS=5/60s) but missed the assertion update. Rewrite
the burst probe to observe the new invariant: at least one 401,
eventual transition to 429, and Retry-After present on the 429.
Adds a small _login_with_headers helper since the existing login()
helper drops response headers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci(studio-ui): set UNSLOTH_STUDIO_INJECT_BOOTSTRAP=1 for Playwright Studios
The Chat UI Playwright test drives the first-boot change-password
form, which (per playwright_chat_ui.py step "1. Change-password
through the UI") pre-seeds the hidden current_password field from
window.__UNSLOTH_BOOTSTRAP__. That global is only emitted when the
backend's _inject_bootstrap path fires, which since the security
pass on this branch is gated behind UNSLOTH_STUDIO_INJECT_BOOTSTRAP
and defaults to off. Without the global, the React form's
current_password validator never satisfies, the submit button stays
disabled, and the composer.wait_for() probe times out on
/change-password.
Re-enable injection only for the CI Studios that drive the chat UI
across linux/mac/windows. Production deployments are unaffected: the
env var has to be explicitly opted into, and the on-disk
auth/.bootstrap_password remains the source of truth for human users
typing the password in by hand.
Covers all eight Studio launch sites: the primary chat-ui boot and
the "extra UI tests" boot for each of the three OSes, plus the
pipeTransport JSON-crash retry relaunches in the macOS workflow that
re-spawn Studio mid-job.
A follow-up frontend PR will add a visible current_password input so
the form satisfies its own validator without needing the bootstrap
auto-fill at all; once that lands this CI knob can come back out.
* studio/sandbox: drop unshare(CLONE_NEWNET); add trusted-host allowlist; block sandbox file uploads; raise CPU rlimit default to 600 s
CLONE_NEWNET inside _sandbox_preexec silently killed every outbound
HTTP request from sandboxed Python whenever the kernel allowed
unprivileged user namespaces. requests.get('https://huggingface.co'),
urllib.request.urlopen('https://en.wikipedia.org/wiki/...'),
socket.connect(('arxiv.org', 443)) all failed despite the AST visitor
intending to allow them. The bash blocklist (curl / wget / nc / ssh /
scp / sftp / rsync / socat / eval / source) plus the AST-level
metadata-host denylist still carry the network policy after this
change; CLONE_NEWNET was redundant with both.
Add _TRUSTED_PUBLIC_HOST_LITERALS + _TRUSTED_PUBLIC_HOST_SUFFIXES
(~100 informational hosts: Wikipedia language subdomains, Wikimedia,
Wikidata, Google search, Bing, DuckDuckGo, HuggingFace, GitHub,
raw.githubusercontent.com, arXiv, StackOverflow / Stack Exchange,
MDN, docs.python.org, PyTorch / TensorFlow / NumPy / pandas docs,
pypi / files.pythonhosted.org / npmjs / crates.io, ReadTheDocs,
arXiv, Britannica, BBC / Reuters / Nature / Science, NASA / CDC /
NIH / WHO open data, api.weather.gov). The visitor now blocks
literal hosts that are neither metadata nor trusted with a short
LLM-readable string so the model can retry with an allowed source
instead of choking on a multi-line error.
Block upload-shape calls regardless of host: requests.post / put /
patch / delete / request with files= or data=open(...) /
data=bytes_literal; httpx equivalents; urllib.request.urlopen /
Request with data=...; HuggingFace upload_file / upload_folder /
upload_large_folder / create_commit (module-level FQ paths AND
method-name match on any receiver). Message: "Blocked: file upload
disallowed in sandbox".
Bump UNSLOTH_STUDIO_SANDBOX_CPU_S default 300 -> 600 s so long
agentic chains that span multiple tool calls don't get SIGXCPU'd
mid-stride. Env-var override path is unchanged.
Host normalisation now strips trailing dot, userinfo @, and explicit
port before allowlist / denylist comparison so trailing-DNS-dot,
userinfo-smuggling, and explicit-:443 URLs are decided correctly.
* studio: raise default request-body cap from 100 MB to 500 MB
UNSLOTH_STUDIO_MAX_BODY_MB default goes 100 -> 500 to comfortably
cover vision + audio + multi-recipe-batch JSON payloads. The
MaxBodyMiddleware stream-counting logic from this branch's earlier
06ec088 already handles chunked bodies up to the new cap; env-var
override path is unchanged for callers that want a tighter limit.
* studio/auth: restore /api/auth/status.default_username to 'unsloth'
This branch's earlier b39e9a4 changed default_username to None on the
public /api/auth/status endpoint so the username field didn't leak to
unauthenticated callers. In practice this regressed third-party
clients (and the in-tree React login form's pre-fill UX) without
adding meaningful security: the bootstrap password is the actual
secret, and the username 'unsloth' is the documented default.
Pin default_username to storage.DEFAULT_ADMIN_USERNAME ('unsloth')
and tighten the response model so the field is required rather than
Optional. Anyone who needs anonymisation can still reach for an
allow-list deployment with auth disabled.
* studio/training: raise max_seq_length / batch_size / lora_r / lora_alpha caps
This branch's 7102815 introduced field validators with conservative
caps. The follow-up loosens them so long-context experiments and
high-rank LoRA exploration aren't gated at the schema layer:
_MAX_BATCH_SIZE 1024 -> 4096
_MAX_SEQ_LENGTH 131_072 -> 2_000_000 (2M tokens)
lora_r cap 512 -> 16_384 (_MAX_LORA_R)
lora_alpha cap 1024 -> 32_768 (_MAX_LORA_ALPHA)
_MAX_GRAD_ACCUM / _MAX_STEPS / _MAX_EPOCHS / lora_dropout /
warmup_ratio / weight_decay are unchanged. Hardware (VRAM, host
RAM, kernel launch latency) is now the binding constraint at the
new caps, which is the correct ordering -- the validator stays a
sanity check on -1 / 0 / 'abc' style garbage, not a usability gate.
* studio/tests: cover sandbox allowlist + upload block + raised training caps
studio/backend/tests/test_sandbox_tools.py (new):
TestMetadataHostDenylist -- short "Blocked: cloud-metadata host"
message on AWS IMDS, GCP metadata,
Alibaba ECS, AWS IPv6 IMDS, 169.254/16.
TestTrustedHostAllowlist -- Wikipedia (any language subdomain),
Google, DuckDuckGo, HF, raw GitHub,
arXiv, StackOverflow / family,
MDN, docs.python.org, pypi, BBC,
api.weather.gov, NumPy / PyTorch docs.
TestUntrustedHostBlock -- example.com / random unlisted host
rejected with the short "Blocked: host
not in sandbox allowlist; use an
allowed informational source" message.
Dynamic URLs (computed var) still pass
-- documented limit of static analysis.
TestHostNormalization -- trailing dot, explicit :443, uppercase,
userinfo-@-smuggle all decided
correctly without false-block /
false-pass.
TestUploadDenylist -- requests / httpx / urllib.urlopen with
files= / data=open / data=bytes,
HfApi().upload_file / upload_folder /
create_commit, module-level
huggingface_hub.upload_folder. POST
json= to trusted host still passes.
TestSandboxCpuRlimitDefault -- pin UNSLOTH_STUDIO_SANDBOX_CPU_S=600
default and confirm CLONE_NEWNET
source line is gone.
TestMaxBodyDefault -- pin UNSLOTH_STUDIO_MAX_BODY_MB=500
default.
studio/backend/tests/test_studio_train_validation.py (new):
Pin at-cap-accepts / over-cap-rejects boundaries for
max_seq_length=2_000_000, batch_size=4_096, lora_r=16_384,
lora_alpha=32_768 so a future regression that tightens them back
without explicit user opt-in is caught.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten code comments across the security-hardening pass
* studio: always inject bootstrap credentials on first boot
The UNSLOTH_STUDIO_INJECT_BOOTSTRAP gate added an extra
terminal-to-browser copy-paste on every fresh install. In practice
the LAN credential leak it guarded against is narrow: the password
is one-time, the user rotates it on the very next click, the
default Studio bind is 127.0.0.1, and -H 0.0.0.0 already exposes
the entire API surface. Drop the gate so the inject fires whenever
a bootstrap password is still pending. The CSP nonce wiring stays
in place; the inline script remains the only inline script the
backend ever emits.
The three Playwright UI smoke workflows lose their
UNSLOTH_STUDIO_INJECT_BOOTSTRAP=1 lines along with the explanatory
comment blocks since the inject now happens by default.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* Studio: pin GPU at 95% headroom and warn on silent CPU fallback
Two related runtime-side fixes for unslothai/unsloth#5106 ("model
loaded fully on RAM instead of VRAM"):
1. GPU pin threshold bump 0.90 -> 0.95
-------------------------------------
``_select_gpus`` and the auto-ctx pin loop in ``start_llama_server``
used a ``pool * 0.90`` threshold to decide whether the model fits on
GPU. Models that needed 91-94% of free VRAM were classified as "does
not fit", so Studio set ``gpu_indices = None`` and shipped
``--fit on`` to llama-server without ``-ngl``. The unsloth
llama.cpp fork's ``--fit on`` then ran with its default
``--fit-target 1024`` (1 GiB margin per device, an upstream default
inherited from ggml-org#18679). On a tight fit where compute
buffers + CUDA context push the projected free below the 1 GiB
target, the fork's fit logic shaves layer weights off the GPU --
slow inference for users whose models would have loaded comfortably
with ``-ngl -1``.
The classic reproducer from #5106 (noahterbest's log):
GGUF size: 20.8 GB, est. KV cache: 0.1 GB, context: 4096,
GPUs free: [(0, 22805)], selected: None, fit: True
20.8 GiB on a 22.27 GiB free RTX 4090 is 94% utilization. The model
fits (1.4 GiB headroom), but the 0.90 threshold kicks it to fit
mode. Bumping to 0.95 keeps these in the fits-on-GPU branch and
emits ``-ngl -1`` directly. The fork's ``--fit on`` still serves as
the safety net for the genuinely-too-large case.
The auto-ctx fallback also re-checks fit at 4096 before handing off
to ``--fit on``: a 20.8 GiB model with a 131072 native context fails
the auto loop at native ctx, falls back to ``min(4096, ctx)``, but
its weights + 4096 KV pin to the GPU comfortably. Without the
re-check we still emitted ``--fit on``.
``_fit_context_to_vram``'s 0.90 budget for context binary search is
intentionally left tighter than the pin fraction. That routine
chooses the slider value, where over-promising would OOM at runtime.
``_select_gpus`` decides whether to pin at all, where being
conservative pushes layers to CPU.
2. Belt-and-suspenders: warn on silent CPU fallback
---------------------------------------------------
After ``_wait_for_health`` succeeds, scan llama-server's stdout for
``model buffer size`` lines. If Studio detected GPUs and intended
GPU use but only CPU buffers were allocated, log a structured
warning citing #5106. Markers cover CUDA / ROCm / Metal / Vulkan /
OpenCL / SYCL backends. New ``_gpu_offload_active: Optional[bool]``
field surfaces the result for any future API consumer.
This catches runtime-load failures the install-time fix cannot
cover (cudart bundle pairing PR #5322 is the install-side
companion): user overriding ``--fit-target``, uncommon driver +
toolkit configurations, future regressions in the install path.
Tests: 10 new cases in studio/backend/tests/test_llama_cpp_context_fit.py:
* TestTightFitPinsToGPU x3: noahterbest's exact reproducer (auto and
explicit ctx pins to GPU at 94%); guard against threshold over-
broadening (genuine overflow still falls back to ``--fit on``).
* TestClassifyGpuOffload x7: CUDA / ROCm / Metal buffer markers
return True; CPU-only buffer lines return False; absent buffer
lines or no GPUs detected return None (no warning).
25 context-fit tests pass (15 baseline + 10 new). 511 tests total
across the affected test files. No regressions.
Refs #5106
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments to be more succinct
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
oobabooga