Round 4 expansion driven by direct fetches of the canonical
SamplingParams + server README pages cited in the user's request.
Adds ten more knobs the docs explicitly support but the panel doesn't
surface yet:
Knob (wire name) llama.cpp vLLM Ollama Source
----------------------------- ---------- ------ -------- ----------------
skip_special_tokens no yes no vLLM SamplingParams
spaces_between_special_tokens no yes no vLLM SamplingParams
include_stop_str_in_output no yes no vLLM SamplingParams
truncate_prompt_tokens no yes no vLLM SamplingParams
n_keep yes no no llama.cpp README
n_probs yes no no llama.cpp README
cache_prompt yes no no llama.cpp README
return_tokens yes no no llama.cpp README
timings_per_token yes no no llama.cpp README
post_sampling_probs yes no no llama.cpp README
Backend rationale:
- vLLM's documented SamplingParams class at
https://docs.vllm.ai/en/latest/api/vllm/sampling_params/ lists
skip_special_tokens (default True), spaces_between_special_tokens
(True), include_stop_str_in_output (False), truncate_prompt_tokens
(None). All four are vLLM-only; llama-server's README does not
document them and Ollama's openai/openai.go translator does not
forward them.
- llama-server's README at
https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
lists n_keep, n_probs, cache_prompt, return_tokens, timings_per_token
and post_sampling_probs as documented per-request fields. vLLM's
SamplingParams has no analog, and Ollama's OAI translator drops them.
Capability matrix:
LLAMA_CPP_CAPABILITIES: 6 llama-only true + 4 vLLM-only false.
VLLM_CAPABILITIES: 4 vLLM-only true + 6 llama-only false.
OLLAMA_CAPABILITIES: all 10 off (OAI translator drops all of them).
Every other bucket: all 10 off.
Skip-when-default rules (mirror upstream defaults):
skip_special_tokens / spaces_between_special_tokens / cache_prompt:
default true upstream — forward only when explicitly false.
include_stop_str_in_output / return_tokens / timings_per_token /
post_sampling_probs: default false — forward only when true.
truncate_prompt_tokens / n_probs: 0 / null = unset — forward when > 0.
n_keep: accepts -1 for "keep all", so the gate is value != 0.
Frontend:
- ProviderCapabilities interface +10 flags.
- InferenceParams +10 nullable fields (3 numeric + 7 boolean), all
null in DEFAULT_INFERENCE_PARAMS.
- OpenAIChatCompletionsRequest wire shape +10 optional fields.
- chat-adapter forwards each in both the external (capability-aware)
and local (capability-bypass) branches.
- chat-settings-storage adds the 3 numeric keys to the existing
nullable-number loop and 7 boolean keys to a new nullable-boolean
loop (alongside ignoreEos).
Backend:
- ChatCompletionRequest +10 Optional Fields with pydantic bounds
(truncate_prompt_tokens ge=1, n_probs ge=0; booleans unbounded;
n_keep accepts -1 so no lower bound).
- llama_cpp.py three payload builders (generate_chat_stream + the
tool-loop payload block + the final-pass stream_payload) each
accept and forward the 10 new kwargs.
- routes/inference.py _build_passthrough_payload accepts and forwards
the 10; both per-request call sites (lines ~2591, ~2790) thread
them from the request payload into the llama_cpp methods.
Test: test_local_passthrough_forwards_vllm_output_and_llama_cpp_
instrumentation round-trips all 10 fields with explicit values
matching each backend's upstream default and confirms each is absent
from the body when unset.
65/65 sampling_params_routing tests pass; frontend tsc clean.
Total local-backend knob coverage now (this PR):
Standard: temperature, top_p, top_k, min_p, repetition_penalty,
presence_penalty, frequency_penalty, seed, stop,
parallel_tool_calls (10)
llama.cpp: typical_p, top_n_sigma, repeat_last_n, dynatemp_range,
dynatemp_exponent, mirostat, mirostat_tau, mirostat_eta,
dry_multiplier, dry_base, dry_allowed_length,
dry_penalty_last_n, xtc_probability, xtc_threshold,
min_keep, ignore_eos, min_tokens, n_keep, n_probs,
cache_prompt, return_tokens, timings_per_token,
post_sampling_probs (23)
vLLM-extra: ignore_eos, min_tokens, skip_special_tokens,
spaces_between_special_tokens, include_stop_str_in_output,
truncate_prompt_tokens (6)
OpenRouter: top_a (1)
Deferred for future PRs (require array / object field shape):
- llama.cpp DRY sequence_breakers (string array)
- llama.cpp samplers ordering (string array)
- llama.cpp / vLLM logit_bias (dict)
- llama.cpp grammar (string) + json_schema (object)
- vLLM guided_json / guided_regex / guided_choice / guided_grammar
- vLLM allowed_token_ids / bad_words / stop_token_ids (int / str arrays)
- OpenAI / Ollama logprobs + top_logprobs (bool + int pairing)
- n / best_of (need SSE multi-choice handling first)
Round 3 expansion driven by direct fetches of the llama.cpp server README,
vLLM's SamplingParams source, and Ollama's openai.go OAI translator.
Adds nine new sampling/control knobs with per-backend capability gating:
Knob (wire name) llama.cpp vLLM Ollama Source
---------------------- ---------- ------ -------- -----------------------
dry_multiplier yes no no llama.cpp README
dry_base yes no no llama.cpp README
dry_allowed_length yes no no llama.cpp README
dry_penalty_last_n yes no no llama.cpp README
xtc_probability yes no no llama.cpp README
xtc_threshold yes no no llama.cpp README
min_keep yes no no llama.cpp README
ignore_eos yes yes no llama.cpp + vLLM SamplingParams
min_tokens yes yes no llama.cpp + vLLM SamplingParams
Backend-side rationale:
- llama.cpp: full chain documented at
https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
- vLLM: SamplingParams source confirms ignore_eos + min_tokens; the
other seven have no field in
https://github.com/vllm-project/vllm/blob/main/vllm/sampling_params.py
- Ollama: openai/openai.go FromChatRequest copies only the OpenAI
subset (temp/top_p/seed/freq/pres/max_tokens/logprobs/topLogprobs/
response_format/reasoning_effort) on the /v1/chat/completions path
Studio uses. All nine new knobs are silently dropped, so the
OLLAMA_CAPABILITIES bucket keeps them off.
Frontend:
- ProviderCapabilities interface gains 9 boolean flags.
- InferenceParams gains 9 nullable fields (8 numeric + ignoreEos
boolean), all defaulting to null in DEFAULT_INFERENCE_PARAMS.
- OpenAIChatCompletionsRequest wire shape gains 9 optional fields
with doc comments.
- LLAMA_CPP_CAPABILITIES: all 9 on. VLLM_CAPABILITIES: 2 on
(ignoreEos + minTokens) via inheritance, 7 off via explicit
override. OLLAMA_CAPABILITIES: all 9 off (inherits + overrides
ignoreEos/minTokens). Every other bucket (openai cloud / chat,
anthropic, gemini, mistral, kimi, deepseek, openrouter) gets all
9 off explicitly.
- chat-adapter.ts gates each knob in both the external (capability-
aware) and local (unconditional-when-meaningful) branches.
Skip-when-default rules:
dry_multiplier > 0 unlocks the 4-field DRY chain
xtc_probability > 0 unlocks the 2-field XTC chain
min_keep > 0, min_tokens > 0 forward only when set higher than 0
ignore_eos forwards only when explicitly true
- chat-settings-storage.ts persists all 9 keys (8 numeric in the
existing nullable-number loop, ignoreEos with its own boolean
handler).
Backend:
- ChatCompletionRequest gains 9 Optional Field declarations with
pydantic ge/le bounds (dry_multiplier ge=0; dry_base ge=1; xtc_*
ge=0 le=1; min_keep / min_tokens / dry_allowed_length ge=0).
- llama_cpp.py: three payload builders (generate_chat_stream + the
two payload-construction blocks inside the tool-loop stream) each
accept the 9 new kwargs and forward via `if x is not None`.
- routes/inference.py: _build_passthrough_payload accepts the 9 new
kwargs and forwards into the body. Two call sites that thread
sampler params from the request payload (lines 2581, 2771) are
extended to forward the 9 new fields.
Test:
- test_local_passthrough_forwards_dry_xtc_min_keep_eos_min_tokens
round-trips all 9 fields through _build_passthrough_payload and
confirms each is absent when unset (so llama-server / vLLM apply
their own defaults).
64/64 sampling_params_routing tests pass; frontend tsc clean.
Deferred for future PRs (require array / object field shape):
- llama.cpp DRY sequence_breakers (string array)
- llama.cpp samplers ordering (string array)
- llama.cpp / vLLM logit_bias (dict)
- llama.cpp n_probs + OpenAI logprobs/top_logprobs
- llama.cpp grammar (string) + json_schema (object)
- vLLM guided_json / guided_regex / guided_choice / guided_grammar
- vLLM allowed_token_ids / bad_words / stop_token_ids
Second 5-Opus reviewer round. Applying high-confidence fixes; speculative
items (gpt-5.5-pro effort restriction, o3 image_generation gating,
o-series parallel_tool_calls per-model, gpt-5.x new model prefixes,
Anthropic fast-mode + Priority exclusion UI gate, Gemini service_tier,
Kimi k2.5 toggleable thinking) deferred to follow-up because they need
type-system changes, more verification, or backend wire work.
OpenAI max-output caps — replace the 3-line table with one driven by
direct dev.openai.com per-model fetches (cross-checked against the Azure
Foundry reasoning table):
- gpt-5.4 / gpt-5.4-pro / gpt-5.4-mini / gpt-5.4-nano: 65536 -> 128000
(https://developers.openai.com/api/docs/models/gpt-5.4 "128,000 max
output tokens"; Azure table same).
- gpt-5.3-codex: 16384 -> 128000
(https://developers.openai.com/api/docs/models/gpt-5.3-codex).
- gpt-5 / gpt-5.1 / gpt-5.2: 32k default -> 128000
(https://developers.openai.com/api/docs/models/gpt-5.2 confirms
128k; Azure table extends to gpt-5/5.1).
- gpt-5.3-chat-latest and gpt-5.1-chat keep 16384 (chat-class
variants per Azure context table row).
- o1 / o3 / o3-mini / o3-pro / o4-mini / codex-mini: 32k default ->
100000 (https://developers.openai.com/api/docs/models/o3 "100,000
max output tokens"; Azure o-series table same).
Implementation: list the two 16k chat-latest ids first so the broader
`gpt-5` 128k entry doesn't shadow them.
OpenAI reasoning_effort levels:
- gpt-5.3-codex: drop "none" from levels + flip supportsOff to false.
Dev page lists the enum as low/medium/high/xhigh only — `none` is
not in the codex variant.
- o-series bucket: change prefix from ["o3"] to
["o1","o3","o4","codex-mini"]. Previously o1 / o4-mini / codex-mini
fell into NO_REASONING_CAPS so the panel HID the effort slider for
them — real UX regression for users on those ids. Azure o-series
table confirms all four accept low/medium/high reasoning_effort.
DeepSeek default_models:
- Add deepseek-v4-pro + deepseek-v4-flash alongside the legacy
deepseek-chat / deepseek-reasoner aliases. The latter retire on
2026-07-24 per https://api-docs.deepseek.com/updates; surfacing
both lets the picker keep working on cutover.
Local backend bucket split (Ollama-stricter):
- Splits the round-1 VLLM_OLLAMA_CAPABILITIES into a vLLM-specific
bucket (keeps top_k / min_p / repetition_penalty / seed on; vLLM's
SamplingParams supports all four) and an Ollama-specific bucket
that ALSO hides top_k / min_p / repetition_penalty. Ollama's OAI
translator (ollama/openai/openai.go FromChatRequest) only copies
the documented OpenAI subset on the /v1/chat/completions path that
Studio uses; the three knobs are silently dropped even though
native /api/chat would forward them via `options`. Hiding them is
the smaller fix vs adding a backend /api/chat rewrite path.
Reviewer claims verified wrong, skipped:
- _ANTHROPIC_NEW_CODE_EXEC_PREFIXES already lists opus-4-7, opus-4-6,
sonnet-4-6 (external_provider.py:337-339). No-op.
- Mistral `seed` already renamed to `random_seed` by backend at
external_provider.py:772. No-op.
- OpenRouter `isOpenRouterMandatoryReasoningModel` uses `Set.has()`
exact match, not prefix match, so deepseek/deepseek-r1-distill-*
cannot accidentally hit the always-on guard. No-op.
Tests: 63/63 sampling_params_routing tests pass; frontend tsc clean.
Five independent reviewers cross-checked every provider's per-model
sampling-knob exposure against live docs (OpenAI, Anthropic, Gemini,
DeepSeek, Kimi, Mistral, OpenRouter, llama.cpp, vLLM, Ollama).
Applying the high-confidence drift fixes here; speculative items (pro
model effort restrictions, gpt-5.3 cap, OpenAI verbosity / o-series
output cap, Gemini topK / service_tier) are deferred to a follow-up
because they need backend wire changes or unverified doc claims.
Anthropic:
- Move claude-opus-4-6 from the 64k group into the 128k group (live
legacy table shows Opus 4.6 Max output = 128k tokens).
https://platform.claude.com/docs/en/about-claude/models/overview
- Add claude-sonnet-4 to the 64k group (was falling through to 32k
default; live legacy table shows Sonnet 4 Max output = 64k tokens).
- Extend ANTHROPIC_REASONING_MODELS with legacy claude-opus-4-1 /
claude-opus-4 / claude-sonnet-4 at none/low/medium/high (live
legacy table marks Extended thinking = Yes for all three).
OpenAI:
- Split the gpt-5/gpt-5.1/gpt-5.2 reasoning bucket. Per Azure docs
footnote ^7^, "minimal is only supported with the original GPT-5
reasoning models. minimal is not supported with gpt-5.1 or greater".
gpt-5.1 / gpt-5.2 now get none/low/medium/high/xhigh with
supportsOff=true; bare gpt-5 keeps minimal/low/medium/high
supportsOff=false. Ordering puts gpt-5.1 / gpt-5.2 before gpt-5 in
the find() loop so the longer prefix matches first.
https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/reasoning
DeepSeek:
- Hide `seed` and `parallel_tool_calls` in the deepseek capability
bucket. Neither field is in the current /chat/completions schema
(body fields: messages, model, thinking, max_tokens, response_format,
stop, stream, stream_options, temperature, top_p, tools, tool_choice,
logprobs, top_logprobs, user_id). Surfacing them in the UI would be
the silent-drop UX the file header warns against.
https://api-docs.deepseek.com/api/create-chat-completion
Mistral:
- magistral-medium-latest / magistral-small-latest are NATIVE
always-on reasoning models; injecting reasoning_effort returns 422
upstream. Switch both to withEnableThinkingStyle({reasoningAlwaysOn:
true}) instead of the old none/medium/high effort ladder.
- mistral-small-latest / mistral-medium-latest / mistral-vibe-cli-latest
expose the documented three-tier adjustable ladder
(none/low/medium/high), not the truncated none/high pair that was
here before. mistral-medium-latest was not handled at all and now
sits in the same bucket as small.
https://docs.mistral.ai/studio-api/conversations/reasoninghttps://mistral.ai/news/magistral
OpenRouter:
- Drop google/gemini-pro-latest from OPENROUTER_MANDATORY_REASONING_
MODELS; the gateway 404s the id today
(https://openrouter.ai/google/gemini-pro-latest). Removing rather
than re-pinning to a versioned id that may rotate again.
Local backends:
- Split LOCAL_LLAMA_CAPABILITIES into LLAMA_CPP_CAPABILITIES (full
chain — for llama_cpp + custom) and VLLM_OLLAMA_CAPABILITIES (OpenAI
subset + top_k/min_p/repetition_penalty/seed, no extended samplers).
vLLM's SamplingParams has no typical_p / top_n_sigma / repeat_last_n
/ dynatemp_* / mirostat* fields, and Ollama's OpenAI translator
(ollama/openai/openai.go FromChatRequest) only copies the OpenAI
subset. Surfacing the eight extra sliders for vllm / ollama was
silent-drop UX.
Tests:
- test_deepseek_payload_omits_seed_and_parallel_tool_calls: read the
TS file as text and assert the bucket has seed:false and
parallelToolCalls:false. Backend has no JS engine; this is the
cheapest way to lock the wire-drop invariant.
- 63/63 sampling_params_routing tests pass; frontend tsc clean.
The 4.7 generation only shipped Claude Opus 4.7; Sonnet stops at 4.6
and Haiku at 4.5 per
https://platform.claude.com/docs/en/about-claude/models/overview.
The earlier `^claude-(?:opus|sonnet|haiku)-4-7` regex on both the
backend strip (_ANTHROPIC_4_7_SAMPLING_REMOVED in external_provider.py)
and the frontend mirror (ANTHROPIC_4_7_SAMPLING_REMOVED_REGEX in
provider-capabilities.ts) would have pre-emptively hidden temperature
/ top_p / top_k for any future claude-sonnet-4-7 or claude-haiku-4-7
id, even though Anthropic has explicitly not extended the sampling
removal beyond Opus. Tighten both regexes to `^claude-opus-4-7(?:[-.]|$)`
and update the routing-test pin so claude-sonnet-4-7 and claude-haiku-4-7
are in `should_not_match`. If those ids ever ship and adopt the same
removal, widening the regex is one-line.
Single conflict on studio/backend/core/inference/external_provider.py:
main added `previous_response_id` plumbing for OpenAI Responses chaining
in the same body-builder block PR 5711 uses for service_tier /
parallel_tool_calls. Kept both sets — service_tier+parallel_tool_calls
write first, then previous_response_id appends. No behavioural change
to either feature.
163 backend routing tests still pass; frontend tsc clean.
Cross-checked every supported sampling field against each provider's
live docs + LiteLLM's drop_params surface + the llama.cpp server
README. Pulled in the most-asked-for samplers that the PR was missing.
New ProviderCapabilities flags (default false on every SaaS provider
since none accept these):
- typicalP (already shipped one commit prior)
- topNSigma llama.cpp `top_n_sigma`
- repeatLastN llama.cpp `repeat_last_n` (paired w/ repeat_penalty)
- dynatempRange llama.cpp `dynatemp_range`
- dynatempExponent llama.cpp `dynatemp_exponent`
- mirostat llama.cpp `mirostat` mode (0/1/2)
- mirostatTau llama.cpp `mirostat_tau`
- mirostatEta llama.cpp `mirostat_eta`
- topA OpenRouter `top_a` (alternate dynamic-top-P)
Capability bucketing split: ALL_SUPPORTED retired in favor of
- LOCAL_LLAMA_CAPABILITIES -> custom / vllm / ollama / llama_cpp
(full llama.cpp sampler chain, top_a off — not a llama.cpp field)
- OPENROUTER_CAPABILITIES -> openrouter
(gateway's documented set incl. top_a, llama.cpp-only knobs off
because OpenRouter docs don't list them and they'd be silently
dropped on most underlying routes)
InferenceParams gains 8 nullable-number fields (mirroring `seed`'s
"null = unset, finite-number = forwarded" shape). DEFAULT_INFERENCE_PARAMS
defaults each to null. Persistence handler in chat-settings-storage
mirrors typicalP's nullable-float handling for all 8.
Backend:
- 8 new ChatCompletionRequest fields with appropriate `ge`/`le`
validators (mirostat 0..2, ranges 0.0..1.0 where applicable).
- llama_cpp.py: signatures + payload forwarding extended on all
three builders (chat-completion, agentic tool-loop, final-pass)
so the new fields survive the local tool-loop too. `is not None`
gating so defaults (e.g. mirostat=0) reach the wire only when the
caller explicitly opted in.
- routes/inference.py: _build_passthrough_payload extends to the
extended sampler chain; 3 call sites (generate_chat_completion,
generate_chat_completion_with_tools, _build_passthrough_payload)
forward each field from `payload.*`.
Frontend chat-adapter: external branch forwards only when capability
allows (so OpenRouter gets top_a but not mirostat, local gets mirostat
but not top_a); local branch forwards unconditionally when the value
is meaningful (e.g. mirostat != 0, dynatemp_range > 0).
Test pinning the new field round-trip through _build_passthrough_payload
added; full PR-touched suite now 163 passing (was 161).
References:
- llama.cpp server params: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
- OpenRouter params: https://openrouter.ai/docs/api/reference/parameters
- LiteLLM provider params: https://docs.litellm.ai/docs/completion/input
Two follow-ups from a closer reading of each provider's published
sampling surface against llama.cpp's own server README.
typical_p (locally typical sampling, `typ_p` in the llama.cpp sampler
chain):
- New ProviderCapabilities.typicalP flag; defaults false on every
SaaS provider (none accept the field) and true only on the
permissive local buckets (custom, vllm, ollama, llama_cpp,
openrouter via ALL_SUPPORTED). InferenceParams.typicalP is
nullable number (null = unset; 1.0 = llama-server default, also
treated as no-op when forwarding).
- Backend: new ChatCompletionRequest.typical_p Field (0.0..1.0).
Threaded through all three llama_cpp.py payload builders
(chat-completion, agentic tool-loop, final-pass) so the field
survives the local tool-loop too. _build_passthrough_payload in
routes/inference.py picks it up and only writes the body when
the caller set a value; left absent it falls back to llama-server
default. Three route call sites (generate_chat_completion,
generate_chat_completion_with_tools, _build_passthrough_payload)
forward payload.typical_p.
- Frontend: chat-adapter forwards on both branches (external opt-in
only when capability allows + value != 1; local forwards
unconditionally when set and != 1). OpenAIChatCompletionsRequest
grows a `typical_p?` field. Persisted via chat-settings-storage
mirroring the seed nullable-float handler.
- Test: pin _build_passthrough_payload's forward + absent behavior.
DeepSeek per-model gating:
- deepseek-reasoner / deepseek-r1 silently ignore temperature, top_p,
presence_penalty, frequency_penalty per
https://api-docs.deepseek.com/guides/reasoning_model — mirror the
OpenAI / Claude 4.7 per-model approach: getProviderCapabilities
downshifts these ids to a stripped capability set so the panel
does not offer knobs the upstream silently drops.
161+1 sampling-routing tests pass; frontend tsc clean.
Refs:
- llama.cpp server params: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
- DeepSeek reasoner restrictions: https://api-docs.deepseek.com/guides/reasoning_model
* Studio: per-card web_search result + shell_call output fallback (OpenAI)
Two empty-output bugs in the OpenAI Responses tool-result rendering that
showed up clearly when a single prompt invoked 9 web_search + 4
code_execution + 1 image_generation in one turn. Reproduction shape in
the SQLite-stored chat history:
- 8 of 9 web_search tool-call records had result == "" (the cards
rendered as empty cards in the thread)
- 4 of 4 code_execution (shell_call) records were missing the result
key entirely (NoneType), so the cards that showed "Ran cat ..." style
commands displayed the command line but no output panel at all
- image_generation worked, as did the very last web_search of the run
Root causes in studio/backend/core/inference/external_provider.py:
1. web_search_call's tool_end emitted result: "" by design, with the
intent of overwriting only the LAST call at response.completed with
the full citation list (the source-pill extractor on the frontend
flatMaps across every web_search result, so a single non-empty
result is enough for the trailing source pills). Side effect: every
intermediate card renders empty in the thread. Fix: seed each call's
own tool_end result with "Searching: <query>" so the per-card text
is never empty, then keep the last-call overwrite path so the
source-pill extractor still works. Falls back to empty when the
model emits an action with no query, so the existing last-call path
stays unchanged for that edge.
2. shell_call's tool_start was emitted from
response.output_item.done for the call item, but tool_end lived in
the separate response.output_item.done handler for shell_call_output.
When OpenAI's Responses stream bundles the output array onto the
shell_call item's own done event (no separate shell_call_output
item), the previous handler emitted tool_start with no following
tool_end. The card spun on "running" indefinitely and stored as
NoneType in the thread DB. Fix: when the shell_call's done event
carries an embedded output list, emit tool_end immediately from
that. Track tool_end_emitted on the shell_calls map so a subsequent
shell_call_output event (some streams ship both) is skipped instead
of double-completing the card. A final flush at response.completed
emits tool_end for any orphan shell_call that received neither
bundled output nor a separate output event, so cards always finalise.
Tests (studio/backend/tests/test_openai_tool_result_fallbacks.py, 6
new):
- web_search: three calls, each card's result is its own Searching:
query (no empties)
- web_search: last call still gets the aggregated citation block when
url_citations arrive (pins the overwrite path)
- web_search: empty action.query falls back to result == "" (no junk
Searching: placeholder)
- shell_call: bundled output on done emits a single tool_end with that
output as the result text
- shell_call: bundled-then-separate output does not double-emit
tool_end (subsequent shell_call_output is skipped)
- shell_call: orphan call with neither bundled nor separate output is
flushed at response.completed so the card finalises
15/15 tests green when combined with the existing 9 in
test_openai_code_execution.py. Pre-commit + ruff format clean.
Scope: OpenAI Responses-API code path only. The Anthropic native
Messages-API path (_stream_anthropic) is untouched, as is the local
llama-server path. Local-model behaviour cannot regress because the
edited handlers only fire inside the OpenAI cloud branch.
* Studio: per-model external max_tokens cap + clamp on model switch
Two related external-provider issues that surfaced from the same
investigation as the per-card web_search / shell_call result bugs in
the previous commit:
A. Slider cap was a one-size-fits-all 32768 for every external model.
provider-capabilities.ts kept a single EXTERNAL_MAX_OUTPUT_TOKENS
constant (32k), well below what most providers actually accept. The
docstring even called out the right per-provider numbers (Anthropic
Opus 128k, GPT-5.x ~128k, Gemini 2.5 ~65k, DeepSeek 8k) but the
code picked the lowest as a conservative floor. Effect: long
generations from gpt-5.5 / claude-opus-4-7 silently truncated at
32k even though the API would have served up to 128k.
Fix: introduce getExternalMaxOutputTokens(providerType, modelId)
returning the documented per-model cap. Patterns are checked
longest-first so e.g. gpt-5.5-pro matches before gpt-5.5. Unknown
provider/model combinations fall back to the existing 32k floor so
no surprise increases for ids we don't know about.
Per-model caps from the official docs:
- OpenAI gpt-5.5 / gpt-5.5-pro: 128000
- OpenAI gpt-5.4 / gpt-5.4-pro: 65536
- OpenAI gpt-5.3: 16384
- Anthropic claude-opus-4-7: 128000
- Anthropic claude-opus-4-6 / sonnet-4-6 / opus-4-5 / sonnet-4-5 /
haiku-4-5: 64000
- Gemini 3.x family: 65535
- DeepSeek: 8192
- OpenRouter: strip provider/ prefix from the id and re-resolve
The slider in chat-settings-sheet.tsx and the send-time clamp in
chat-adapter.ts both call the new function so the slider's max=
matches what the wire layer will accept.
B. Slider value lied after switching from a local model to external.
When Studio auto-loads the helper Gemma-4-E2B-it on first chat,
chat-adapter sets params.maxTokens to Gemma's context_length
(262144 for Gemma 4). Switching the model picker to gpt-5.5 then
flips the slider's max prop to the external cap, but the stored
params.maxTokens is never reset. The numeric value next to the
slider would render 262144 against a track that ended at the
external cap. The send-time clamp brought the outbound max_tokens
back down to the cap, so the API call was safe, but the displayed
number had no relationship to what was actually being sent.
Fix: chat-runtime-store.setCheckpoint now clamps params.maxTokens
to getExternalMaxOutputTokens(...) on transitions into an external
model. Looks up the provider via useExternalProvidersStore so we
can derive providerType from the parsed external model id. No-op
when the stored maxTokens is already at or below the new cap, so
user-tuned values within range survive the switch.
Scope: pure frontend changes scoped to external-provider code paths.
Local model behaviour is untouched -- the ggufContextLength branch of
the slider's max= is unchanged, and setCheckpoint only mutates
maxTokens when isExternalModelId(modelId) is true. The send-time
clamp continues to be the safety net for any in-flight request that
crosses a model switch before the store-level clamp has applied.
Typecheck (tsc -b) clean; bun run build succeeds (2.13s).
Co-changes with the previous commit (7fe1adbf, per-card web_search +
shell_call output fallback) form a single PR: every empty-output and
silent-truncation issue surfaced from the same animal-popularity
prompt reproduction is now addressed in one branch.
* Studio: correct external max_tokens caps for Gemini and DeepSeek
Per-doc corrections to the per-model cap table added in 95da8d52:
- Gemini 3.x family: 65535 -> 65536, per
https://ai.google.dev/gemini-api/docs/models/gemini-3.1-pro-preview
(the published max_output_tokens is exactly 64K = 65536). The earlier
65535 was an off-by-one rough cap.
- DeepSeek (deepseek-chat / deepseek-reasoner aliases): 8192 -> 384000,
per https://api-docs.deepseek.com/quick_start/pricing. DeepSeek V4
Flash / Pro both list MAX OUTPUT = 384K; the chat / reasoner ids are
deprecated aliases for V4 Flash non-thinking / thinking modes. The
8192 value was carried over from V3 and silently truncated V4 traffic
at 2% of its actual ceiling.
Affects only the slider max and the send-time clamp for these provider
types. Other providers' caps unchanged. tsc -b clean.
* Studio: also flush orphan shell_calls on response.incomplete
Addresses gemini-code-assist[bot] high-priority inline review on PR
5785: the orphan-shell_call final flush added in 7fe1adbf landed only
in the response.completed branch. Truncated OpenAI Responses streams
emit response.incomplete instead (for example when the request hits
max_output_tokens), which left in-flight shell_call cards spinning
indefinitely in the UI.
Mirror the same flush block in the response.incomplete handler so the
truncated-stream path finalizes every pending tool card. The
tool_end_emitted guard keeps the path idempotent: if a shell_call
already completed via bundled output on its done event, the incomplete
flush is a no-op for it.
Two new tests in test_openai_tool_result_fallbacks.py:
- test_shell_call_flushed_on_response_incomplete_truncation pins the
bug repro: an in-flight shell_call followed by response.incomplete
must emit tool_end so the card finalizes.
- test_shell_call_incomplete_does_not_double_emit pins idempotency:
a shell_call that completed via bundled output and is then followed
by response.incomplete emits exactly one tool_end with the bundled
result text.
17/17 tests green (8 fallback tests + 9 existing code-execution). Pre-
commit + ruff format clean.
* Studio: trim verbose comments across PR 5785 edits
Compress the in-code commentary added across this branch to one or two
lines per block; the verbose prose was easier as a PR description than
as inline noise. No behavioural changes: 17/17 tests still green, tsc -b
still clean.
* feat(recipes): round-trip local model variants
* feat(recipes): add local model selector
* feat(recipes): wire selector into model editors
* fix(recipes): clear stale model state on relink
* feat(recipes): load selected local models for jobs
* chore(frontend): simplify biome scripts
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(recipes): handle local selector edge cases
* fix(recipes): polish local model selector behavior
* fix(recipes): delay local model restore until terminal runs
* fix(recipes): accept resolved default gguf variants
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Resolve 5 conflicts where main added Anthropic Opus 4.6/4.7 fast-mode
support that touches the same signatures PR 5711 extended:
studio/backend/core/inference/external_provider.py
Keep both PR 5711 sampling fields (frequency_penalty, seed, stop,
service_tier, parallel_tool_calls) and main's fast_mode in the
stream signature, docstring, and _stream_anthropic call site.
studio/backend/models/inference.py
Append fast_mode Field alongside PR 5711's new ChatCompletionRequest
fields; both flow through the existing dispatch.
studio/backend/routes/inference.py
Forward fast_mode and the PR 5711 sampling fields to the stream
generator together.
studio/frontend/src/features/chat/types/api.ts
Add fast_mode? to OpenAIChatCompletionsRequest after the PR 5711
field block.
studio/frontend/src/features/chat/utils/chat-settings-storage.ts
Persist fastMode alongside seed / stop / serviceTier /
parallelToolCalls.
No semantic changes to either feature surface. 161 backend routing
tests still pass; frontend tsc clean.
* Studio: surface Anthropic document citations inline + in Sources panel
Anthropic's Messages API streams ``citations_delta`` events on
``content_block_delta`` when the request enables
``citations: {enabled: true}`` on document blocks. Each event carries
one citation pointing at the source document; previously they were
silently dropped, so reader-visible references never reached the chat
UI even when the model was citing properly.
The proxy now:
- dedupes by the type-specific anchor (char_location / page_location /
content_block_location / search_result_location) so re-cites of the
same span collapse onto a single footnote;
- injects ``[N]`` inline right after the matching text run;
- forwards the full list as a synthetic ``document_citations``
tool_event at ``message_stop`` so the Sources panel can render
per-document footnotes next to web_search / web_fetch citations.
Streams that never emit ``citations_delta`` stay byte-identical.
References:
- https://platform.claude.com/docs/en/build-with-claude/citations
- https://platform.claude.com/docs/en/build-with-claude/search-results
Tests (5 in test_anthropic_citations.py): passthrough, single
char_location, dedup of repeat citations, distinct sources get
distinct numbers, search_result_location supported.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: surface Anthropic document_citations in the Sources panel
The PR added a backend _toolEvent.type='document_citations' on
message_stop and an inline [N] marker in the assistant text, but the
chat-adapter only handles container_*/tool_*/sources from
web_search and web_fetch tool calls. Reviewers flagged that the
inline [N] markers had no matching footnote entries in the Sources
panel.
Capture the new event into a documentCitationParts buffer, convert
each citation dict into a Sources-panel source entry (using
document_title or search-result source URL plus cited_text as the
snippet), dedupe by id, and append to the final yield alongside
the existing web_search/web_fetch sourceParts.
* Studio: dedupe search_result_location citations by search_result_index
Anthropic's documented search_result_location citation shape carries
search_result_index, source, title, and start/end_block_index --
NOT document_index/document_title. The previous key keyed on
document_index + document_title + source + start_block_index, so
two distinct search results from the same source collapsed onto the
same footnote and the second [N] marker was lost.
Switch the search_result_location branch to key on the documented
fields, and pin the behaviour with a regression test asserting that
two citations sharing source/title but with different
search_result_index get distinct [1] [2] markers.
* Studio: keep each citation distinct across the end-anchor
Codex follow-ups on the citations PR:
* Backend _anthropic_citation_key now includes the end anchor for
every variant (end_char_index, end_page_number,
end_block_index). Anthropic ranges are start-AND-end pairs, so
a same-start / different-end pair is two distinct citations
that previously collapsed onto one footnote.
* Frontend documentCitationToSource ids include the position
fields (search_result_index, start/end char/page/block) instead
of being keyed on URL alone. Two citations from the same
document or two search_result_locations with the same source
now produce distinct Sources-panel entries, matching the
inline [N] numbering.
* Studio: key Sources list by per-citation id instead of url
Codex flagged that the Sources renderer keys badges on source.url,
so two Anthropic document citations sharing the same source URL
collide as React keys and one badge gets dropped (or duplicated).
The chat-adapter already mints a per-citation id that folds the
position fields (search_result_index, start/end char/page/block)
into the URL, so the two citations have distinct ids even when
their URL matches. Plumb that id through SourceData and use it as
the React key for both the measurement badges and the visible
SourceBadge list. Falls back to the URL when no id is supplied
(web_search and web_fetch source parts).
* Studio: enable Anthropic doc citations on input_document blocks
Plumb citations: {enabled: true} onto the translated Anthropic document
block (both base64 and URL source branches) so the upstream actually
emits citations_delta events. Without this opt-in the inline [N] +
Sources panel plumbing added in this PR is a no-op for real user
PDF / doc uploads.
Refs https://platform.claude.com/docs/en/build-with-claude/citations
Also add edge-case coverage for the citations_delta path:
malformed citations, mixed types per document, reversed indices,
missing document_index, non-int block indices, unknown citation
type, internal _key never leaking, footnote numbering across
content blocks, and the input_document wire-through itself.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Reject unsafe citation sources, bound cited_text payload
Three follow-ups on top of #5718 surfaced by a deeper review pass:
1) javascript: / data: / vbscript: in citation source is XSS-able.
``documentCitationToSource`` was assigning ``cit.source`` straight
into ``Source.url`` and rendering it as an <a href>. A hostile
model emitting ``cit.source = "javascript:alert(document.domain)"``
would execute on click (openLink only intercepts URLs that contain
"://" or start with "mailto:", which both miss the javascript:
scheme). Restrict the navigable path to http(s):// only; anything
else falls back to the existing #anthropic-doc anchor and the
source title still renders the raw identifier for context. Also
reject CR/LF inside the URL string.
2) Frontend sources collapse distinct backend footnotes when the
citation type differs but positions match. char_location(0,5) and
page_location(0,5) over the same source previously deduped into
one entry because the id only carried position. Fold citation
type into the id anchor so the 1:1 mapping with inline [N]
markers is preserved across every citation shape.
3) ``cited_text`` was forwarded unbounded inside the synthetic
document_citations tool_event. The Sources panel trims to 240
chars for display anyway; for large RAG / search_result spans
(~10kB cited_text is plausible) this inflates SSE bytes 40x
for no UI benefit. Truncate server-side at 512 chars with an
ellipsis so the description-trim downstream still has room to
work and the wire stays bounded.
Tests grow from 21 to 22; existing 7 + edge 15 still green. Frontend
typecheck clean.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: apply http(s) URL guard to all Sources-panel link sources
The previous round only filtered ``cit.source`` inside
``documentCitationToSource``. Two parallel code paths still copied
provider/tool-controlled ``URL:`` text directly into clickable
``<a href>`` Sources-panel links:
* ``parseSourcesFromResult`` in chat-adapter.ts (legacy web_search /
web_fetch tool result parser)
* ``parseSearchResults`` in tool-ui-web-search.tsx (inline tool card)
A hostile tool response like ``URL: javascript:alert(1)`` or
``URL: data:text/html,...`` was therefore still rendered as a
navigable badge in the Sources panel.
Centralise the safe-URL test (``isSafeNavigableSourceUrl``,
``isSafeHttpUrl``) using ``new URL()`` + protocol allowlist + CR/LF
rejection, and apply it to both parsers. Unsafe blocks are dropped
rather than rewritten to a hash anchor because the web_search /
web_fetch parsers have no document-index fallback.
Citation conversion now uses the same helper so the in-place
http(s) regex and CR/LF check stay in one place.
* Shorten citation comments for PR #5718
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: surface Anthropic web_fetch as a standalone Fetch pill
web_fetch used to be silently bundled with the Search pill on the
assumption that "search returns URLs, fetch reads them" is the
typical workflow. Two problems with that:
- Anthropic bills each web_fetch invocation separately from
web_search hits, so combining them made the per-message cost
surface ambiguous.
- It blocked "just fetch this one URL" workflows where the user
already knows the page they want read and does not want a search
round-trip.
Adds:
- `webFetchToolsEnabled` to the chat-runtime-store, persisted to
localStorage under `unsloth_chat_web_fetch_tools_enabled`, with a
matching `supportsBuiltinWebFetch` capability flag and a
`setWebFetchToolsEnabled` setter.
- A new Fetch pill in the chat composer, rendered next to Images and
only when the active provider returns true from
`providerSupportsBuiltinWebFetch` (Anthropic today). The pill
defaults off so per-fetch billing is always a deliberate opt-in.
- chat-page bootstraps `webFetchToolsEnabled` from the same stored-
preference fallback the other pills use.
- chat-adapter reads `webFetchToolsEnabled` directly when deciding
whether to append "web_fetch" to `enabled_tools`, decoupling it
from `toolsEnabled` (Search).
Backend translation is unchanged: when `enabled_tools` already
contains "web_fetch", `_stream_anthropic` appends the
`web_fetch_20250910` / `web_fetch_20260209` tool exactly as before
(test_anthropic_web_fetch.py pins the standalone-only path at
`test_web_fetch_tool_appended_to_request_body` and the combined
path at `test_web_fetch_combined_with_web_search_and_code_execution`).
Frontend tsc passes.
* ci: re-trigger after transient GitHub API HTTP flake (checkout + ggml-org release fetch)
* Studio: include web_fetch in the disabled-tool guard axis
Reviewer P1 / High on PR #5742 (codex + gemini): after introducing
the standalone Fetch pill, `disabledToolGuard` still only branched on
`webSearchEnabledForThisTurn`. With Fetch ON and Search OFF the
system prompt would tell Claude "you do not have web search or web
fetch tools in this conversation", which contradicts the actual tool
schema being sent and suppresses `web_fetch` tool calls, defeating
the standalone-fetch workflow this PR adds.
Treat search and fetch as a single "any web tool enabled" axis. The
guard only needs to warn the model when no web tool is wired in for
this turn; once either pill is on the model can pick the right one
from the tool schema. The existing `webLabel` already covers both
names, so the user-visible guard text stays accurate in every
combination.
tsc clean.
* ci: re-trigger after transient infra flake on Windows prebuilt / actions/checkout
* Studio: route web_fetch through per-model version dispatch
The web_fetch tool body in `_stream_anthropic` hardcoded
`web_fetch_20250910` instead of calling `_anthropic_web_fetch_version`,
so Opus 4.6 / 4.7 and Sonnet 4.6 missed the `web_fetch_20260209`
dynamic-filtering variant. The picker, the unit tests for it, and a
deliberate "follow-up" note in `test_anthropic_web_fetch.py` already
existed; this just threads it through the emission site.
Mirrors how web_search and code_execution are dispatched per model.
Old models still resolve to `web_fetch_20250910` and continue to work.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shorten web_fetch comments for PR #5742
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: add Anthropic fast_mode toggle + surface streaming refusals
Fast mode (beta `fast-mode-2026-02-01`) lets Claude Opus 4.6 and 4.7
generate output tokens up to 2.5x faster at 6x standard Opus
pricing. The toggle lives in Configuration → Provider when the
selected Anthropic model is Opus 4.6 or 4.7 and is otherwise
hidden. Backend gates the same prefixes a second time so a stale
frontend cannot make Anthropic 400 the request, and the
`fast-mode-2026-02-01` beta header is merged onto whatever other
betas the request already needed (code-execution, compaction).
Streaming refusals (`message_delta.delta.stop_reason="refusal"` on
Claude 4 models) now surface a short user-facing notice in the
assistant message before the translated OpenAI chunk emits the
existing `finish_reason="content_filter"`. Previously the chat
bubble truncated silently because the SSE stopped mid-stream with
no visible explanation. Per the upstream docs the conversation
must be reset before continuing, so the notice tells the user
exactly that.
Reference:
- https://platform.claude.com/docs/en/build-with-claude/fast-mode
- https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/handle-streaming-refusals
Tests:
- studio/backend/tests/test_anthropic_fast_mode_and_refusal.py (8 cases
pinning fast_mode pass-through on 4.6/4.7, silent drop on Sonnet /
Haiku / older Opus / None / False, and the refusal notice + finish
reason on a synthetic refusal stream).
* Studio: drop refused Anthropic turns from the next request
Anthropic's streaming-refusal guidance says the refused assistant
turn must be removed or updated before the next call -- otherwise
the safety classifier keeps refusing. The PR only added a
user-visible notice; the partial assistant output (plus the notice
itself) still rode the next request via toOpenAIMessage.
Tag the refusal turn with an HTML-comment sentinel emitted alongside
the notice. The chat-adapter checks for that sentinel in
toOpenAIMessage and returns null, so the refused turn is excluded
from outboundMessages. The notice still renders in the transcript
(HTML comments don't display), so users keep the explanation.
* Studio: filter None finish_reason entries in test helper
test_refusal_maps_to_content_filter expects only ['content_filter']
in the finish_reasons list, but the post-PR refusal path emits a
user-visible content notice chunk first. Every _content_chunk
carries 'finish_reason: None' by construction; the helper was
appending those, so the assertion saw [None, 'content_filter']
instead of ['content_filter'].
None is not a finish reason -- it's just mid-stream delta noise.
Skip None values in _finish_reasons so the helper reflects what
the test names actually claim to check. Same fix applies cleanly
to the other helper usages (pause_turn test expects [] and the
sibling stop test expects ['stop'], both unaffected).
* Studio: cover Anthropic fast-mode edge cases
Adds 19 cases on top of the 9 in test_anthropic_fast_mode_and_refusal.
The base file pins the happy path; this file fills in the cliffs:
* Dated-snapshot prefix matching: claude-opus-4-7-2026-02-01 and
claude-opus-4-6-2026-02-01 still gate fast_mode through, while
claude-opus-4-5-2025-08-01 and claude-sonnet-4-6-2026-02-01 do not.
* Strict opt-in: a future claude-opus-4-8 or claude-opus-5 does NOT
auto-enable fast_mode -- the prefix tuple must be bumped explicitly
when a new family is whitelisted upstream.
* Beta-header merge: fast_mode coexists with code-execution-2025-08-25
and compact-2026-01-12 in one comma-separated anthropic-beta header
with no duplicates and no truncation. Pins the value to the exact
fast-mode-2026-02-01 docs token so a typo would fail CI.
* Non-destruction: fast_mode=None produces byte-identical outbound
body and headers to the version that omits the argument entirely.
Same for fast_mode=False. Guarantees the upgrade path is
non-breaking on existing Anthropic streams.
* Refusal stream ordering: the user-visible notice precedes the
finish_reason chunk so a streaming UI paints text before flipping
to content_filter. Refusal sentinel emitted exactly once. Notice
rides a normal content delta chunk with finish_reason still null.
Partial assistant deltas survive before the notice.
* Provider-side refusal coverage: a refusal on Sonnet (not just Opus)
still emits the notice + sentinel + content_filter mapping, since
refusal handling is not gated on fast-mode capability.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Persist fastMode, drop refused user message on retry
Two follow-ups on #5715:
1) sanitizeInferenceParams stripped fastMode. fastMode is in
PERSISTED_INFERENCE_PARAM_KEYS but the storage sanitizer only kept
numeric fields plus systemPrompt and trustRemoteCode, so the new
toggle was silently dropped on reload and on the
/api/chat/settings round-trip. Save it the same way trustRemoteCode
is saved.
2) Refusal recovery now also drops the triggering user turn.
Returning null from toOpenAIMessage on the assistant side left the
user prompt that caused the refusal in the outbound history, so
the very next request would re-trigger the same classifier.
Anthropic's refusal-handling guidance is explicit on this: remove
the refused turn AND the user message that triggered it before
the next call. Implemented via a pre-pass that pops the trailing
user message when an assistant carries the refusal sentinel.
Typecheck clean.
* Studio: out-of-band refusal signal + fast-mode prefix/usage/pricing fixes
The text sentinel for the Anthropic refusal drop signal was spoofable:
any assistant message containing the literal
<!--studio:anthropic-refusal--> would prune the prior user + assistant
pair on the next request. Move the signal onto a separate _toolEvent
chunk that the chat adapter latches into
assistant.metadata.custom.anthropicRefusal; assistant text can no
longer control the pruner.
Tighten the fast-mode model gate (backend + frontend) to require a "-"
family boundary so claude-opus-4-70 / claude-opus-4-7b style IDs do
not get speed: "fast" on a naive startswith match.
Use survivingMessages for the image / audio attachment scan so a
refused user turn does not gate or mis-attribute the next non-refused
turn.
Propagate Anthropic usage.speed onto the OpenAI-style usage chunk and
apply the documented 6x fast-mode multiplier in the cost calculator
(stacks with prompt-cache multipliers per the docs); expose the new
multiplier on the pricing snapshot for the UI tooltip.
Tests cover the tool-event chunk shape, the prefix-collision rejects,
usage.speed propagation, the 6x pricing math, and that the visible
refusal text carries no embedded sentinel.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shorten fast-mode and refusal comments for PR #5715
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: surface external-provider cache hits and writes in context bar
The Anthropic / OpenAI Responses streaming paths already emit an
include_usage-style SSE chunk carrying prompt_tokens_details.cached_tokens
and cache_creation_input_tokens / cache_read_input_tokens (see
_build_usage_chunk in external_provider.py), but the chat-adapter only
read the local llama-server timings.cache_n field. As a result, the
context-usage tooltip never showed cache hits or writes for external
providers, even though the backend was computing them.
Read the external usage envelope as a fallback when timings.cache_n is
absent, and surface Anthropic cache_creation_input_tokens as a separate
"Cache writes" line in the tooltip so users can tell a cache miss from a
cache hit on a turn that both reads and writes the cache.
- ServerUsage gains optional prompt_tokens_details.cached_tokens,
cache_creation_input_tokens, cache_read_input_tokens.
- contextUsage store entry gains optional cacheWriteTokens.
- ContextUsageBar gains optional cacheWrites tooltip line.
- chat-page wires both fields through to the bar.
* Studio: render cache stats for external providers too
Reviewer round on the original PR caught three asymmetric-fix sites
where the producer side surfaced external prompt-cache stats but the
consumer side still gated on ggufContextLength (which is only ever set
for the local llama-server runtime). Result: the entire cache-stats
PR shipped invisible for Anthropic / OpenAI Responses / Gemini, which
is exactly the set of providers it was added for.
- chat-page.tsx: drop the ggufContextLength precondition on the
ContextUsageBar mount. The bar already tracks usage; let it decide
what to render based on what it knows.
- context-usage-bar.tsx: make `total` optional. When absent, drop the
"/ total" ratio + percentage progress bar + "approaching limit"
helper, and just show per-turn counters + cache stats. Bootstrap
guard tightened so an all-zero, all-undefined state still renders
nothing.
- runtime-provider.tsx: external-provider rehydration was rejected by
the `store.ggufContextLength` check. Keep the "fits inside window"
sanity check when a local context window IS known, drop it when
it isn't.
- message-timing.tsx: the per-message timing popover used a separate
"Cache hits" code path that only read llama-server's timings.cache_n.
Fall through to custom.contextUsage for external providers, and add
a parallel "Cache writes" line for Anthropic cache_creation events.
* Studio: tighten cache-stats comments
* Scope contextUsage to active checkpoint
Three follow-ups on #5736 so the relaxed external-provider render
gate does not show stale token / cache stats from a different model:
1) setCheckpoint now clears contextUsage on a real checkpoint
change. setActiveThreadId and clearCheckpoint already did this;
the most-traveled transition path (the user switching models from
the picker) leaked the prior turn's counts because they were never
cleared.
2) The external-selection branch in chat-page.tsx now also clears
contextUsage at the same time it nulls ggufContextLength /
activeNativePathToken. Without this an in-session switch from a
local model to an external provider would visibly carry the
previous local turn's counters into the new provider's bar.
3) exitCompare's rehydration is now scoped: restore the saved
usage only when the message's modelId matches the active
checkpoint AND, for local turns where a context window is known,
when the saved total fits inside that window. Without this the
bar could render a stale local-model usage on top of an external
provider, or an oversized usage object that exceeds the now-
active window.
Typecheck clean.
* Plug remaining stale-contextUsage paths
Follow-up to 042e0ac4 that catches four asymmetric-fix sites the
checkpoint-scoping pass missed:
1) setParams now also clears contextUsage on a real checkpoint
change. The local model load path in use-chat-model-runtime calls
setParams(mergeBackendRecommendedInference(...)) which mutates
params.checkpoint before refresh() eventually fires setCheckpoint;
the intermediate window rendered the previous model's counters
under the new checkpoint.
2) chat-adapter.ts setContextUsage on stream completion now gates on
the captured params.checkpoint still being active. A late
completion from provider A used to clobber the context bar after
the user switched to provider B mid-stream.
3) chat-page.tsx exitCompare rehydration no longer accepts a saved
modelId-stamped usage when the active checkpoint is empty. A user
who entered compare, cleared the model, and exited compare would
otherwise see the cleared model's stats reappear.
4) runtime-provider.tsx thread-load no longer restores legacy
unscoped usage (no modelId) unless a local context window is
known. With the relaxed external-provider render gate, old
pre-PR persisted messages without a modelId stamp could attach
their counts to an unrelated active provider.
Also switches message-timing.tsx cache-hit fallback from || to ??
so an explicit cache_n=0 is not replaced by a stale cachedTokens.
Typecheck clean.
* Shorten cache-stats comments for PR #5736
OpenAI gating was per-provider — the restrictive reasoning-class capability
applied to gpt-4o too, even though gpt-4o on /v1/responses still accepts
temperature / top_p / seed / frequency_penalty / presence_penalty. Anthropic
4.7 was the inverse: backend stripped temperature / top_p / top_k per-model
but the UI still showed the sliders, so moving a knob silently did nothing.
Split openai capabilities into OPENAI_REASONING_CAPABILITIES (current
restrictive set, used for gpt-5.x / o1 / o3 / o4) and OPENAI_CHAT_CAPABILITIES
(full sampling minus top_k and stop, used for gpt-4o and any non-reasoning id
from the registry). Mirror the backend _ANTHROPIC_4_7_SAMPLING_REMOVED regex
on the frontend so claude-(opus|sonnet|haiku)-4-7 hides temperature / top_p /
top_k in the panel instead of relying on backend strip. getProviderCapabilities
now takes an optional modelId so chat-settings-sheet and chat-adapter both
resolve the same per-model variant; no behavior change for unspecified modelId.
Verified live OpenAI / Anthropic docs:
- GPT-5 temperature must equal 1: platform.openai.com/docs/guides/reasoning,
community.openai.com/t/temperature-in-gpt-5-models/1337133
- GPT-4o accepts full sampling on Responses: docs.aimlapi.com gpt-4o ref,
OpenAI cookbook seed example
- Claude 4.7 sampling removed (400 on any non-default temperature/top_p/
top_k): platform.claude.com/docs/en/about-claude/models/whats-new-claude-4-7
160 backend routing tests still pass; frontend tsc clean.
Round 19 added scale to /v1/responses based on the openai-python
SDK type, but round 20 reviewers (3/10 against) and the round 18
aggregator both noted that the live OpenAI Responses reference
limits Responses service tiers to auto/default/flex/priority. The
PR contract in the original description also lists scale only for
Chat Completions, not Responses. Studio routes OpenAI through
Responses, so forwarding scale risks a 400 from the upstream and
exposes a picker option the API does not accept.
Restore the conservative drop behaviour: only documented Responses
tiers reach the wire; legacy scale settings still validate (the
ServiceTier Literal and chat-settings storage allowlist keep it
for forward-compat).
Round 19 reviewer consensus (3/10 plus an asymmetric-fix call-out
across rounds 8/9/12/14/17): the openai-python SDK ships
service_tier as Literal["auto","default","flex","scale","priority"]
on /v1/responses, and enterprise Scale Tier customers need to opt in
explicitly. Drop the defensive scale-filter on the backend and add
"scale" to the OpenAI picker option list so the field reaches the
wire when set. Other providers remain at auto/default per their docs.
Update the routing tests so `scale` lives in the forwarded-set fixture
and the dropped-set fixture only carries truly out-of-enum values
(Anthropic-only `standard_only`, typos, empty string).
Round 13 P1 finding: the Service tier picker rendered `null` as the
displayed `auto` and converted any explicit `auto` selection back to
`null`, so the chat-adapter's truthy guard then omitted `service_tier`
on the wire. For Anthropic the docs distinguish:
- omitting `service_tier` -> provider default
- `service_tier="auto"` -> opts into Priority Tier when available
- `service_tier="standard_only"` -> opts out
Drop the auto -> null conversion so the user's explicit pick reaches
the adapter and the wire reflects it. `null` still means "never set"
and falls through to the provider default; the existing serviceTier
allowlist already includes "auto" everywhere it matters.
Two related findings from round 12 reviewers:
1. The local GGUF tool loop in `generate_chat_completion_with_tools`
iterates every entry of `tool_calls` returned by llama-server, even
when the caller explicitly opted out of parallel tool calls. The
`parallel_tool_calls` flag is forwarded to llama-server, but llama
.cpp does not enforce it on every jinja template
(https://github.com/ggml-org/llama.cpp/issues/22043), so a model
that ignores the flag still ran multiple tools per turn. Cap
`tool_calls` to the first entry when the flag is False so the
client-side contract holds regardless of upstream behavior.
2. llama-server documents `parallel_tool_calls` as defaulting to FALSE
(https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md),
so the previous chat-adapter shape (forward only on explicit false)
meant the UI's default-on state could never enable parallel tool
calls there. Always forward the user's preference on the local
path so the toggle actually does what it says. External providers
default to true everywhere, so the external branch is unchanged.
Test pins the GGUF tool-loop cap by source-level assertion (the loop
itself is integration-only).
Round 11 P2 finding: the stop-sequence chips input rejected any draft
that strip to empty, which silently dropped pasted whitespace stops
like `"\n\n"` for blank-line halts. Local llama-server and OpenAI-
compat backends accept those; the Anthropic helper independently
filters whitespace entries before they hit the wire, so allowing them
in the UI cannot turn into a 400.
Drop the .trim() gate; reject only the truly empty draft. Single-line
Input behaviour is unchanged for the common typed-letters path.
Gemini exposes its OpenAI-compatible endpoint at
https://generativelanguage.googleapis.com/v1beta/openai. Google's own
docs (https://ai.google.dev/gemini-api/docs/openai) list the supported
parameters and inherit OpenAI's 4-entry stop cap. Without an explicit
`stop_max=4` on the registry the default 16 leaks through and the
upstream silently drops the overflow.
Add the backend registry entry, mirror it in the frontend
`PROVIDER_STOP_MAX` map, and pin the cap with a focused unit test.
The local non-external GGUF path (llama-server) accepts frequency_penalty,
seed, stop and parallel_tool_calls, but the safetensors / HF transformers
worker has no equivalent kwargs and silently drops them. Showing the
controls there has been confusing reviewers: the UI promises a knob that
does nothing.
Gate frequencyPenalty / seed / stop / parallelToolCalls on `isGguf` for
non-external local models so safetensors sessions only show controls the
backend actually honours. External-provider gating is unchanged.
Stale persisted values from a prior GGUF session are still sent on the
wire but the safetensors worker keeps absorbing them via **_unused, so
this is a presentation-only change with no behaviour difference.
5/10 reviewers in the last round flagged Kimi forwarding non-default frequency_penalty as a 400 risk for K2.5 / K2.6, mirroring the existing lock on temperature and top_p. Hide the slider on the frontend and add frequency_penalty to Kimi's body_omit so even stale clients have the field stripped before the request hits the wire.
service_tier on the generic OpenAI-compatible branch was forwarding whatever value the dispatcher received, so a stale frontend could send standard_only (Anthropic) or scale to providers like Mistral that do not document the field, producing 400s. Gate the forward on an explicit accepts_service_tier=True provider registry opt-in; Anthropic and OpenAI Responses already handle service_tier inside their own helpers.
OpenRouter normalises to OpenAI's chat schema and inherits the 4-entry stop cap. The default 16-cap was too permissive; add stop_max=4 on both the backend provider registry and the frontend PROVIDER_STOP_MAX map.
The GGUF tool-iteration final-answer pass at llama_cpp.py:5182 was carrying only the legacy sampling fields. Forward frequency_penalty, seed, and parallel_tool_calls there too so the cap-exhausted path matches the per-iteration loop.
Test pins the OpenRouter 4-cap.
Kimi's official Chat Completion schema at https://platform.kimi.ai/docs/api/chat does not list seed or parallel_tool_calls. Hide both controls so users are not offered settings the upstream may silently drop or 400 on. Frequency penalty, presence penalty, and stop sequences remain exposed because Kimi documents them with full ranges.
Round 5 review flagged two asymmetries:
1. Kimi web-search bypass hard-capped stops at 4 while the default OAI-compat path honours provider_info["stop_max"]. Apply the same provider-aware logic in _stream_kimi_web_search so kimi-with-search and kimi-without-search match. Also add Kimi's documented 5-stop max (https://platform.kimi.ai/docs/api/chat) to the provider registry so the cap actually fires.
2. chat-settings-sheet.tsx caps every non-Anthropic external provider at 4 stops. Replace with a per-provider getProviderStopMax helper in provider-capabilities.ts so DeepSeek, Mistral, and local backends are not artificially restricted while OpenAI Chat still hits its 4-entry hard limit and Kimi hits its documented 5-entry cap.
Tests pin the Kimi 5-cap on both Kimi paths.
Mistral chat completions uses random_seed not seed; map the field via a new seed_field on the provider registry so the new seed control actually works on Mistral. Default for other providers stays seed.
DeepSeek and Mistral both accept up to 16 stop sequences but the default OAI-compat branch was hard-capping at 4 (the OpenAI Chat limit). Studio routes the openai provider through /v1/responses not /v1/chat/completions so the 4-cap only applies if we explicitly added an openai entry. Raise the default to 16 and let per-provider stop_max overrides tighten if needed.
The local GGUF direct chat path (gguf_generate / gguf_generate_with_tools) bypassed _build_openai_passthrough_body and therefore dropped frequency_penalty, seed, stop, and parallel_tool_calls on the floor for users on the default no-tools and with-tools paths. Thread the new fields through LlamaCppBackend.generate_chat_completion and generate_chat_completion_with_tools and the two callsites that invoke them.
Also tighten comments to drop review-process narration that crept in and to remove the em dashes I had introduced in this PR's earlier commits.
Tests pin the Mistral random_seed rename, the DeepSeek 16-cap, and confirm the openai-compat default cap is 16.
Round 4 reviewer consensus (~9/20 independent reviewers) flagged
service_tier=scale as a 400 risk on /v1/responses. The earlier commit
added scale based on the openai-python SDK literal, but the live
OpenAI Responses API reference, the PR's own provider matrix, and the
9-reviewer round-4 consensus all agree the documented Responses enum
is auto|default|flex|priority only. Drop scale on this path to remove
the risk.
Keeps scale on the Chat Completions / OAI-compat path where the SDK
enum is honored and where users who want Scale Tier can still select
it. The widened TypeScript ServiceTier / ServiceTierOption / api.ts
union and the storage sanitizer allowlist remain permissive so legacy
persisted "scale" values do not get silently dropped on reload; the
runtime per-provider gate makes the routing decision.
Tests are updated to pin the restricted Responses enum and the
explicit drop of scale + standard_only + bogus values.
Round 3 reviewer feedback:
- studio/backend/routes/inference.py: _build_chat_request (the
/v1/responses → /v1/chat/completions translator) was dropping
parallel_tool_calls on the floor. A Responses-API caller that set
`parallel_tool_calls=false` saw the flag accepted at the schema
layer but never reach llama-server because the translated
ChatCompletionRequest had no first-class field for it. Now that
parallel_tool_calls IS a first-class field on ChatCompletionRequest
(added by this PR's earlier commits), translate it through the
bridge so the preference actually fires.
- studio/frontend/src/features/chat/utils/chat-settings-storage.ts:
the stop sanitizer silently dropped `stop: []` instead of persisting
the empty array. That meant a user could not clear the last chip —
on reload, the previously-persisted stops came back. Persist empty
arrays explicitly so the cleared state round-trips.
- studio/backend/tests/test_sampling_params_routing.py: pin both with
the raw reproductions reviewers cited.
Round 2 reviewer feedback:
- studio/frontend/src/features/chat/types/api.ts: `OpenAIChatCompletionsRequest.service_tier` did not include `"scale"`, so the request builder in chat-adapter.ts failed typecheck after the runtime ServiceTier union widened (`Type 'ServiceTier | undefined' is not assignable...`). Widen the type to match the SDK and keep the typecheck green.
- studio/frontend/src/components/ui/stop-sequences-input.tsx: the chip editor used `draft.trim()` for storage, which silently mutated semantically meaningful stops like " End", "### ", and "\n\n". Keep the whitespace-only rejection (Anthropic 400s on those, OpenAI silently drops them) but persist the raw draft so leading/trailing whitespace inside otherwise-meaningful stops survives.
- studio/backend/core/inference/external_provider.py: the Kimi web-search bypass dropped a single string `stop="\n\n"` via `stop.strip()` while the normal default OAI-compat path forwards it verbatim. Mirror the default path's behavior here so kimi-with-search and kimi-without-search apply the same rules (asymmetric provider-path fix flagged in round-2 review).
Round-2 round of review-feedback fixes for the sampling-knobs PR:
- studio/backend/routes/chat_history.py: ChatInferenceSettings still had
the pre-PR field list with extra="forbid", so every settings save the
new frontend issued would 422 on the new keys (frequencyPenalty,
seed, stop, serviceTier, parallelToolCalls). Add the fields with the
same range / enum constraints the chat-completions schema uses, so
the settings-persistence path round-trips cleanly.
- studio/backend/routes/inference.py: _build_passthrough_payload and
_build_openai_passthrough_body now thread frequency_penalty, seed,
and parallel_tool_calls through to llama-server. The frontend exposes
these knobs for local backends; without the forwarding the UI was a
decoration. Each field is gated on `is not None` so 0 / False / "0"
still reach the body.
- studio/backend/core/inference/external_provider.py: the Kimi
$web_search bypass takes an early return into _stream_kimi_web_search
before the default OAI-compat body builder runs, so the new sampling
fields never landed on Kimi-with-search. Forward them through the
helper, with the same dedupe / truncate behavior the main path
applies to `stop`. Also extend the OpenAI Responses service_tier
allowlist to include `scale` per the live openai-python SDK
(response_create_params.py declares
Literal["auto","default","flex","scale","priority"]).
- studio/frontend/src/features/chat/provider-capabilities.ts +
types/runtime.ts: add `scale` to ServiceTier / ServiceTierOption and
surface it on the OpenAI Responses options so the UI matches the
upstream enum.
- studio/backend/tests/test_sampling_params_routing.py: add tests for
every gap above: Kimi web-search bypass forwarding, local OpenAI
passthrough forwarding, ChatSettingsPayload round-trip, and the full
Responses service_tier enum (parametrized over the five accepted
values plus a drop check for the Anthropic-only standard_only).
Anthropic Messages API rejects `disable_parallel_tool_use` as a
top-level field; it is only accepted as a property on the `tool_choice`
object. Move the inversion into a tool_choice merge that defaults to
`{type:"auto"}` when no choice is supplied, and skip the field entirely
when no tools are defined (it is a no-op without tools).
The same path also dropped `stop` chips that contain only whitespace,
because Anthropic 400s with `each stop sequence must contain
non-whitespace` on entries like " ", "\n", and "\n\n". The previous
filter only dropped truly empty strings; switch to `s.strip()` so the
common newline-stop defaults are also filtered out client-side.
Frontend persistence had three round-trip data-loss bugs:
- `VALID_SERVICE_TIERS` was missing `standard_only`, so any Anthropic
user who picked that tier lost it on the next reload.
- The settings sanitizer truncated `stop` to 4 entries on save,
which defeated the Anthropic UI cap of 16. Use 16 here and let the
per-provider stream helper cap to the wire's allowed length.
- The chat-settings sheet's `stopMaxEntries` capped local backends
(llama.cpp / vLLM / ollama / generic OpenAI-compat) at 4 even
though those backends happily accept more. Match Anthropic's 16
for the local path.
Preset policy now carries `frequencyPenalty` and `stop` so a saved
preset can fix a user's preferred decoding style. `seed`,
`serviceTier`, and `parallelToolCalls` stay out of presets because
they are per-request determinism / per-provider account / per-tool
state, not reusable preset values.
Drops the test that pinned the buggy top-level placement of
`disable_parallel_tool_use` and adds two tests for the nested shape
plus the without-tools skip path, plus a test pinning the
whitespace-stop filter against the documented Anthropic error.
Codex P1: the runtime store added frequencyPenalty, seed, stop,
serviceTier, parallelToolCalls but the save/load path went through
sanitizeInferenceParams, which only whitelisted the older numeric set
plus systemPrompt / trustRemoteCode. The new keys were silently
stripped on save and dropped on reload.
Extend the whitelist:
- frequencyPenalty added to the numeric finite-number set.
- seed: integer or explicit null (null = "no seed field on the wire").
- stop: string array, capped at 4 entries per OpenAI's limit.
- serviceTier: nullable enum (auto/default/flex/priority/scale).
- parallelToolCalls: boolean.
- Drop `scale` from the OpenAI service-tier picker (frontend types and
picker option list). OpenAI in Studio routes through `/v1/responses`,
which does not accept `scale`; offering it in the UI silently
dropped the value at the backend and misled users into thinking
their selection was applied. Backend Literal still accepts it on
input so stale clients are not 422'd, and `_stream_openai_responses`
continues to drop it from the wire body.
- Dedupe + drop empty entries for OpenAI Chat `stop` and Anthropic
`stop_sequences` before forwarding so whitespace chips or accidental
repeats do not waste the 4-entry OpenAI cap or the 16-entry
Anthropic cap. Anthropic over-cap now logs and truncates, matching
the OpenAI path.
- Static `aria-label="Parallel tool calls"` on the Switch; screen
readers already announce checked / unchecked state, so the dynamic
Enable/Disable label was redundant.
- Forward an `aria-label` onto the inner Input inside
`StopSequencesInput` so screen-reader users can identify the field.
- Regression tests covering the new dedup, truncation, and the
preserved silent-drop of `scale` on Responses.
Adds the missing sampling parameters that the upstream APIs accept and
that Studio's chat UI previously hid. Each knob is gated per provider
so the picker never offers a field the upstream would 400 on, and the
per-provider stream functions translate / drop fields to match each
API's naming.
New `InferenceParams` fields (round-trip through PersistedInferenceParams
and the chat-settings server store automatically):
- frequencyPenalty (-2..2): OpenAI Chat Completions only.
- seed (int | null): OpenAI Chat + OpenAI-compat local backends.
- stop (string[]): all OpenAI Chat + Anthropic Messages. Backend
truncates to 4 entries on OpenAI Chat per docs and renames to
`stop_sequences` on Anthropic.
- serviceTier (auto|default|flex|priority|scale|standard_only):
per-provider enum sets resolved by getServiceTierOptions.
- parallelToolCalls (bool, default true): forwarded as
`parallel_tool_calls` on both OpenAI APIs and inverted into
`disable_parallel_tool_use` on Anthropic.
OpenAI Responses (gpt-5.x / o3) explicitly drops frequencyPenalty /
seed / stop alongside the existing temperature / top_p drop, since
the upstream 400s on all of them. service_tier on Responses accepts a
subset (no `scale`) which the dispatch already enforces.
UI rows land in the existing Sampling section of the chat settings
sheet using ParamSlider (frequency penalty), a numeric Input (seed),
a new chips editor `StopSequencesInput` (stop), Select (service tier),
and Switch (parallel tool calls). Each row's visibility follows the
new ProviderCapabilities flag.
Tests pin the gating contract: stop_sequences renamed on Anthropic,
4-entry truncation on OpenAI Chat, every Responses-rejected field
dropped, schema-level validation for the service_tier Literal and
frequency_penalty range.
Plan: plans/hashed-riding-porcupine.md
Bundles three independent CI regressions hitting the maintainer PR
backlog. Each one is verified end-to-end on a staging fork against
real Ubuntu / macOS / Windows GitHub-hosted runners before this
lands.
1. Windows --no-torch install: pydantic + pydantic-core drift to
incompatible versions under `uv pip install --no-deps -r
no-torch-runtime.txt` because pip resolves each independently
from latest. pydantic.VERSION 2.13.4 pins pydantic-core==2.46.4
but pydantic-core 2.47.0 was the freshest published wheel, so
`import pydantic` raised
`SystemError: pydantic-core 2.47.0 is incompatible with the
current pydantic version`. Resolve pydantic WITH deps in a
focused pip call (install.sh, install.ps1,
install_python_stack.py) before the --no-deps no-torch-runtime
pass so pip pins pydantic-core to the version pydantic declares.
pydantic's transitive deps (annotated-types, pydantic-core,
typing-extensions, typing-inspection) are torch-free. Drop the
redundant `Patch Studio venv with full typer / pydantic dep
trees` workaround from the four Windows smoke YAMLs.
Supersedes #5733 + #5734.
2. Linux Studio Update CI: upstream llama.cpp b9261+ split each
binary's entry code into a paired `libllama-<binary>-impl.so`
shared library. `llama-server` and `llama-quantize` NEEDED-link
against `libllama-server-impl.so` / `libllama-quantize-impl.so`
with RUNPATH `$ORIGIN`, so the prebuilt overlay must copy those
alongside the binaries. Without that, ldd reports them missing,
preflight rejects, the installer falls back to source build, and
studio-update-smoke annotates `setup.sh idempotency regressed`.
Add `libllama-*-impl.so*` to the Linux runtime patterns and lock
the pattern in test_rocm_support.TestRuntimePatterns.
3. Mac Studio UI Chat: change-password submit clicked while
disabled. The disable gate only checked new + confirm password
length, but Playwright's first click landed before the
current-password field's React state had committed, so the form
was simultaneously logically-invalid (current_password empty) and
the button was disabled. Tighten the gate to require
`currentPassword.length >= 8` and mirror the same check in the
submit handler so Enter / autofill cannot bypass.
Supersedes #5738.
The pill wired the request end of the loop but the response was lost
on the client: the backend emits a `tool_end` _toolEvent carrying the
base64 PNG on `image_b64` / `image_mime`, but the chat-adapter only
read the `result` string and the generic ToolFallback printed the
prompt as JSON args with an empty Result block -- the "I see no
image" symptom in the chat.
- chat-adapter: when the closing `tool_end` is for `image_generation`,
repackage `image_b64` + `image_mime` (+ size/quality/background)
into a structured result object instead of dropping them.
- New `ImageGenerationToolUI` reads that result and renders the image
inline via `<img src="data:image/...;base64,...">` with the prompt
as a caption. Falls back to a spinner while the request is still
running.
- Register the component under `image_generation` in thread.tsx's
tools.by_name map so it preempts ToolFallback for this tool only.
#5685 wired the backend to honor `prompt_cache_ttl` on the request,
but there was no UI to actually pick it -- every Studio chat ended up
on Anthropic's default 5 minute pool. This adds a Cache TTL selector
to the chat settings sheet's Provider section, visible only when the
provider supports the choice (Anthropic today) and Prompt caching is
on.
- New `promptCacheTtl?: "5m" | "1h"` on `ExternalProviderConfig`.
Normalizer drops the field on providers that don't support the
choice so localStorage stays clean across provider swaps.
- `supportsProviderPromptCacheTtl` + `isPromptCacheTtl` helpers so
the picker, normalizer, and adapter all agree on which values are
valid.
- Settings sheet renders a small Select (5 minutes / 1 hour) right
under the Prompt caching switch when the toggle is on; flipping
it persists on the provider config like the other per-provider
knobs.
- chat-adapter passes `prompt_cache_ttl` on outbound requests when
the value is valid; omitted otherwise so the backend keeps
inheriting Anthropic's 5m default.
The backend already wires OpenAI's Responses-API image_generation
server tool: when `enabled_tools` carries "image_generation" on an
OpenAI cloud request, _stream_openai_responses appends
`{type: "image_generation"}` to the request's tools array and emits
`image_generation_call` output items back to the assistant stream
(see backend/core/inference/external_provider.py and
backend/tests/test_openai_image_generation.py for the round-trip).
This wires the frontend half so a user can actually opt into it from
the composer next to the Search and Code pills, instead of the tool
sitting dormant.
- `providerSupportsBuiltinImageGeneration` gates on OpenAI cloud
(`api.openai.com`) + a Responses-API model prefix (gpt-5.x, o3).
Mirror of the backend's `is_openai_cloud` guard so the pill is hidden
on custom OpenAI-compat backends (ollama / llama.cpp / vLLM) that
report `provider_type="openai"` but would 400 on the tool.
- New `imageToolsEnabled` flag in chat-runtime-store, persisted under
`unsloth_chat_image_tools_enabled` and reset on model change in
chat-page exactly like `codeToolsEnabled`.
- `chat-adapter` appends "image_generation" to `enabled_tools` and
flips `enable_tools: true` when the pill is on, so the existing
backend dispatch picks it up.
- Composer renders an Images pill (lucide `ImageIcon`) immediately
after the Code pill, only when the active model advertises the
capability. The in-thread composer (assistant-ui/thread.tsx) gets
the matching `ImagesToggle` for parity.
The first pass only wired the localStorage mirror into `setCheckpoint`,
but the main chat-page picker actually selects an external model by
calling `setParams({ ...store.params, checkpoint: value })`. That path
never hit `setCheckpoint`, so the persisted slot stayed empty and a
refresh fell back to whatever `/api/inference/status.active_model`
returned -- the previously loaded local model (Qwen3.5 etc) or null
("Select model") when nothing was loaded locally.
Mirror the persistence in `setParams` whenever the checkpoint changes
so every entry point converges on the same behavior. `setCheckpoint`
still does it directly so the load path (compare, GGUF auto-load,
gemma fallback in chat-adapter) keeps working.
* Add Anthropic prompt guards for disabled tools
* fix: merge Anthropic tool guard into structured system prompts
* fix: scope Anthropic disabled-tool guard wording
* chore: adjust claude guard prompt
* chore: add openai to list of prompt guarded providers
* Studio: include web_fetch in the per-turn disabled-tool guard
Add webFetchEnabledForThisTurn alongside webSearchEnabledForThisTurn
and codeExecEnabledForThisTurn. Use it in the enabled_tools payload
so web_fetch follows the Search pill the same way web_search does,
and mention "web fetch" in the disabled-tool guard prose on providers
that ship the tool (Anthropic today; other providers stay inert via
providerSupportsBuiltinWebFetch).
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Selecting a connected external provider (Anthropic, OpenAI, Google, etc.)
and refreshing the page reverted the picker back to no selection. Root
cause is that `PersistedInferenceParams` in `chat-settings-api.ts`
excludes `checkpoint` from the server-side settings payload by design.
Local model selections survive refresh because the backend re-derives
them from `/api/inference/status.active_model`, but external selections
have no backend mirror, so they were lost.
Fix: persist `external::*` checkpoints to a small dedicated
`localStorage` key (`unsloth_chat_last_external_checkpoint`) and hydrate
from it on store init. Local checkpoints continue to come from the
backend status as before; only external ids are mirrored client-side.
`setCheckpoint` writes the key when an external id is selected and
clears it when switching back to a local id, and `clearCheckpoint`
clears it so the picker does not snap back after an explicit reset.
Daniel Han