ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Studio: per-card web_search result + shell_call output fallback (OpenAI) (#5785)

Browse source 
* Studio: per-card web_search result + shell_call output fallback (OpenAI)

Two empty-output bugs in the OpenAI Responses tool-result rendering that
showed up clearly when a single prompt invoked 9 web_search + 4
code_execution + 1 image_generation in one turn. Reproduction shape in
the SQLite-stored chat history:

- 8 of 9 web_search tool-call records had result == "" (the cards
  rendered as empty cards in the thread)
- 4 of 4 code_execution (shell_call) records were missing the result
  key entirely (NoneType), so the cards that showed "Ran cat ..." style
  commands displayed the command line but no output panel at all
- image_generation worked, as did the very last web_search of the run

Root causes in studio/backend/core/inference/external_provider.py:

1. web_search_call's tool_end emitted result: "" by design, with the
   intent of overwriting only the LAST call at response.completed with
   the full citation list (the source-pill extractor on the frontend
   flatMaps across every web_search result, so a single non-empty
   result is enough for the trailing source pills). Side effect: every
   intermediate card renders empty in the thread. Fix: seed each call's
   own tool_end result with "Searching: <query>" so the per-card text
   is never empty, then keep the last-call overwrite path so the
   source-pill extractor still works. Falls back to empty when the
   model emits an action with no query, so the existing last-call path
   stays unchanged for that edge.

2. shell_call's tool_start was emitted from
   response.output_item.done for the call item, but tool_end lived in
   the separate response.output_item.done handler for shell_call_output.
   When OpenAI's Responses stream bundles the output array onto the
   shell_call item's own done event (no separate shell_call_output
   item), the previous handler emitted tool_start with no following
   tool_end. The card spun on "running" indefinitely and stored as
   NoneType in the thread DB. Fix: when the shell_call's done event
   carries an embedded output list, emit tool_end immediately from
   that. Track tool_end_emitted on the shell_calls map so a subsequent
   shell_call_output event (some streams ship both) is skipped instead
   of double-completing the card. A final flush at response.completed
   emits tool_end for any orphan shell_call that received neither
   bundled output nor a separate output event, so cards always finalise.

Tests (studio/backend/tests/test_openai_tool_result_fallbacks.py, 6
new):
- web_search: three calls, each card's result is its own Searching:
  query (no empties)
- web_search: last call still gets the aggregated citation block when
  url_citations arrive (pins the overwrite path)
- web_search: empty action.query falls back to result == "" (no junk
  Searching: placeholder)
- shell_call: bundled output on done emits a single tool_end with that
  output as the result text
- shell_call: bundled-then-separate output does not double-emit
  tool_end (subsequent shell_call_output is skipped)
- shell_call: orphan call with neither bundled nor separate output is
  flushed at response.completed so the card finalises

15/15 tests green when combined with the existing 9 in
test_openai_code_execution.py. Pre-commit + ruff format clean.

Scope: OpenAI Responses-API code path only. The Anthropic native
Messages-API path (_stream_anthropic) is untouched, as is the local
llama-server path. Local-model behaviour cannot regress because the
edited handlers only fire inside the OpenAI cloud branch.

* Studio: per-model external max_tokens cap + clamp on model switch

Two related external-provider issues that surfaced from the same
investigation as the per-card web_search / shell_call result bugs in
the previous commit:

A. Slider cap was a one-size-fits-all 32768 for every external model.

   provider-capabilities.ts kept a single EXTERNAL_MAX_OUTPUT_TOKENS
   constant (32k), well below what most providers actually accept. The
   docstring even called out the right per-provider numbers (Anthropic
   Opus 128k, GPT-5.x ~128k, Gemini 2.5 ~65k, DeepSeek 8k) but the
   code picked the lowest as a conservative floor. Effect: long
   generations from gpt-5.5 / claude-opus-4-7 silently truncated at
   32k even though the API would have served up to 128k.

   Fix: introduce getExternalMaxOutputTokens(providerType, modelId)
   returning the documented per-model cap. Patterns are checked
   longest-first so e.g. gpt-5.5-pro matches before gpt-5.5. Unknown
   provider/model combinations fall back to the existing 32k floor so
   no surprise increases for ids we don't know about.

   Per-model caps from the official docs:
   - OpenAI gpt-5.5 / gpt-5.5-pro: 128000
   - OpenAI gpt-5.4 / gpt-5.4-pro: 65536
   - OpenAI gpt-5.3: 16384
   - Anthropic claude-opus-4-7: 128000
   - Anthropic claude-opus-4-6 / sonnet-4-6 / opus-4-5 / sonnet-4-5 /
     haiku-4-5: 64000
   - Gemini 3.x family: 65535
   - DeepSeek: 8192
   - OpenRouter: strip provider/ prefix from the id and re-resolve

   The slider in chat-settings-sheet.tsx and the send-time clamp in
   chat-adapter.ts both call the new function so the slider's max=
   matches what the wire layer will accept.

B. Slider value lied after switching from a local model to external.

   When Studio auto-loads the helper Gemma-4-E2B-it on first chat,
   chat-adapter sets params.maxTokens to Gemma's context_length
   (262144 for Gemma 4). Switching the model picker to gpt-5.5 then
   flips the slider's max prop to the external cap, but the stored
   params.maxTokens is never reset. The numeric value next to the
   slider would render 262144 against a track that ended at the
   external cap. The send-time clamp brought the outbound max_tokens
   back down to the cap, so the API call was safe, but the displayed
   number had no relationship to what was actually being sent.

   Fix: chat-runtime-store.setCheckpoint now clamps params.maxTokens
   to getExternalMaxOutputTokens(...) on transitions into an external
   model. Looks up the provider via useExternalProvidersStore so we
   can derive providerType from the parsed external model id. No-op
   when the stored maxTokens is already at or below the new cap, so
   user-tuned values within range survive the switch.

Scope: pure frontend changes scoped to external-provider code paths.
Local model behaviour is untouched -- the ggufContextLength branch of
the slider's max= is unchanged, and setCheckpoint only mutates
maxTokens when isExternalModelId(modelId) is true. The send-time
clamp continues to be the safety net for any in-flight request that
crosses a model switch before the store-level clamp has applied.

Typecheck (tsc -b) clean; bun run build succeeds (2.13s).

Co-changes with the previous commit (7fe1adbf, per-card web_search +
shell_call output fallback) form a single PR: every empty-output and
silent-truncation issue surfaced from the same animal-popularity
prompt reproduction is now addressed in one branch.

* Studio: correct external max_tokens caps for Gemini and DeepSeek

Per-doc corrections to the per-model cap table added in 95da8d52:

- Gemini 3.x family: 65535 -> 65536, per
  https://ai.google.dev/gemini-api/docs/models/gemini-3.1-pro-preview
  (the published max_output_tokens is exactly 64K = 65536). The earlier
  65535 was an off-by-one rough cap.
- DeepSeek (deepseek-chat / deepseek-reasoner aliases): 8192 -> 384000,
  per https://api-docs.deepseek.com/quick_start/pricing. DeepSeek V4
  Flash / Pro both list MAX OUTPUT = 384K; the chat / reasoner ids are
  deprecated aliases for V4 Flash non-thinking / thinking modes. The
  8192 value was carried over from V3 and silently truncated V4 traffic
  at 2% of its actual ceiling.

Affects only the slider max and the send-time clamp for these provider
types. Other providers' caps unchanged. tsc -b clean.

* Studio: also flush orphan shell_calls on response.incomplete

Addresses gemini-code-assist[bot] high-priority inline review on PR
5785: the orphan-shell_call final flush added in 7fe1adbf landed only
in the response.completed branch. Truncated OpenAI Responses streams
emit response.incomplete instead (for example when the request hits
max_output_tokens), which left in-flight shell_call cards spinning
indefinitely in the UI.

Mirror the same flush block in the response.incomplete handler so the
truncated-stream path finalizes every pending tool card. The
tool_end_emitted guard keeps the path idempotent: if a shell_call
already completed via bundled output on its done event, the incomplete
flush is a no-op for it.

Two new tests in test_openai_tool_result_fallbacks.py:
- test_shell_call_flushed_on_response_incomplete_truncation pins the
  bug repro: an in-flight shell_call followed by response.incomplete
  must emit tool_end so the card finalizes.
- test_shell_call_incomplete_does_not_double_emit pins idempotency:
  a shell_call that completed via bundled output and is then followed
  by response.incomplete emits exactly one tool_end with the bundled
  result text.

17/17 tests green (8 fallback tests + 9 existing code-execution). Pre-
commit + ruff format clean.

* Studio: trim verbose comments across PR 5785 edits

Compress the in-code commentary added across this branch to one or two
lines per block; the verbose prose was easier as a PR description than
as inline noise. No behavioural changes: 17/17 tests still green, tsc -b
still clean.
This commit is contained in:
Daniel Han 2026-05-26 04:31:22 -07:00 • committed by GitHub
commit 41d24227cd
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 564 additions and 32 deletions
Download patch file Download diff file Expand all files Collapse all files

85
studio/backend/core/inference/external_provider.py
View file

@ -3873,14 +3873,16 @@ class ExternalProviderClient:
),
}
)
# Per-card text; last call gets overwritten
# with citations at response.completed.
per_call_result = (
f"Searching: {query}" if query else ""
)
yield _emit_tool_event(
{
"type": "tool_end",
"tool_call_id": item_id,
# Empty result — the last call gets
# overwritten with citations at
# response.completed.
"result": "",
"result": per_call_result,
}
)
elif item.get("type") == "shell_call":
@ -3908,7 +3910,11 @@ class ExternalProviderClient:
)
shell_calls.setdefault(
item_id,
{"commands": [], "output": None},
{
"commands": [],
"output": None,
"tool_end_emitted": False,
},
)
shell_calls[item_id]["commands"] = (
list(commands)
@ -3926,6 +3932,24 @@ class ExternalProviderClient:
},
}
)
# Fallback: output may be bundled on the
# shell_call done event itself.
embedded_output = item.get("output")
if (
isinstance(embedded_output, list)
and embedded_output
):
shell_calls[item_id]["output"] = embedded_output
shell_calls[item_id]["tool_end_emitted"] = True
yield _emit_tool_event(
{
"type": "tool_end",
"tool_call_id": item_id,
"result": _format_shell_output(
embedded_output
),
}
)
elif item.get("type") == "shell_call_output":
# `call_id` links back to the shell_call's
# `id`, which is what we used as the
@ -3936,8 +3960,15 @@ class ExternalProviderClient:
item.get("call_id") or item.get("id") or ""
)
output = item.get("output") or []
# Skip if bundled-output path already
# finalised this card.
if shell_calls.get(call_id, {}).get(
"tool_end_emitted"
):
continue
if call_id in shell_calls:
shell_calls[call_id]["output"] = output
shell_calls[call_id]["tool_end_emitted"] = True
result_text = _format_shell_output(output)
yield _emit_tool_event(
{
@ -4093,15 +4124,10 @@ class ExternalProviderClient:
}
)
container_id_emitted = True
# Apply the aggregated citation list onto the
# *last* web_search call by overwriting its
# tool_end result. The frontend's
# parseSourcesFromResult flatMaps every
# web_search tool-call result, so a single
# non-empty result is enough to surface the
# whole source-pill set at the message tail —
# no need to fan out across every card (which
# would just duplicate the same pills).
# Overwrite the last web_search call with the
# citation list; the source-pill extractor
# flatMaps across cards. Earlier cards keep
# their per-call "Searching:" text.
if web_search_calls and all_url_citations:
last_id = list(web_search_calls.keys())[-1]
blocks: list[str] = []
@ -4119,6 +4145,21 @@ class ExternalProviderClient:
"result": "\n---\n".join(blocks),
}
)
# Final flush: finalise any orphan shell_call
# so the card stops spinning.
for sc_id, sc_state in shell_calls.items():
if sc_state.get("tool_end_emitted"):
continue
yield _emit_tool_event(
{
"type": "tool_end",
"tool_call_id": sc_id,
"result": _format_shell_output(
sc_state.get("output") or []
),
}
)
sc_state["tool_end_emitted"] = True
chunk = {
"id": completion_id,
"object": "chat.completion.chunk",
@ -4197,6 +4238,22 @@ class ExternalProviderClient:
"result": "\n---\n".join(blocks),
}
)
# Mirror the response.completed flush so
# truncated streams also finalise orphan
# shell_calls.
for sc_id, sc_state in shell_calls.items():
if sc_state.get("tool_end_emitted"):
continue
yield _emit_tool_event(
{
"type": "tool_end",
"tool_call_id": sc_id,
"result": _format_shell_output(
sc_state.get("output") or []
),
}
)
sc_state["tool_end_emitted"] = True
chunk = {
"id": completion_id,
"object": "chat.completion.chunk",

372
studio/backend/tests/test_openai_tool_result_fallbacks.py Normal file
View file

@ -0,0 +1,372 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
"""Regression tests for OpenAI Responses tool-result rendering.
Covers two bug classes: empty web_search cards (per-card result seeded
with "Searching: <query>") and orphan shell_call cards (bundled-output
fallback + final flush at response.completed / response.incomplete).
"""
import asyncio
import json
import httpx
from core.inference import external_provider as ep_mod
from core.inference.external_provider import ExternalProviderClient
def _drive(coro):
return asyncio.new_event_loop().run_until_complete(coro)
async def _collect(agen):
out = []
async for line in agen:
out.append(line)
return out
def _mock_http_client(monkeypatch, handler):
transport = httpx.MockTransport(handler)
monkeypatch.setattr(ep_mod, "_http_client", httpx.AsyncClient(transport = transport))
def _make_client(base_url: str = "https://api.openai.com/v1") -> ExternalProviderClient:
return ExternalProviderClient(
provider_type = "openai",
base_url = base_url,
api_key = "sk-test",
)
def _openai_sse(events: list[dict]) -> bytes:
chunks: list[str] = []
for event in events:
chunks.append(f"event: {event['type']}")
chunks.append(f"data: {json.dumps(event)}")
chunks.append("")
return ("\n".join(chunks) + "\n").encode("utf-8")
def _tool_events(lines: list[str]) -> list[dict]:
out: list[dict] = []
for line in lines:
if not line.startswith("data:"):
continue
raw = line[len("data:") :].strip()
if not raw or raw == "[DONE]":
continue
try:
parsed = json.loads(raw)
except json.JSONDecodeError:
continue
if isinstance(parsed, dict) and "_toolEvent" in parsed:
out.append(parsed["_toolEvent"])
return out
def _drive_stream(sse_events, enabled_tools, monkeypatch):
def handler(request):
return httpx.Response(
200,
content = _openai_sse(sse_events),
headers = {"content-type": "text/event-stream"},
)
_mock_http_client(monkeypatch, handler)
async def run():
client = _make_client()
return await _collect(
client._stream_openai_responses(
messages = [{"role": "user", "content": "x"}],
model = "gpt-5.5",
temperature = 0.7,
top_p = 0.95,
max_tokens = 4096,
enable_thinking = None,
reasoning_effort = None,
enabled_tools = enabled_tools,
)
)
return _drive(run())
# ── web_search per-card result ─────────────────────────────────────────
def test_web_search_each_call_carries_its_own_query_as_result(monkeypatch):
"""Each card carries its own `Searching: <query>` text; no empties."""
sse_events = [
{
"type": "response.output_item.done",
"item": {
"type": "web_search_call",
"id": "ws_1",
"action": {"query": "popular animals 2026"},
},
},
{
"type": "response.output_item.done",
"item": {
"type": "web_search_call",
"id": "ws_2",
"action": {"query": "most loved animals poll"},
},
},
{
"type": "response.output_item.done",
"item": {
"type": "web_search_call",
"id": "ws_3",
"action": {"query": "tiger ranking"},
},
},
{"type": "response.completed", "response": {}},
]
lines = _drive_stream(sse_events, ["web_search"], monkeypatch)
events = _tool_events(lines)
ends = [e for e in events if e["type"] == "tool_end"]
by_id = {e["tool_call_id"]: e for e in ends}
assert by_id["ws_1"]["result"] == "Searching: popular animals 2026"
assert by_id["ws_2"]["result"] == "Searching: most loved animals poll"
assert by_id["ws_3"]["result"] == "Searching: tiger ranking"
def test_web_search_last_call_overwritten_with_citations(monkeypatch):
"""Last call still gets the aggregated citation list; earlier calls
keep their per-call `Searching:` text."""
sse_events = [
{
"type": "response.output_item.done",
"item": {
"type": "web_search_call",
"id": "ws_1",
"action": {"query": "first query"},
},
},
{
"type": "response.output_item.done",
"item": {
"type": "web_search_call",
"id": "ws_2",
"action": {"query": "second query"},
},
},
{
"type": "response.output_text.annotation.added",
"annotation": {
"type": "url_citation",
"url": "https://example.com/a",
"title": "Example A",
},
},
{"type": "response.completed", "response": {}},
]
lines = _drive_stream(sse_events, ["web_search"], monkeypatch)
events = _tool_events(lines)
ends = [e for e in events if e["type"] == "tool_end"]
by_id: dict = {}
# Keep the LAST tool_end per id (the citation overwrite for ws_2).
for e in ends:
by_id[e["tool_call_id"]] = e
# First call keeps its own query.
assert by_id["ws_1"]["result"] == "Searching: first query"
# Last call gets overwritten with the citation block.
assert "Title: Example A" in by_id["ws_2"]["result"]
assert "URL: https://example.com/a" in by_id["ws_2"]["result"]
def test_web_search_empty_query_falls_back_to_empty_result(monkeypatch):
"""No query -> empty result (no `Searching:` placeholder)."""
sse_events = [
{
"type": "response.output_item.done",
"item": {
"type": "web_search_call",
"id": "ws_only",
"action": {},
},
},
{"type": "response.completed", "response": {}},
]
lines = _drive_stream(sse_events, ["web_search"], monkeypatch)
events = _tool_events(lines)
ends = [e for e in events if e["type"] == "tool_end"]
assert len(ends) == 1
assert ends[0]["result"] == ""
# ── shell_call output fallbacks ────────────────────────────────────────
def test_shell_call_emits_tool_end_when_output_bundled_on_done(monkeypatch):
"""Output bundled on the shell_call done event emits tool_end."""
sse_events = [
{
"type": "response.output_item.done",
"item": {
"type": "shell_call",
"id": "scall_bundled",
"action": {"commands": ["echo hi"]},
"output": [
{
"stdout": "hi\n",
"stderr": "",
"outcome": {"type": "exit", "exit_code": 0},
}
],
},
},
{"type": "response.completed", "response": {}},
]
lines = _drive_stream(sse_events, ["code_execution"], monkeypatch)
events = _tool_events(lines)
starts = [e for e in events if e["type"] == "tool_start"]
ends = [e for e in events if e["type"] == "tool_end"]
assert len(starts) == 1
assert starts[0]["tool_call_id"] == "scall_bundled"
assert len(ends) == 1
assert ends[0]["tool_call_id"] == "scall_bundled"
assert "hi" in ends[0]["result"]
def test_shell_call_bundled_then_separate_output_does_not_double_emit(monkeypatch):
"""Separate shell_call_output after bundled-output is a no-op."""
sse_events = [
{
"type": "response.output_item.done",
"item": {
"type": "shell_call",
"id": "scall_both",
"action": {"commands": ["echo bundle"]},
"output": [
{
"stdout": "bundle\n",
"stderr": "",
"outcome": {"type": "exit", "exit_code": 0},
}
],
},
},
{
"type": "response.output_item.done",
"item": {
"type": "shell_call_output",
"id": "scout_both",
"call_id": "scall_both",
"output": [
{
"stdout": "should not double-emit\n",
"stderr": "",
"outcome": {"type": "exit", "exit_code": 0},
}
],
},
},
{"type": "response.completed", "response": {}},
]
lines = _drive_stream(sse_events, ["code_execution"], monkeypatch)
events = _tool_events(lines)
ends = [e for e in events if e["type"] == "tool_end"]
assert len(ends) == 1
assert ends[0]["tool_call_id"] == "scall_both"
assert "bundle" in ends[0]["result"]
assert "should not double-emit" not in ends[0]["result"]
def test_shell_call_final_flush_on_completed_when_no_output_event(monkeypatch):
"""Orphan shell_call finalises via the response.completed flush."""
sse_events = [
{
"type": "response.output_item.added",
"item": {
"type": "shell_call",
"id": "scall_orphan",
"action": {"commands": ["true"]},
},
},
{
"type": "response.output_item.done",
"item": {
"type": "shell_call",
"id": "scall_orphan",
"action": {"commands": ["true"]},
"status": "completed",
},
},
{"type": "response.completed", "response": {}},
]
lines = _drive_stream(sse_events, ["code_execution"], monkeypatch)
events = _tool_events(lines)
ends = [e for e in events if e["type"] == "tool_end"]
assert any(e["tool_call_id"] == "scall_orphan" for e in ends)
def test_shell_call_flushed_on_response_incomplete_truncation(monkeypatch):
"""Truncated streams (response.incomplete) also flush orphan calls."""
sse_events = [
{
"type": "response.output_item.added",
"item": {
"type": "shell_call",
"id": "scall_truncated",
"action": {"commands": ["long_running"]},
},
},
{
"type": "response.output_item.done",
"item": {
"type": "shell_call",
"id": "scall_truncated",
"action": {"commands": ["long_running"]},
"status": "in_progress",
},
},
{
"type": "response.incomplete",
"response": {
"incomplete_details": {"reason": "max_output_tokens"},
},
},
]
lines = _drive_stream(sse_events, ["code_execution"], monkeypatch)
events = _tool_events(lines)
ends = [e for e in events if e["type"] == "tool_end"]
assert any(e["tool_call_id"] == "scall_truncated" for e in ends)
def test_shell_call_incomplete_does_not_double_emit(monkeypatch):
"""response.incomplete is idempotent against already-finalised calls."""
sse_events = [
{
"type": "response.output_item.done",
"item": {
"type": "shell_call",
"id": "scall_done",
"action": {"commands": ["echo done"]},
"output": [
{
"stdout": "done\n",
"stderr": "",
"outcome": {"type": "exit", "exit_code": 0},
}
],
},
},
{
"type": "response.incomplete",
"response": {
"incomplete_details": {"reason": "max_output_tokens"},
},
},
]
lines = _drive_stream(sse_events, ["code_execution"], monkeypatch)
events = _tool_events(lines)
ends = [e for e in events if e["type"] == "tool_end"]
assert len(ends) == 1
assert ends[0]["tool_call_id"] == "scall_done"
assert "done" in ends[0]["result"]

14
studio/frontend/src/features/chat/api/chat-adapter.ts
View file

@ -21,6 +21,7 @@ import { pickFriendlyContainerName } from "../lib/friendly-names";
import {
EXTERNAL_MAX_OUTPUT_TOKENS,
clampReasoningEffortToLevels,
getExternalMaxOutputTokens,
getExternalMinOutputTokens,
getExternalReasoningCapabilities,
getProviderCapabilities,
@ -1703,18 +1704,17 @@ export function createOpenAIStreamAdapter(): ChatModelAdapter {
...(externalCapabilities?.topP !== false
? { top_p: params.topP }
: {}),
// Clamp to the cross-provider output cap so a maxTokens value
// carried over from a local-model session does not blow past
// provider limits (e.g. Claude Opus 400s on >128k). Also
// floor to the provider's documented minimum — Kimi's
// thinking models need >=16k or the response truncates
// before the answer fits alongside reasoning_content.
// Floor at the provider's documented min (Kimi thinking
// needs >=16k); clamp at the per-model max.
max_tokens: Math.min(
Math.max(
params.maxTokens,
getExternalMinOutputTokens(externalProvider?.providerType),
),
EXTERNAL_MAX_OUTPUT_TOKENS,
getExternalMaxOutputTokens(
externalProvider?.providerType,
externalSelection?.modelId,
),
),
// Only forward sampling knobs the provider actually accepts; the
// backend's external-provider proxy is param-permissive and would

6
studio/frontend/src/features/chat/chat-settings-sheet.tsx
View file

@ -85,6 +85,7 @@ import {
import {
EXTERNAL_MAX_OUTPUT_TOKENS,
type ProviderCapabilities,
getExternalMaxOutputTokens,
getExternalMinOutputTokens,
providerSupportsBuiltinCodeExecution,
providerSupportsFastMode,
@ -1309,7 +1310,10 @@ export function ChatSettingsPanel({
}
max={
isExternalModel
? EXTERNAL_MAX_OUTPUT_TOKENS
? getExternalMaxOutputTokens(
externalProviderType,
externalSelection?.modelId,
)
: isGguf && ggufContextLength
? ggufContextLength
: 32768

95
studio/frontend/src/features/chat/provider-capabilities.ts
View file

@ -71,18 +71,95 @@ export function clampReasoningEffortToLevels(
}
/**
* Output-token cap for any external provider request. Picked to stay below the
* tightest declared limit across the providers we ship (Anthropic Claude Opus
* tops out at 128k, GPT-5.x ~128k, Gemini 2.5 ~65k, DeepSeek 8k) while staying
* well above what a typical chat reply needs. The local-model path is not
* subject to this — local backends honour whatever the loaded context allows.
*
* If a user's stored maxTokens (e.g. carried over from a prior local-model
* session with a 128k+ context) exceeds this, chat-adapter clamps the
* outbound request so the provider does not 400 on it.
* Fallback cap for unknown providers / models. Prefer
* `getExternalMaxOutputTokens(providerType, modelId)` for the real cap.
*/
export const EXTERNAL_MAX_OUTPUT_TOKENS = 32768;
/**
* Per-model max-output caps from each provider's docs:
* OpenAI: developers.openai.com/api/docs/models/gpt-5.5
* Anthropic: platform.claude.com/docs/en/about-claude/models
* Gemini: ai.google.dev/gemini-api/docs/models/gemini-3.1-pro-preview
* DeepSeek: api-docs.deepseek.com/quick_start/pricing (V4 family)
* Local-model path is unaffected.
*/
const EXTERNAL_MAX_OUTPUT_TOKENS_BY_MODEL: Array<{
providerType: string;
prefixes: readonly string[];
cap: number;
}> = [
// OpenAI
{ providerType: "openai", prefixes: ["gpt-5.5-pro", "gpt-5.5"], cap: 128000 },
{ providerType: "openai", prefixes: ["gpt-5.4-pro", "gpt-5.4"], cap: 65536 },
{ providerType: "openai", prefixes: ["gpt-5.3"], cap: 16384 },
// Anthropic
{
providerType: "anthropic",
prefixes: ["claude-opus-4-7"],
cap: 128000,
},
{
providerType: "anthropic",
prefixes: [
"claude-opus-4-6",
"claude-sonnet-4-6",
"claude-opus-4-5",
"claude-sonnet-4-5",
"claude-haiku-4-5",
],
cap: 64000,
},
// Gemini
{
providerType: "gemini",
prefixes: ["gemini-3", "gemini-pro", "gemini-flash"],
cap: 65536,
},
// DeepSeek (V4: deepseek-chat / deepseek-reasoner alias V4-flash).
{ providerType: "deepseek", prefixes: ["deepseek"], cap: 384000 },
];
/**
* Documented per-model output cap; unknown ids fall back to
* `EXTERNAL_MAX_OUTPUT_TOKENS` (32k). OpenRouter ids are
* `provider/model`; the prefix is stripped before matching.
*/
export function getExternalMaxOutputTokens(
providerType: string | null | undefined,
modelId: string | null | undefined,
): number {
if (!providerType || !modelId) return EXTERNAL_MAX_OUTPUT_TOKENS;
const normalized = modelId.trim().toLowerCase();
if (!normalized) return EXTERNAL_MAX_OUTPUT_TOKENS;
const stripped =
providerType === "openrouter" && normalized.includes("/")
? normalized.split("/").slice(-1)[0]
: normalized;
const effectiveProvider =
providerType === "openrouter"
? _inferProviderFromOpenrouterId(normalized) ?? providerType
: providerType;
for (const entry of EXTERNAL_MAX_OUTPUT_TOKENS_BY_MODEL) {
if (entry.providerType !== effectiveProvider) continue;
if (entry.prefixes.some((prefix) => stripped.startsWith(prefix))) {
return entry.cap;
}
}
return EXTERNAL_MAX_OUTPUT_TOKENS;
}
function _inferProviderFromOpenrouterId(
normalizedId: string,
): string | null {
// Map OpenRouter `provider/model` prefix to our internal providerType.
if (normalizedId.startsWith("openai/")) return "openai";
if (normalizedId.startsWith("anthropic/")) return "anthropic";
if (normalizedId.startsWith("google/")) return "gemini";
if (normalizedId.startsWith("deepseek/")) return "deepseek";
return null;
}
/**
* Whether the external provider offers a built-in web-search tool that the
* model invokes server-side. When `true`, the chat composer's Search button

24
studio/frontend/src/features/chat/stores/chat-runtime-store.ts
View file

@ -14,7 +14,9 @@ import {
DEFAULT_INFERENCE_PARAMS,
type InferenceParams,
} from "../types/runtime";
import { isExternalModelId } from "../external-providers";
import { isExternalModelId, parseExternalModelId } from "../external-providers";
import { getExternalMaxOutputTokens } from "../provider-capabilities";
import { useExternalProvidersStore } from "./external-providers-store";
import {
loadChatSettingsWithLegacyImport,
savePersistedChatSettingsPatch,
@ -747,10 +749,30 @@ export const useChatRuntimeStore = create<ChatRuntimeStore>((set, get) => ({
// external-provider render gate would otherwise show old counters
// until the next completion overwrites them.
const checkpointChanged = state.params.checkpoint !== modelId;
// Clamp maxTokens to the new model's cap on switch into an
// external model so a value carried over from a prior local
// session does not render above the slider's max.
let nextMaxTokens = state.params.maxTokens;
if (checkpointChanged && isExternalModelId(modelId)) {
const parsed = parseExternalModelId(modelId);
const provider = parsed
? useExternalProvidersStore
.getState()
.providers.find((p) => p.id === parsed.providerId)
: null;
const cap = getExternalMaxOutputTokens(
provider?.providerType,
parsed?.modelId,
);
if (nextMaxTokens > cap) {
nextMaxTokens = cap;
}
}
return {
params: {
...state.params,
checkpoint: modelId,
maxTokens: nextMaxTokens,
},
activeGgufVariant: ggufVariant ?? null,
...(checkpointChanged ? { contextUsage: null } : {}),