From 9a81a5e8e7d049a4894dbe409a95831b8d095f06 Mon Sep 17 00:00:00 2001 From: Roland Tannous <115670425+rolandtannous@users.noreply.github.com> Date: Fri, 15 May 2026 15:49:08 +0400 Subject: [PATCH 1/4] Update version-compat-ci.yml (#5445) --- .github/workflows/version-compat-ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/version-compat-ci.yml b/.github/workflows/version-compat-ci.yml index 1ebea81066..2fbdd15747 100644 --- a/.github/workflows/version-compat-ci.yml +++ b/.github/workflows/version-compat-ci.yml @@ -214,7 +214,7 @@ jobs: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: persist-credentials: false - with: { path: unsloth } + path: unsloth - name: Clone unsloth-zoo @ main run: | # github.com occasionally 500s on the git fetch; retry so a From e81b942d2698e5d8a977f7412d338e1b1a72de67 Mon Sep 17 00:00:00 2001 From: Roland Tannous <115670425+rolandtannous@users.noreply.github.com> Date: Fri, 15 May 2026 16:05:14 +0400 Subject: [PATCH 2/4] ci: merge duplicate `with:` keys in workflow checkout steps (#5447) Two `with:` mapping keys on the same step caused GitHub's workflow loader to reject the file (silently dropping persist-credentials: false under YAML "last key wins"). Merge into a single `with:` block in notebooks-ci.yml (3 sites) and version-compat-ci.yml (1 site). --- .github/workflows/notebooks-ci.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/notebooks-ci.yml b/.github/workflows/notebooks-ci.yml index 0881c5ef3a..673b2f3cc5 100644 --- a/.github/workflows/notebooks-ci.yml +++ b/.github/workflows/notebooks-ci.yml @@ -200,7 +200,7 @@ jobs: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: persist-credentials: false - with: { path: unsloth } + path: unsloth - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: repository: unslothai/notebooks @@ -246,7 +246,7 @@ jobs: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: persist-credentials: false - with: { path: unsloth } + path: unsloth - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: repository: unslothai/notebooks @@ -352,7 +352,7 @@ jobs: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: persist-credentials: false - with: { path: unsloth } + path: unsloth - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: repository: unslothai/notebooks From 3f8c67263638281d91ac304f5ac44040992ae68a Mon Sep 17 00:00:00 2001 From: Roland Tannous <115670425+rolandtannous@users.noreply.github.com> Date: Fri, 15 May 2026 16:34:14 +0400 Subject: [PATCH 3/4] studio/chat: built-in web search for OpenAI, Anthropic, OpenRouter, Kimi (#5443) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * studio: wire the chat Search button to OpenAI's built-in web_search tool When the active model is an OpenAI external provider and the user clicks the existing Search pill in the composer, the chat-completion request now carries the unified enable_tools shorthand: enable_tools: true enabled_tools: ["web_search"] The backend's stream_chat_completion threads enabled_tools through to _stream_openai_responses, which translates it into the Responses API tool schema: body["tools"] = [{"type": "web_search"}] per the OpenAI Responses tool spec (https://developers.openai.com/api/docs/guides/tools). OpenAI then runs the search server-side before the model replies; the search- informed answer streams back through the existing response.output_text.delta path. web_search_call lifecycle events are silently ignored for now — sources / status indicators are follow-up scope. Frontend: - provider-capabilities.ts: new providerSupportsBuiltinWebSearch() helper. Returns true only for `openai` today; Anthropic (web_search_20250305), Gemini grounded-search, and OpenRouter variants can be added later with matching backend translation. - chat-page.tsx: both model-switch paths (the onChange handler and the inferenceParams.checkpoint useEffect) set supportsTools to match the new helper, and force toolsEnabled=false on every external switch so the Search toggle is opt-in by default. - chat-adapter.ts: external branch adds enable_tools + enabled_tools=["web_search"] to the request body when the toggle is on AND the active provider supports built-in web-search. Local-model branch is unchanged — it continues to route the same shorthand through our local tool runtime. Backend: - routes/inference.py: forwards payload.enabled_tools to stream_chat_completion at the proxy site (line 1599). - external_provider.py: stream_chat_completion gains an enabled_tools parameter; _stream_openai_responses appends {"type": "web_search"} to body["tools"] when the list contains "web_search". Other tools (file_search, code_interpreter, image_generation, computer_use_preview) are easy follow-ups in the same block. Reuses the existing pydantic ChatCompletionRequest.enabled_tools field, so no schema migrations. * studio/backend: surface OpenAI server-side web_search in the chat UI When the user has the chat Search button toggled on and OpenAI's /v1/responses invokes the built-in web_search tool, _stream_openai_responses now translates the tool's lifecycle events and citation annotations into the same _toolEvent shape that local-tool calls use. The result: the chat UI shows a web_search tool-call card mid-stream, then lists the cited sources at the end of the message — identical to how local web_search renders. SSE event translation: - response.output_item.added with item.type=web_search_call -> emit _toolEvent tool_start. Carries item.action.query as args when OpenAI ships it on the added event. - response.output_item.done with item.type=web_search_call -> backfill the query if it only arrives on the done variant. The existing reasoning branch on the same event is preserved as an if/elif under a shared isinstance guard. - response.output_text.annotation.added with type=url_citation -> collect into the most-recent web_search_call.citations list. - response.output_text.delta with inline annotations[] (older API variant) -> same collection path, so both wire shapes work. - response.completed -> emit _toolEvent tool_end per call with citations formatted as Title: \nURL: <url>\nSnippet: <snippet> blocks joined by `\n---\n`. The frontend's parseSourcesFromResult already lifts this format into source content parts at end-of-stream. - response.incomplete -> close out web_search cards with whatever citations had landed, so a truncated response does not leave a perpetually "running" tool card in the UI. Both reasoning and web_search work simultaneously on the same turn — the body sends `reasoning: {effort, summary}` and `tools: [{type: "web_search"}]` independently, and the SSE handler tracks them through separate channels. Diagnostic: finally-block logger now reports per stream web_search_requested - whether the client asked for it web_search_invocations - how many calls OpenAI actually made citations - total URLs cited queries - the search queries the model issued reasoning_emitted - whether <think> content was streamed so reports of "I clicked Search and nothing happened" can be triaged from the backend log without browser devtools. * studio/backend: fix empty query + per-card '(no sources cited)' on OpenAI web_search Two display bugs on the OpenAI Responses web_search → chat-UI bridge: 1. Tool cards showed "Searching for ''" — query missing. OpenAI's response.output_item.added for web_search_call does not reliably populate action.query across API versions; the canonical place is output_item.done. The previous code emitted tool_start at added with empty args and tried to backfill at done, but the frontend's _toolEvent: tool_start is a one-shot push (no update mechanism), so the args stayed empty. Fix: defer both tool_start *and* a placeholder tool_end emission to output_item.done, where action.query is guaranteed populated. added now just initialises tracking. Frontend then renders one card per call with the right "Searching for: <query>" label. 2. Every card showed "(no sources cited)". The previous code tried to attribute url_citation annotations to individual web_search_call invocations, but OpenAI's annotations carry no link back to a specific search call — they're just URLs the model cited from the aggregated search pool. With N invocations and M annotations, the previous logic bucketed all M into the last call and stamped "(no sources cited)" on the rest. Fix: collect citations into a single shared all_url_citations list, dedup by URL. At response.completed (and response.incomplete) overwrite the *last* web_search_call's tool_end result with the aggregated Title:/URL:/Snippet: blocks. The frontend's parseSourcesFromResult already flatMaps every web_search result, so one non-empty result is enough to surface the full source-pill set at the message tail. Other tool cards get an empty result string (no '(no sources)' text). Diagnostic log unchanged in shape; total_citations now reads len(all_url_citations) directly. * studio/chat: split Code and Search pill gates so external models cannot enable Code The previous wire-up set supportsTools=true for OpenAI external models to light up the Search pill, but supportsTools also gates the Code pill, so Code became clickable for OpenAI even though external providers have no local code execution. Separate the two gates so each pill reflects what's actually available: - chat-runtime-store: new `supportsBuiltinWebSearch: boolean` flag. Distinct from supportsTools — that one still means "runtime has a local tool sandbox" (Code, python, our DuckDuckGo web_search). This one means "the active external provider exposes a server-side web_search tool we can opt into" (OpenAI's /v1/responses today). - chat-page model-switch (both code paths): for external models, supportsTools is now forced to false (no local Code path) and supportsBuiltinWebSearch follows providerSupportsBuiltinWebSearch. Local-model paths are unaffected — they only set supportsTools. - shared-composer: Search pill gates on `searchDisabled = !modelLoaded || !(supportsTools || supportsBuiltinWebSearch)`. Code pill gates on `codeDisabled = !modelLoaded || !supportsTools` — strictly the local runtime, so external models keep Code greyed out. A `toolsDisabled = codeDisabled` alias is left in place for any later-touched call site that may still reference the old name. No backend changes — chat-adapter already calls providerSupportsBuiltinWebSearch directly, independent of the store flags, so the request shape and the backend translation are unchanged. * studio/chat: default external reasoning effort to medium, not the carry-over When switching to an external model with reasoning support, the effort dropdown was inheriting whatever value the user had set on a prior model — frequently "xhigh" left over from a previous Opus/gpt-5 session. That meant every fresh OpenAI/Anthropic selection started at Extra High, burning tokens unintentionally. Both model-switch sites in chat-page (the useEffect on inferenceParams.checkpoint and the onChange callback) now pick "medium" whenever the new model's level list contains it, instead of the clamped carry-over. The clamp still fires as a fallback for the narrow case where a model doesn't expose medium (e.g. gpt-5.3-chat- latest which only has medium anyway — no change there). Users can still pick another level explicitly via the Think dropdown. * studio/chat: also light the Search pill in the welcome-screen composer There are two composers in the chat feature. shared-composer.tsx renders inside an active thread, and assistant-ui/thread.tsx has its own WebSearchToggle / CodeToolsToggle that ship the welcome-screen "Send a message…" composer (visible before the first user message). The previous fix split supportsTools and supportsBuiltinWebSearch in shared-composer but never touched the welcome-screen toggles in thread.tsx — they both still gated on supportsTools alone, so the Search pill stayed greyed on the welcome screen even for OpenAI external models that legitimately support web_search server-side. Mirror the shared-composer rule in WebSearchToggle: disabled = !modelLoaded || !(supportsTools || supportsBuiltinWebSearch) CodeToolsToggle is left as-is — its current `disabled = !(modelLoaded && supportsTools)` is correct: external models have no local code-execution sandbox, so Code stays greyed when supportsTools=false (which is what chat-page now writes for external selections). * studio/backend: wire Anthropic server-side web_search end-to-end Mirrors the OpenAI web_search integration for Anthropic's web_search_20250305 tool. When the user toggles Search on with an Anthropic model selected, the request now carries the documented tool entry: tools: [{type: "web_search_20250305", name: "web_search", max_uses: 5}] on /v1/messages, and the SSE translation surfaces tool cards + source pills in the chat UI exactly the same way as OpenAI. stream_chat_completion now forwards enabled_tools into the Anthropic branch (was only doing this for the OpenAI Responses branch). _stream_anthropic gains an enabled_tools parameter and the web_search request-body block plus three additional event handlers: - content_block_start with type=server_tool_use, name=web_search: start tracking a new call. id becomes the tool_call_id. - content_block_delta with type=input_json_delta inside a server_tool_use block: buffer the partial_json so we can read out the search query when the block closes. - content_block_start with type=web_search_tool_result: capture the per-call result list (urls + titles) that Anthropic ships inline. - content_block_stop: closes whichever block we're inside — * server_tool_use -> emit _toolEvent: tool_start with the parsed query as args. * web_search_tool_result -> emit _toolEvent: tool_end with Title:/URL: blocks the frontend's parseSourcesFromResult lifts into source pills. * thinking block -> existing </think> close. Unlike OpenAI we get per-call results directly, so no aggregated- last-call fallback is needed — each tool card carries its own citations. Diagnostic log on stream completion now reports web_search_requested / invocations / total_results / queries, matching the OpenAI shape. Frontend providerSupportsBuiltinWebSearch returns true for 'anthropic' as well, so the Search pill lights up on Claude models the same way it does on OpenAI. The existing chat-adapter external branch already sends enabled_tools=['web_search'] based on this helper — no adapter changes needed. * studio: wire OpenRouter built-in web search via :online model suffix OpenRouter exposes a universal "add web search to any model" shortcut: append `:online` to the model id and the gateway runs the search server-side, streaming citations back as annotations on text deltas. Documented at https://openrouter.ai/docs/features/web-search Hook the existing Search toggle into that path: Backend (external_provider.py, default OAI-compat branch): - When provider_type == 'openrouter' and enabled_tools contains 'web_search', rewrite body['model']: openai/gpt-4o -> openai/gpt-4o:online anthropic/claude-sonnet-4-5:free -> anthropic/claude-sonnet-4-5:online Any existing `:variant` (`:free`, `:nitro`, etc.) is replaced — OpenRouter variants are mutually exclusive. - `openrouter/free` is skipped: it's a meta-router and `:online` is not a valid suffix on it (the gateway 400s). - A one-line INFO log fires whenever the rewrite happens so the diagnostic backend log shows exactly which model id the request was promoted to. Frontend (provider-capabilities.ts): - providerSupportsBuiltinWebSearch now returns true for 'openrouter' alongside 'openai' and 'anthropic'. The Search pill lights up and the existing chat-adapter external branch already forwards enabled_tools=['web_search'] based on this helper — no adapter changes needed. No new SSE event handling: OpenRouter does not emit a separate web_search_call event the way OpenAI/Anthropic do. Citations come back as text annotations via the existing reasoning_details path the adapter already parses, so source data flows through without extra translation. A per-call tool-card UX ("Searching for: …") would require synthesizing one client-side; deferred to a follow-up if the bare-citation flow feels too minimal. * studio: wire Mistral built-in web search connector Same shape as OpenAI's web_search tool, lives on /v1/chat/completions instead of /v1/responses. When the chat Search pill is toggled on with a Mistral model selected, the backend now appends {"type": "web_search"} to body["tools"] before the request goes out. Idempotent — won't double-append if a future call site adds it first. Models in the registry allowlist that don't support the connector (codestral, devstral, ministral, mistral-tiny) will surface a 400 from upstream; the existing default-path error log captures it. Mistral's docs: https://docs.mistral.ai/capabilities/agents/connectors/websearch Frontend providerSupportsBuiltinWebSearch returns true for 'mistral' now, alongside openai / anthropic / openrouter. The Search pill lights up for Mistral models and the existing adapter branch already sends enabled_tools=['web_search'] off this helper — no adapter changes. No SSE translation yet — Mistral streams citations inline as text annotations or `references` in the final assistant content, not as a separate web_search_call event. Citations flow through to the message body as text; a per-call tool-card UX with "Searching for: …" indicators is a follow-up if needed. * studio/backend: fix OpenRouter web_search to use plugins shape + synthesize tool card Two changes against the actual OpenRouter docs at https://openrouter.ai/docs/guides/features/plugins/web-search: Request shape: The previous commit appended :online to the model id, which works on concrete model ids but rejects on meta-routers like openrouter/free — and that's exactly the model the user was testing with, so neither the request rewrite nor the diagnostic log fired. Switch to the universal plugins shape: body["plugins"] = [{"id": "web"}] Per the docs this is "exactly equivalent" to :online but works on every model id including openrouter/free and openrouter/auto. No model suffix manipulation, idempotent if added twice. Tool-card synthesis: OpenRouter doesn't emit a structured web_search_call event the way OpenAI/Anthropic do — citations come back only as `annotations` of type=url_citation on delta/message objects. To match the chat-UI tool-card UX the user expects ("Searching for: …" indicator, source pills at message tail), synthesize the events client-side in the default OAI-compat stream loop: - On stream open (after the 200 status check): yield a synthetic _toolEvent: tool_start with tool_name=web_search, fixed id "openrouter_web_search". The chat-UI then renders the running tool card before any text streams. - During the SSE loop: scan every chunk's choices[].delta and choices[].message for `annotations: [{type: "url_citation", url_citation: {url, title, content}}]` entries. Dedup by URL into a citations list. Handles both the nested-url_citation shape OpenRouter documents and the flat-on-annotation shape some upstreams ship. - On [DONE] (or stream-close without [DONE]): emit synthetic tool_end carrying the citations as Title: …\nURL: …\nSnippet: …\n---\n… blocks the existing parseSourcesFromResult lifts into source pills at message tail. Diagnostic log on completion now also reports web_search_requested + citation count alongside the existing chosen-model / event-count telemetry. * studio: drop Mistral built-in web_search — connector lives on Agents API only Mistral's web_search is exclusively on /v1/agents + /v1/conversations; sending it on /v1/chat/completions returns "WebSearchTool connector is not supported". Wiring it would require a dedicated Agents streaming path. Remove from the frontend capability map and revert the chat-completions tool injection. * studio: wire Kimi $web_search builtin via two-call round-trip Kimi's $web_search lives on /v1/chat/completions but requires a client round-trip per https://platform.kimi.ai/docs/guide/use-web-search: the first call returns tool_calls with function.arguments populated; the caller echoes those arguments back as a role=tool message; the second call streams the final answer with search results incorporated. The docs also mandate thinking=disabled while the builtin is active. Backend: new _stream_kimi_web_search helper dispatched from stream_chat_completion when provider_type=='kimi' and 'web_search' in enabled_tools. Buffers tool_calls across deltas, falls back to a plain stream if the model declines to search, and synthesizes tool_start (with parsed query) / tool_end (with any url_citation annotations) so the chat UI's web-search card behaves the same as other providers. Frontend: kimi added to providerSupportsBuiltinWebSearch so the Search pill lights up in the composer. * studio/chat: mutual exclusion of Think + Search on Kimi composer Kimi's $web_search builtin requires thinking=disabled per https://platform.kimi.ai/docs/guide/use-web-search, so the two states cannot coexist. Make the pills mutually exclusive in both composers (shared and welcome-screen): clicking Search turns Think off; clicking Think back on turns Search off. Default Think to on when a Kimi model is selected — k2.6/k2.5 ship with thinking enabled out of the box. * studio/chat: fix wrong provider var name in onChange branch selectedProvider, not provider — TS2304 in tsc -b. * studio/backend: add diagnostics to Kimi $web_search round-trip Log the actual function.arguments from the first call (so we can see the model's search query) and the second call's usage.prompt_tokens + any annotation type names that came through. prompt_tokens spiking above the input message length is direct proof the server injected search results into context. annotation_types lets us learn the shape Kimi uses for citations if/when they emit any. * studio: per-provider defaults — Anthropic xhigh + Search on, OpenAI high + Search on, Opus 4.7 gains max Anthropic: Think effort defaults to the highest level the model supports (xhigh on 4.6/4.7, high on 4.5) and Search starts on, since the web_search_20250305 tool returns structured citations end-to-end. OpenAI: Think effort defaults to 'high' (the gpt-5.x reasoning sweet spot for /v1/responses + web_search) and Search starts on. Opus 4.7: 'max' added as an effort level above 'xhigh' in both backend (_ANTHROPIC_THINKING_SPECS) and frontend (ANTHROPIC_REASONING_MODELS). Kimi diagnostics: emit tool_end immediately after tool_start so the web-search card transitions to 'complete' before the second-call answer streams, log first-call args + second-call usage/prompt_tokens + any annotation type names, request stream_options.include_usage so the second call exposes usage in SSE. * studio/backend: harden Kimi fallback path with HTTPError handler + manual aiter_lines loop Addresses PR review feedback (#5443): the no-search fallback streaming path was using `async for response.aiter_lines()` and had no `httpx.HTTPError` guard around the POST. Switch to the manual __anext__ loop pattern used elsewhere in this module (avoids the Python 3.13 + httpcore 1.0.x GeneratorExit propagation issue) and wrap the whole request in a try/except so network failures surface as a proper SSE error frame instead of a raw traceback. --- .../core/inference/external_provider.py | 1007 ++++++++++++++++- studio/backend/routes/inference.py | 1 + .../src/components/assistant-ui/thread.tsx | 45 +- .../src/features/chat/api/chat-adapter.ts | 15 + .../frontend/src/features/chat/chat-page.tsx | 113 +- .../features/chat/provider-capabilities.ts | 40 +- .../src/features/chat/shared-composer.tsx | 56 +- .../chat/stores/chat-runtime-store.ts | 12 + 8 files changed, 1258 insertions(+), 31 deletions(-) diff --git a/studio/backend/core/inference/external_provider.py b/studio/backend/core/inference/external_provider.py index f5b67eef70..6937518b5d 100644 --- a/studio/backend/core/inference/external_provider.py +++ b/studio/backend/core/inference/external_provider.py @@ -41,7 +41,7 @@ _ANTHROPIC_THINKING_SPECS = ( _AnthropicThinkingSpec( prefixes = ("claude-opus-4-7",), kind = "adaptive", - efforts = ("none", "low", "medium", "high", "xhigh"), + efforts = ("none", "low", "medium", "high", "xhigh", "max"), ), _AnthropicThinkingSpec( prefixes = ("claude-opus-4-6", "claude-sonnet-4-6"), @@ -141,6 +141,34 @@ def _apply_mistral_reasoning_controls( _http_client = httpx.AsyncClient() +def _build_kimi_tool_end( + synthetic_chunk_fn: Any, + tool_call_id: str, + citations: list[dict[str, str]], +) -> str: + """Format Kimi web_search citations into the tool_end payload. + + Same shape parseSourcesFromResult on the frontend expects for the + other built-in web_search providers: `Title: ...\\nURL: ...\\n + Snippet: ...\\n---\\n...`. If no citations were emitted, fall back + to a generic "(search complete)" string so the UI still shows the + tool card transitioning to a completed state. + """ + blocks: list[str] = [] + for cit in citations: + line = f"Title: {cit['title']}\nURL: {cit['url']}" + if cit.get("snippet"): + line += f"\nSnippet: {cit['snippet']}" + blocks.append(line) + return synthetic_chunk_fn( + { + "type": "tool_end", + "tool_call_id": tool_call_id, + "result": "\n---\n".join(blocks) if blocks else "(search complete)", + } + ) + + class ExternalProviderClient: """Async proxy for OpenAI-compatible external LLM APIs.""" @@ -199,6 +227,7 @@ class ExternalProviderClient: top_k: Optional[int] = None, enable_thinking: Optional[bool] = None, reasoning_effort: Optional[str] = None, + enabled_tools: Optional[list[str]] = None, stream: bool = True, ) -> AsyncGenerator[str, None]: """ @@ -222,6 +251,7 @@ class ExternalProviderClient: top_k, enable_thinking, reasoning_effort, + enabled_tools, ): yield line return @@ -240,6 +270,28 @@ class ExternalProviderClient: max_tokens, enable_thinking, reasoning_effort, + enabled_tools, + ): + yield line + return + + # Kimi's $web_search is a builtin_function that requires a client + # round-trip: the first call returns a tool_calls envelope with + # function.arguments populated; the caller echoes those arguments + # back as a role=tool message; the second call streams the final + # answer with the search incorporated. The doc also mandates + # disabling thinking while $web_search is active. Route to a + # dedicated helper so the default OAI-compat path stays single-pass. + # https://platform.kimi.ai/docs/guide/use-web-search + if ( + self.provider_type == "kimi" + and enabled_tools + and "web_search" in enabled_tools + ): + async for line in self._stream_kimi_web_search( + messages, + model, + max_tokens, ): yield line return @@ -317,6 +369,29 @@ class ExternalProviderClient: else: body["reasoning"] = {"enabled": False} + # OpenRouter web-search plugin — universal shape that works + # for every model id, including the `openrouter/free` and + # `openrouter/auto` meta-routers. Documented at + # https://openrouter.ai/docs/guides/features/plugins/web-search + # The `:online` model-suffix shortcut is "exactly equivalent + # to" this plugin per the same doc, but only works on + # concrete model ids — meta-routers reject the suffix. + # `plugins: [{id: "web"}]` works everywhere, no model id + # rewrite needed, and idempotent if some future call site + # adds the entry first. + if enabled_tools and "web_search" in enabled_tools: + plugins = list(body.get("plugins") or []) + if not any( + isinstance(p, dict) and p.get("id") == "web" for p in plugins + ): + plugins.append({"id": "web"}) + body["plugins"] = plugins + logger.info( + "OpenRouter web_search: attached plugins=[{id: 'web'}] " + "(model=%s)", + body.get("model"), + ) + url = f"{self.base_url}/chat/completions" logger.info( "Proxying chat completion to %s (provider=%s, model=%s)", @@ -362,6 +437,94 @@ class ExternalProviderClient: # error" in the UI with no trail on the server side. event_counts: dict[str, int] = {} chosen_model: Optional[str] = None + # Web-search tool-card synthesis for OpenRouter. The gateway + # doesn't emit structured web_search_call events — citations + # come back as `annotations` of type=url_citation on delta / + # message objects. Mirror the OpenAI/Anthropic UX by yielding + # a synthetic tool_start at stream open and tool_end at + # stream close with the collected citation list. + web_search_active = ( + self.provider_type == "openrouter" + and bool(enabled_tools) + and "web_search" in (enabled_tools or []) + ) + web_search_tool_id = "openrouter_web_search" + web_search_citations: list[dict[str, str]] = [] + web_search_tool_started = False + web_search_tool_ended = False + + def _emit_synthetic_tool_event(payload: dict[str, Any]) -> str: + chunk = { + "id": f"chatcmpl-{self.provider_type}-synthetic", + "object": "chat.completion.chunk", + "choices": [ + { + "index": 0, + "delta": {}, + "finish_reason": None, + } + ], + "_toolEvent": payload, + } + return f"data: {_json.dumps(chunk)}" + + def _record_or_url_citation(payload: Any) -> None: + if not isinstance(payload, dict): + return + if payload.get("type") != "url_citation": + return + # OpenRouter (and OpenAI Chat Completions web_search) + # nest the citation under url_citation; some variants + # ship the fields flat on the annotation itself. Accept + # both. + cit = payload.get("url_citation") + if not isinstance(cit, dict): + cit = payload + url = cit.get("url", "") if isinstance(cit, dict) else "" + if not url or not isinstance(url, str): + return + if any(c["url"] == url for c in web_search_citations): + return + title = cit.get("title") or url + snippet = cit.get("content") or cit.get("snippet") or "" + web_search_citations.append( + { + "url": url, + "title": title, + "snippet": snippet if isinstance(snippet, str) else "", + } + ) + + def _build_web_search_tool_end() -> str: + blocks: list[str] = [] + for cit in web_search_citations: + line = f"Title: {cit['title']}\nURL: {cit['url']}" + if cit.get("snippet"): + line += f"\nSnippet: {cit['snippet']}" + blocks.append(line) + return _emit_synthetic_tool_event( + { + "type": "tool_end", + "tool_call_id": web_search_tool_id, + "result": ( + "\n---\n".join(blocks) + if blocks + else "(search complete)" + ), + } + ) + + if web_search_active: + yield _emit_synthetic_tool_event( + { + "type": "tool_start", + "tool_name": "web_search", + "tool_call_id": web_search_tool_id, + "arguments": {}, + } + ) + web_search_tool_started = True + try: while True: try: @@ -374,6 +537,17 @@ class ExternalProviderClient: data_str = line[len("data:") :].strip() if data_str == "[DONE]": event_counts["done"] = event_counts.get("done", 0) + 1 + # Emit synthetic tool_end with collected + # citations BEFORE forwarding [DONE], so the + # tool-card transitions to "complete" in the + # UI before the stream closes. + if ( + web_search_active + and web_search_tool_started + and not web_search_tool_ended + ): + yield _build_web_search_tool_end() + web_search_tool_ended = True elif data_str: try: parsed = _json.loads(data_str) @@ -406,17 +580,52 @@ class ExternalProviderClient: parsed.get("model"), str ): chosen_model = parsed["model"] + # When the user has web_search on, scan + # every chunk's delta and message + # objects for url_citation annotations. + # Different OpenRouter upstreams place + # them in different spots. + if web_search_active: + choices = parsed.get("choices") or [] + if isinstance(choices, list): + for choice in choices: + if not isinstance(choice, dict): + continue + for envelope in ( + choice.get("delta"), + choice.get("message"), + ): + if not isinstance(envelope, dict): + continue + for ann in ( + envelope.get("annotations") + or [] + ): + _record_or_url_citation(ann) yield line + # Stream ended without [DONE] (some upstreams just close + # the connection). Emit tool_end so the card doesn't + # stay in "running" forever. + if ( + web_search_active + and web_search_tool_started + and not web_search_tool_ended + ): + yield _build_web_search_tool_end() + web_search_tool_ended = True except GeneratorExit: await response.aclose() # set PoolByteStream._closed=True FIRST await lines_gen.aclose() # now safe — aclose() is a no-op raise finally: logger.info( - "%s stream complete (model=%s, chosen=%s, events=%s)", + "%s stream complete (model=%s, chosen=%s, " + "web_search_requested=%s, citations=%s, events=%s)", self.provider_type, model, chosen_model, + web_search_active, + len(web_search_citations), event_counts, ) await response.aclose() @@ -444,6 +653,384 @@ class ExternalProviderClient: self.provider_type, ) + async def _stream_kimi_web_search( + self, + messages: list[dict[str, Any]], + model: str, + max_tokens: Optional[int], + ) -> AsyncGenerator[str, None]: + """ + Kimi $web_search round-trip. + + Wire flow (per https://platform.kimi.ai/docs/guide/use-web-search): + 1. POST messages with tools=[{type: "builtin_function", + function: {name: "$web_search"}}] and thinking=disabled. + 2. Stream the first response — accumulate function.arguments + across tool_call deltas until finish_reason="tool_calls". + Do NOT forward those tool_call chunks to the client (they + are an internal protocol step, not user-visible output). + 3. Build a second request: original messages + the assistant + message carrying the tool_calls + a role=tool message that + echoes the same arguments back verbatim (per Kimi docs, + the caller "just needs to submit tool_call.function.arguments + to Kimi as they are" — the server actually runs the search). + 4. Stream the second response — that is the final answer the + user sees, with search results already incorporated. + + We synthesize tool_start (with the parsed query) when step (2) + completes, and tool_end (with any url_citation annotations the + second stream emits) before [DONE], so the chat UI shows the + same web-search tool card as the other providers. + """ + url = f"{self.base_url}/chat/completions" + body: dict[str, Any] = { + "model": model, + "messages": messages, + "stream": True, + # $web_search forbids thinking; sending the toggle silently + # would have the server reject the request with 400. + "thinking": {"type": "disabled"}, + "tools": [ + {"type": "builtin_function", "function": {"name": "$web_search"}} + ], + } + if max_tokens is not None: + body["max_tokens"] = max_tokens + + # Strip body fields the Kimi registry declares unusable + # (temperature/top_p — see body_omit in providers.py). + from core.inference.providers import get_provider_info + + provider_info = get_provider_info(self.provider_type) or {} + for field in provider_info.get("body_omit", ()): + body.pop(field, None) + + tool_call_id = "kimi_web_search" + synthetic_id = f"chatcmpl-{self.provider_type}-synthetic" + + def _synthetic_chunk(payload: dict[str, Any]) -> str: + chunk = { + "id": synthetic_id, + "object": "chat.completion.chunk", + "choices": [{"index": 0, "delta": {}, "finish_reason": None}], + "_toolEvent": payload, + } + return f"data: {_json.dumps(chunk)}" + + logger.info( + "Kimi $web_search round-trip starting (model=%s, url=%s)", + model, + url, + ) + + # ---- First call: collect the model's $web_search tool_call ---- + tool_calls_acc: dict[int, dict[str, Any]] = {} + try: + async with _http_client.stream( + "POST", + url, + json = body, + headers = self._auth_headers(), + timeout = self._stream_timeout, + ) as response: + if response.status_code != 200: + error_body = await response.aread() + error_text = error_body.decode("utf-8", errors = "replace") + logger.error( + "Kimi first-call returned %d: %s", + response.status_code, + error_text[:500], + ) + yield _error_sse_line( + response.status_code, error_text, self.provider_type + ) + return + + lines_gen = response.aiter_lines().__aiter__() + try: + while True: + try: + line = await lines_gen.__anext__() + except StopAsyncIteration: + break + if not line.strip() or not line.startswith("data:"): + continue + data_str = line[len("data:") :].strip() + if data_str == "[DONE]": + break + try: + parsed = _json.loads(data_str) + except Exception: + continue + for choice in parsed.get("choices") or []: + if not isinstance(choice, dict): + continue + delta = choice.get("delta") or {} + for tc in delta.get("tool_calls") or []: + if not isinstance(tc, dict): + continue + idx = tc.get("index", 0) + slot = tool_calls_acc.setdefault( + idx, + { + "id": tc.get("id") or f"call_{idx}", + "type": "function", + "function": {"name": "", "arguments": ""}, + }, + ) + if tc.get("id"): + slot["id"] = tc["id"] + fn = tc.get("function") or {} + if fn.get("name"): + slot["function"]["name"] = fn["name"] + if fn.get("arguments"): + slot["function"]["arguments"] += fn["arguments"] + if choice.get("finish_reason") == "tool_calls": + break + except GeneratorExit: + await response.aclose() + await lines_gen.aclose() + raise + finally: + await response.aclose() + await lines_gen.aclose() + except httpx.HTTPError as exc: + logger.error("Kimi first-call HTTP error: %s", exc) + yield _error_sse_line( + 502, + f"Error communicating with kimi: {exc}", + self.provider_type, + ) + return + + # If the model decided not to search, fall back to a plain + # streaming call without the builtin tool. That mirrors the UX + # of every other provider when web_search is on but the model + # didn't actually need it. + search_calls = [ + tc + for tc in tool_calls_acc.values() + if tc["function"]["name"] == "$web_search" + ] + if not search_calls: + logger.info( + "Kimi $web_search: model did not invoke search; " + "falling back to plain stream" + ) + fallback_body = dict(body) + fallback_body.pop("tools", None) + try: + async with _http_client.stream( + "POST", + url, + json = fallback_body, + headers = self._auth_headers(), + timeout = self._stream_timeout, + ) as response: + if response.status_code != 200: + error_body = await response.aread() + error_text = error_body.decode("utf-8", errors = "replace") + logger.error( + "Kimi fallback returned %d: %s", + response.status_code, + error_text[:500], + ) + yield _error_sse_line( + response.status_code, error_text, self.provider_type + ) + return + # Manual __anext__ loop instead of `async for` — see the + # comment in stream_chat_completion for the Python 3.13 + + # httpcore 1.0.x GeneratorExit interaction this avoids. + lines_gen = response.aiter_lines().__aiter__() + try: + while True: + try: + line = await lines_gen.__anext__() + except StopAsyncIteration: + break + if line.strip(): + yield line + except GeneratorExit: + await response.aclose() + await lines_gen.aclose() + raise + finally: + await response.aclose() + await lines_gen.aclose() + except httpx.HTTPError as exc: + logger.error("Kimi fallback HTTP error: %s", exc) + yield _error_sse_line( + 502, + f"Error communicating with kimi: {exc}", + self.provider_type, + ) + return + + # Synthesize tool_start with the parsed search query so the + # chat UI's web-search card shows "Searching for: ...". + first_args_raw = search_calls[0]["function"]["arguments"] or "{}" + try: + first_args = _json.loads(first_args_raw) + except Exception: + first_args = {} + # Log the raw arguments so we can confirm the server actually + # ran the search. The shape is documented loosely but in practice + # the model emits `{"search_result":{"search_id":...}, + # "usage":{"total_tokens":N}}` — an opaque receipt where N is the + # token cost of the injected search context. The query string is + # NOT present; Kimi runs the search server-side during the first + # call and bakes the results straight into the model's context. + logger.info( + "Kimi $web_search: %d tool_call(s), args[0]=%s", + len(search_calls), + first_args_raw[:500], + ) + first_args_search_tokens: Optional[int] = None + if isinstance(first_args, dict): + usage_block = first_args.get("usage") + if isinstance(usage_block, dict): + tok = usage_block.get("total_tokens") + if isinstance(tok, int): + first_args_search_tokens = tok + yield _synthetic_chunk( + { + "type": "tool_start", + "tool_name": "web_search", + "tool_call_id": tool_call_id, + "arguments": first_args if isinstance(first_args, dict) else {}, + } + ) + # Kimi's search has already executed server-side by the time the + # first call returns (the tool_call envelope encodes the search + # result reference, not a query for us to dispatch). Emit + # tool_end NOW so the UI's web-search card transitions to + # "complete" before the second call starts streaming the + # answer, instead of after — otherwise the card sits in + # "running" all the way through the answer streaming and the + # user perceives the model answering before search finishes. + yield _build_kimi_tool_end(_synthetic_chunk, tool_call_id, []) + + # ---- Second call: echo the tool_calls back and stream answer ---- + assistant_msg = { + "role": "assistant", + "content": "", + "tool_calls": list(tool_calls_acc.values()), + } + tool_msgs = [ + { + "role": "tool", + "tool_call_id": tc["id"], + "name": tc["function"]["name"], + "content": tc["function"]["arguments"], + } + for tc in tool_calls_acc.values() + ] + followup_body = dict(body) + followup_body["messages"] = list(messages) + [assistant_msg] + tool_msgs + # Ask the SSE stream to include a final `usage` block so we can + # see prompt_tokens (which jumps to thousands when the server + # injects search context). Without this, OpenAI-compat streams + # omit usage entirely. Kimi follows the same convention. + followup_body["stream_options"] = {"include_usage": True} + # Keep the tool definition on the second call so the model can + # decide to search again mid-turn if needed. Kimi's doc shows + # the same tools array on every step. + + try: + async with _http_client.stream( + "POST", + url, + json = followup_body, + headers = self._auth_headers(), + timeout = self._stream_timeout, + ) as response: + if response.status_code != 200: + error_body = await response.aread() + error_text = error_body.decode("utf-8", errors = "replace") + logger.error( + "Kimi second-call returned %d: %s", + response.status_code, + error_text[:500], + ) + yield _error_sse_line( + response.status_code, error_text, self.provider_type + ) + return + + lines_gen = response.aiter_lines().__aiter__() + # Diagnostics: latch usage.prompt_tokens from the final + # chunk. The Kimi docs say search results count toward + # prompt_tokens, so a big value here is direct evidence + # the server actually injected results into context. + last_usage: Optional[dict[str, Any]] = None + annotation_shapes: set[str] = set() + try: + while True: + try: + line = await lines_gen.__anext__() + except StopAsyncIteration: + break + if not line.strip(): + continue + if line.startswith("data:"): + data_str = line[len("data:") :].strip() + if data_str and data_str != "[DONE]": + try: + parsed = _json.loads(data_str) + except Exception: + parsed = None + if isinstance(parsed, dict): + usage = parsed.get("usage") + if isinstance(usage, dict): + last_usage = usage + # Scan annotations only for diagnostics — + # Kimi today doesn't emit url_citation, but + # if a future model version starts to we'll + # see the type name in the final log line + # and can wire it into the tool_end payload. + for choice in parsed.get("choices") or []: + if not isinstance(choice, dict): + continue + for envelope in ( + choice.get("delta"), + choice.get("message"), + ): + if not isinstance(envelope, dict): + continue + for ann in ( + envelope.get("annotations") or [] + ): + if isinstance(ann, dict): + annotation_shapes.add( + str(ann.get("type") or "?") + ) + yield line + except GeneratorExit: + await response.aclose() + await lines_gen.aclose() + raise + finally: + logger.info( + "Kimi $web_search complete (model=%s, " + "search_ctx_tokens=%s, annotation_types=%s, " + "prompt_tokens=%s, completion_tokens=%s)", + model, + first_args_search_tokens, + sorted(annotation_shapes) or None, + (last_usage or {}).get("prompt_tokens"), + (last_usage or {}).get("completion_tokens"), + ) + await response.aclose() + await lines_gen.aclose() + except httpx.HTTPError as exc: + logger.error("Kimi second-call HTTP error: %s", exc) + yield _error_sse_line( + 502, + f"Error communicating with kimi: {exc}", + self.provider_type, + ) + async def _stream_anthropic( self, messages: list[dict[str, Any]], @@ -454,6 +1041,7 @@ class ExternalProviderClient: top_k: Optional[int] = None, enable_thinking: Optional[bool] = None, reasoning_effort: Optional[str] = None, + enabled_tools: Optional[list[str]] = None, ) -> AsyncGenerator[str, None]: """ Call the Anthropic Messages API and translate its SSE to OpenAI format. @@ -602,6 +1190,25 @@ class ExternalProviderClient: if body.get("max_tokens", 0) <= budget_tokens: body["max_tokens"] = budget_tokens + 1024 + # Anthropic server-side web_search — see + # https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/web-search-tool + # The tool type is date-pinned (web_search_20250305 today) and + # Anthropic dispatches search calls server-side, returning + # server_tool_use + web_search_tool_result blocks in the SSE + # stream, plus url-citation annotations on text deltas. We + # translate all of that into our local _toolEvent shape so the + # chat UI renders web_search exactly like OpenAI's path. + if enabled_tools and "web_search" in enabled_tools: + anthropic_tools = list(body.get("tools") or []) + anthropic_tools.append( + { + "type": "web_search_20250305", + "name": "web_search", + "max_uses": 5, + } + ) + body["tools"] = anthropic_tools + url = f"{self.base_url}/messages" completion_id = f"chatcmpl-anthropic-{model.replace('/', '-')}" @@ -659,6 +1266,17 @@ class ExternalProviderClient: # "no thinking content" — distinguishes "Anthropic never sent # thinking_delta" from "frontend didn't render the chunks". event_counts: dict[str, int] = {} + # web_search state. Anthropic emits the query inside an + # `input_json_delta` stream on a `server_tool_use` content + # block, then a separate `web_search_tool_result` block + # with the URL list. Unlike OpenAI we get per-call results + # directly, so each tool card carries its own citations. + # `current_server_tool_use`: {id, name, partial_json_buffer} + # `current_result_block`: {tool_use_id, results} + # Both go to None when the matching content_block_stop fires. + current_server_tool_use: Optional[dict[str, Any]] = None + current_result_block: Optional[dict[str, Any]] = None + web_search_calls: dict[str, dict[str, Any]] = {} def _content_chunk(text: str) -> str: chunk = { @@ -674,6 +1292,37 @@ class ExternalProviderClient: } return f"data: {_json.dumps(chunk)}" + def _emit_tool_event(payload: dict[str, Any]) -> str: + chunk = { + "id": completion_id, + "object": "chat.completion.chunk", + "choices": [ + { + "index": 0, + "delta": {}, + "finish_reason": None, + } + ], + "_toolEvent": payload, + } + return f"data: {_json.dumps(chunk)}" + + def _format_web_search_results( + results: list[Any], + ) -> str: + blocks: list[str] = [] + for r in results: + if not isinstance(r, dict): + continue + if r.get("type") != "web_search_result": + continue + url = r.get("url", "") + title = r.get("title") or url + if not url: + continue + blocks.append(f"Title: {title}\nURL: {url}") + return "\n---\n".join(blocks) + try: while True: try: @@ -702,7 +1351,39 @@ class ExternalProviderClient: key = event_type or "<unknown>" event_counts[key] = event_counts.get(key, 0) + 1 - if event_type == "content_block_delta": + if event_type == "content_block_start": + content_block = event.get("content_block") or {} + block_type = content_block.get("type") + if ( + block_type == "server_tool_use" + and content_block.get("name") == "web_search" + ): + tool_use_id = content_block.get("id", "") or ( + f"ws_{len(web_search_calls)}" + ) + current_server_tool_use = { + "id": tool_use_id, + "buffer": "", + } + web_search_calls[tool_use_id] = { + "query": "", + "results": [], + } + elif block_type == "web_search_tool_result": + tool_use_id = content_block.get("tool_use_id", "") + # Anthropic sometimes ships the full results + # list on the start event; sometimes deltas + # follow. Capture whatever is present and + # finalize on content_block_stop. + content = content_block.get("content") or [] + current_result_block = { + "tool_use_id": tool_use_id, + "results": list(content) + if isinstance(content, list) + else [], + } + + elif event_type == "content_block_delta": delta = event.get("delta", {}) delta_type = delta.get("type") if delta_type == "thinking_delta": @@ -730,16 +1411,79 @@ class ExternalProviderClient: text = delta.get("text", "") if text: yield _content_chunk(text) + # Citations on text deltas are attached + # per-call by Anthropic via the + # `web_search_tool_result` block; we don't + # need to scrape them off the text events. + elif ( + delta_type == "input_json_delta" + and current_server_tool_use is not None + ): + # Streamed partial_json carrying the search + # query. Buffer until content_block_stop. + current_server_tool_use["buffer"] += delta.get( + "partial_json", "" + ) # signature_delta and any other delta types are # intentionally skipped — they carry trust / # verification metadata, not user-visible content. elif event_type == "content_block_stop": - # Close the <think> tag when the thinking block - # ends, in case no text_delta follows (e.g. - # display=omitted on Claude 4.7, or thinking-only - # turns). - if thinking_open: + if current_server_tool_use is not None: + # End of the server_tool_use block — parse the + # accumulated input_json into a query and + # emit tool_start. The matching tool_end fires + # later when the web_search_tool_result block + # closes with the actual results. + buffer = current_server_tool_use["buffer"] + query = "" + if buffer: + try: + parsed = _json.loads(buffer) + if isinstance(parsed, dict): + q = parsed.get("query", "") + if isinstance(q, str): + query = q + except Exception: + query = "" + tool_use_id = current_server_tool_use["id"] + if tool_use_id in web_search_calls: + web_search_calls[tool_use_id]["query"] = query + yield _emit_tool_event( + { + "type": "tool_start", + "tool_name": "web_search", + "tool_call_id": tool_use_id, + "arguments": ( + {"query": query} if query else {} + ), + } + ) + current_server_tool_use = None + elif current_result_block is not None: + # End of a web_search_tool_result — emit + # tool_end carrying the search results as + # Title:/URL: blocks. parseSourcesFromResult + # on the frontend lifts these into source + # pills at message tail. + tool_use_id = current_result_block["tool_use_id"] + results = current_result_block["results"] + if tool_use_id in web_search_calls: + web_search_calls[tool_use_id]["results"] = results + result_text = _format_web_search_results(results) + yield _emit_tool_event( + { + "type": "tool_end", + "tool_call_id": tool_use_id, + "result": (result_text or "(search complete)"), + } + ) + current_result_block = None + elif thinking_open: + # Close the <think> tag when the thinking block + # ends, in case no text_delta follows (e.g. + # display=omitted on Claude 4.7, or thinking- + # only turns). yield _content_chunk("</think>") thinking_open = False @@ -778,16 +1522,30 @@ class ExternalProviderClient: await lines_gen.aclose() # now safe — aclose() is a no-op raise finally: - # Surface per-event-type counts so reports of "no - # reasoning panel content" can be triaged at a glance: - # zero `content_block_delta:thinking_delta` entries - # means Anthropic skipped thinking for this prompt - # (adaptive can choose to); non-zero means thinking - # arrived and we wrapped it — any visual gap is then - # on the frontend. + # Surface per-event-type counts + web_search summary so + # reports of "no reasoning panel content" / "Search + # didn't do anything" can be triaged at a glance. + web_search_requested = bool( + enabled_tools and "web_search" in enabled_tools + ) + web_search_invocations = len(web_search_calls) + total_results = sum( + len(sc.get("results") or []) for sc in web_search_calls.values() + ) + queries = [ + sc["query"] + for sc in web_search_calls.values() + if sc.get("query") + ] logger.info( - "Anthropic stream event counts (model=%s): %s", + "Anthropic stream complete (model=%s, " + "web_search_requested=%s, web_search_invocations=%s, " + "results=%s, queries=%s, events=%s)", model, + web_search_requested, + web_search_invocations, + total_results, + queries, event_counts, ) await response.aclose() @@ -824,6 +1582,7 @@ class ExternalProviderClient: max_tokens: Optional[int], enable_thinking: Optional[bool], reasoning_effort: Optional[str], + enabled_tools: Optional[list[str]] = None, ) -> AsyncGenerator[str, None]: """ Call OpenAI's /v1/responses endpoint and translate its SSE stream back @@ -918,6 +1677,20 @@ class ExternalProviderClient: if max_tokens is not None: body["max_output_tokens"] = max_tokens + # OpenAI server-side tools — see + # https://developers.openai.com/api/docs/guides/tools + # The frontend's Search button maps to the unified + # enabled_tools=["web_search"] shorthand; translate that into the + # Responses-API tool schema. Other built-in tools (file_search, + # code_interpreter, image_generation, computer_use_preview) can be + # added with the same pattern when we surface their toggles. + if enabled_tools: + tools_array: list[dict[str, Any]] = [] + if "web_search" in enabled_tools: + tools_array.append({"type": "web_search"}) + if tools_array: + body["tools"] = tools_array + url = f"{self.base_url}/responses" completion_id = f"chatcmpl-openai-{model.replace('/', '-')}" @@ -950,6 +1723,64 @@ class ExternalProviderClient: done_emitted = False reasoning_open = False reasoning_emitted = False + # Per-call state for OpenAI's server-side web_search tool. Mapped + # back into our local _toolEvent shape so the existing chat-UI + # renderer surfaces web_search the same way it does for local + # tool calls: a "Searching…" tool-call card, then a `tool_end` + # carrying citations formatted as + # Title: …\nURL: …\nSnippet: …\n---\n… + # blocks (which the frontend's parseSourcesFromResult lifts + # into source content parts at end of stream). + # web_search_calls preserves insertion order so we can apply + # the aggregated citation list onto the *last* call's + # tool_end — that's the one the frontend's source-pill + # extraction reads (parseSourcesFromResult flatMaps every + # web_search result, so a single non-empty result is enough + # to surface all sources at message tail). + # OpenAI emits url_citation annotations on text deltas, not + # per call — there's no wire field linking a citation back + # to a specific search invocation. Hence the shared list. + # web_search_calls: { item_id -> {query} } + web_search_calls: dict[str, dict[str, Any]] = {} + all_url_citations: list[dict[str, str]] = [] + + def _emit_tool_event(payload: dict[str, Any]) -> str: + chunk = { + "id": completion_id, + "object": "chat.completion.chunk", + "choices": [ + { + "index": 0, + "delta": {}, + "finish_reason": None, + } + ], + "_toolEvent": payload, + } + return f"data: {_json.dumps(chunk)}" + + def _record_url_citation(payload: dict[str, Any]) -> None: + """Append a url_citation onto the shared all_url_citations + list. Dedup by URL — the same source can be cited multiple + times across deltas. We do NOT try to attribute citations + to individual web_search_call invocations because OpenAI's + annotation events don't carry that linkage.""" + if payload.get("type") != "url_citation": + return + url = payload.get("url", "") + if not url: + return + if any(c["url"] == url for c in all_url_citations): + return + title = payload.get("title") or url + snippet = payload.get("snippet") or payload.get("quote") or "" + all_url_citations.append( + { + "url": url, + "title": title, + "snippet": snippet, + } + ) def _extract_reasoning_text(payload: Any) -> str: if payload is None: @@ -1023,13 +1854,39 @@ class ExternalProviderClient: yield _chunk_with_text("</think>") reasoning_open = False yield _chunk_with_text(delta_text) + # Some API versions inline url citations on the + # delta event itself rather than as a separate + # response.output_text.annotation.added event. + for ann in event.get("annotations") or []: + if isinstance(ann, dict): + _record_url_citation(ann) - elif event_type == "response.output_item.done": + elif event_type == "response.output_text.annotation.added": + ann = event.get("annotation") + if isinstance(ann, dict): + _record_url_citation(ann) + + elif event_type == "response.output_item.added": + # Track the call early but do NOT emit tool_start + # yet — action.query is not reliably populated on + # added across OpenAI API versions, and the + # frontend's tool_start is a one-shot push (no + # update mechanism). Wait for output_item.done. item = event.get("item", {}) if ( isinstance(item, dict) - and item.get("type") == "reasoning" + and item.get("type") == "web_search_call" ): + item_id = item.get("id", "") or ( + f"ws_{len(web_search_calls)}" + ) + web_search_calls.setdefault(item_id, {"query": ""}) + + elif event_type == "response.output_item.done": + item = event.get("item", {}) + if not isinstance(item, dict): + continue + if item.get("type") == "reasoning": summary_text = _extract_reasoning_text( item.get("summary") ) @@ -1039,6 +1896,46 @@ class ExternalProviderClient: reasoning_open = True yield _chunk_with_text(summary_text) reasoning_emitted = True + elif item.get("type") == "web_search_call": + # done is the canonical place to read the + # query, so emit both tool_start and tool_end + # here. Frontend then renders a card per call + # with the proper "Searching: <query>" label. + # Citations are aggregated separately and the + # *last* call's result is overwritten at + # response.completed with the citation list + # (so the source-pill extraction at message + # tail surfaces them once). + item_id = item.get("id", "") or ( + f"ws_{len(web_search_calls)}" + ) + action = item.get("action") + query = ( + action.get("query", "") + if isinstance(action, dict) + else "" + ) + web_search_calls[item_id] = {"query": query} + yield _emit_tool_event( + { + "type": "tool_start", + "tool_name": "web_search", + "tool_call_id": item_id, + "arguments": ( + {"query": query} if query else {} + ), + } + ) + yield _emit_tool_event( + { + "type": "tool_end", + "tool_call_id": item_id, + # Empty result — the last call gets + # overwritten with citations at + # response.completed. + "result": "", + } + ) elif isinstance(event_type, str) and "reasoning" in event_type: reasoning_delta = _extract_reasoning_text(event) @@ -1053,6 +1950,32 @@ class ExternalProviderClient: if reasoning_open: yield _chunk_with_text("</think>") reasoning_open = False + # Apply the aggregated citation list onto the + # *last* web_search call by overwriting its + # tool_end result. The frontend's + # parseSourcesFromResult flatMaps every + # web_search tool-call result, so a single + # non-empty result is enough to surface the + # whole source-pill set at the message tail — + # no need to fan out across every card (which + # would just duplicate the same pills). + if web_search_calls and all_url_citations: + last_id = list(web_search_calls.keys())[-1] + blocks: list[str] = [] + for cit in all_url_citations: + line = ( + f"Title: {cit['title']}\n" f"URL: {cit['url']}" + ) + if cit.get("snippet"): + line += f"\nSnippet: {cit['snippet']}" + blocks.append(line) + yield _emit_tool_event( + { + "type": "tool_end", + "tool_call_id": last_id, + "result": "\n---\n".join(blocks), + } + ) chunk = { "id": completion_id, "object": "chat.completion.chunk", @@ -1070,6 +1993,29 @@ class ExternalProviderClient: if reasoning_open: yield _chunk_with_text("</think>") reasoning_open = False + # Same backfill as response.completed — apply + # whatever citations we managed to gather + # before truncation onto the last call. All + # earlier tool cards already have their proper + # query + empty placeholder result from the + # output_item.done emissions above. + if web_search_calls and all_url_citations: + last_id = list(web_search_calls.keys())[-1] + blocks = [] + for cit in all_url_citations: + line = ( + f"Title: {cit['title']}\n" f"URL: {cit['url']}" + ) + if cit.get("snippet"): + line += f"\nSnippet: {cit['snippet']}" + blocks.append(line) + yield _emit_tool_event( + { + "type": "tool_end", + "tool_call_id": last_id, + "result": "\n---\n".join(blocks), + } + ) chunk = { "id": completion_id, "object": "chat.completion.chunk", @@ -1103,6 +2049,31 @@ class ExternalProviderClient: await lines_gen.aclose() raise finally: + # Summarise what the model actually did this turn so + # support reports of "I clicked Search and got nothing" + # can be triaged at a glance: was the tool requested, + # did OpenAI invoke it, and how many sources came back? + web_search_requested = bool( + enabled_tools and "web_search" in enabled_tools + ) + web_search_invocations = len(web_search_calls) + total_citations = len(all_url_citations) + queries = [ + sc["query"] + for sc in web_search_calls.values() + if sc.get("query") + ] + logger.info( + "OpenAI Responses stream complete (model=%s, " + "web_search_requested=%s, web_search_invocations=%s, " + "citations=%s, queries=%s, reasoning_emitted=%s)", + model, + web_search_requested, + web_search_invocations, + total_citations, + queries, + reasoning_emitted, + ) await response.aclose() await lines_gen.aclose() diff --git a/studio/backend/routes/inference.py b/studio/backend/routes/inference.py index 59928be3cf..55d74d1cbe 100644 --- a/studio/backend/routes/inference.py +++ b/studio/backend/routes/inference.py @@ -1595,6 +1595,7 @@ async def _proxy_to_external_provider( top_k = payload.top_k, enable_thinking = payload.enable_thinking, reasoning_effort = payload.reasoning_effort, + enabled_tools = payload.enabled_tools, stream = payload.stream, ) try: diff --git a/studio/frontend/src/components/assistant-ui/thread.tsx b/studio/frontend/src/components/assistant-ui/thread.tsx index d75a8cce10..4988cd5a46 100644 --- a/studio/frontend/src/components/assistant-ui/thread.tsx +++ b/studio/frontend/src/components/assistant-ui/thread.tsx @@ -496,6 +496,9 @@ const ReasoningToggle: FC = () => { externalSelection != null ? externalProviders.find((p) => p.id === externalSelection.providerId) : undefined; + const isKimiExternal = selectedExternalProvider?.providerType === "kimi"; + const toolsEnabled = useChatRuntimeStore((s) => s.toolsEnabled); + const setToolsEnabled = useChatRuntimeStore((s) => s.setToolsEnabled); const effectiveExternalModelId = selectedExternalProvider?.providerType === "openrouter" && externalSelection?.modelId === "openrouter/free" && @@ -587,6 +590,11 @@ const ReasoningToggle: FC = () => { setReasoningEffort(level); setReasoningEnabled(true); applyQwenThinkingParams(true); + // Kimi's $web_search builtin forbids thinking, so + // enabling thinking flips the Search pill off. + if (isKimiExternal && toolsEnabled) { + setToolsEnabled(false); + } }} > {formatEffortLabel(level)} @@ -613,6 +621,11 @@ const ReasoningToggle: FC = () => { const next = !reasoningEnabled; setReasoningEnabled(next); applyQwenThinkingParams(next); + // Mutual exclusion with the Search pill on Kimi — see the + // dropdown branch above and shared-composer for the same rule. + if (isKimiExternal && next && toolsEnabled) { + setToolsEnabled(false); + } }} className="composer-pill-btn" data-active={ @@ -680,16 +693,44 @@ const WebSearchToggle: FC = () => { const modelLoaded = useChatRuntimeStore( (s) => !!s.params.checkpoint && !s.modelLoading, ); + const checkpoint = useChatRuntimeStore((s) => s.params.checkpoint); const supportsTools = useChatRuntimeStore((s) => s.supportsTools); + // External providers (OpenAI today) expose a server-side web_search tool + // even when the local tool runtime is unavailable — gate the Search pill + // on either source so it lights up on external models too. Mirror of + // shared-composer's searchDisabled. + const supportsBuiltinWebSearch = useChatRuntimeStore( + (s) => s.supportsBuiltinWebSearch, + ); const toolsEnabled = useChatRuntimeStore((s) => s.toolsEnabled); const setToolsEnabled = useChatRuntimeStore((s) => s.setToolsEnabled); - const disabled = !(modelLoaded && supportsTools); + const setReasoningEnabled = useChatRuntimeStore((s) => s.setReasoningEnabled); + const externalProviders = useExternalProvidersStore((s) => s.providers); + const externalSelection = parseExternalModelId(checkpoint); + const selectedExternalProvider = + externalSelection != null + ? externalProviders.find((p) => p.id === externalSelection.providerId) + : undefined; + const isKimiExternal = selectedExternalProvider?.providerType === "kimi"; + const disabled = + !modelLoaded || !(supportsTools || supportsBuiltinWebSearch); return ( <button type="button" disabled={disabled} - onClick={() => setToolsEnabled(!toolsEnabled)} + onClick={() => { + const next = !toolsEnabled; + setToolsEnabled(next); + // Kimi's $web_search builtin requires thinking=disabled (see + // https://platform.kimi.ai/docs/guide/use-web-search). Keep + // the two pills mutually exclusive so the visible state always + // matches what the backend ends up sending. + if (isKimiExternal) { + setReasoningEnabled(!next); + applyQwenThinkingParams(!next); + } + }} className="composer-pill-btn" data-active={toolsEnabled && !disabled ? "true" : "false"} aria-label={toolsEnabled ? "Disable web search" : "Enable web search"} diff --git a/studio/frontend/src/features/chat/api/chat-adapter.ts b/studio/frontend/src/features/chat/api/chat-adapter.ts index 3e8cec0443..30b256f8ba 100644 --- a/studio/frontend/src/features/chat/api/chat-adapter.ts +++ b/studio/frontend/src/features/chat/api/chat-adapter.ts @@ -35,6 +35,7 @@ import { getExternalMinOutputTokens, getExternalReasoningCapabilities, getProviderCapabilities, + providerSupportsBuiltinWebSearch, } from "../provider-capabilities"; import { useChatRuntimeStore } from "../stores/chat-runtime-store"; import { isMultimodalResponse } from "../types/api"; @@ -1005,6 +1006,20 @@ export function createOpenAIStreamAdapter(): ChatModelAdapter { ...(externalCapabilities?.presencePenalty ? { presence_penalty: params.presencePenalty } : {}), + // Built-in web search: when the user has the Search toggle + // on AND the active provider supports a server-side + // web_search tool (currently OpenAI's /v1/responses), pass + // the enable_tools shorthand. Backend translates + // enabled_tools=["web_search"] into the provider's tool + // schema — for OpenAI that's `tools: [{type:"web_search"}]` + // on the Responses body, see _stream_openai_responses. + ...(toolsEnabled && + providerSupportsBuiltinWebSearch(externalProvider.providerType) + ? { + enable_tools: true, + enabled_tools: ["web_search"], + } + : {}), provider_id: externalProvider.id, provider_type: externalBackendProviderType, external_model: externalSelection.modelId, diff --git a/studio/frontend/src/features/chat/chat-page.tsx b/studio/frontend/src/features/chat/chat-page.tsx index 29428b2ef9..3fa294d465 100644 --- a/studio/frontend/src/features/chat/chat-page.tsx +++ b/studio/frontend/src/features/chat/chat-page.tsx @@ -50,6 +50,7 @@ import { clampReasoningEffortToLevels, getExternalReasoningCapabilities, getProviderCapabilities, + providerSupportsBuiltinWebSearch, } from "./provider-capabilities"; import { useChatModelRuntime } from "./hooks/use-chat-model-runtime"; import { @@ -678,9 +679,56 @@ export function ChatPage(): ReactElement { preferredEffort, effortLevels, ); + // Per-provider default effort. Anthropic gets the highest available + // level (xhigh on 4.6/4.7, high on 4.5) since Claude's adaptive + // thinking adjusts cost per turn — sitting at the top of the dial + // gives users the strongest answers and the model can still skip + // thinking when the turn is trivial. OpenAI gets "high" by default + // — the gpt-5.x reasoning models accept high across the board and + // it's the right cost/quality sweet spot for Responses-API tools + // (web search included). Everyone else gets "medium" as a balanced + // default. Users can pick another level via the Think dropdown. + const isAnthropic = provider?.providerType === "anthropic"; + const isOpenAI = provider?.providerType === "openai"; + const anthropicTopEffort = effortLevels.includes("xhigh") + ? "xhigh" + : effortLevels.includes("high") + ? "high" + : clampedEffort; + const openaiDefaultEffort = effortLevels.includes("high") + ? "high" + : effortLevels.includes("medium") + ? "medium" + : clampedEffort; const nextReasoningEffort = reasoningCaps.supportsReasoning - ? clampedEffort + ? isAnthropic + ? anthropicTopEffort + : isOpenAI + ? openaiDefaultEffort + : effortLevels.includes("medium") + ? "medium" + : clampedEffort : state.reasoningEffort; + const supportsBuiltinWebSearch = providerSupportsBuiltinWebSearch( + provider?.providerType, + ); + // Kimi's k2.6/k2.5 default to thinking enabled on the server side + // (per https://platform.kimi.ai/docs/models). Mirror that default + // in the UI so the Think pill comes up clicked when the user picks + // a Kimi model. The Search pill stays off by default; the mutual- + // exclusion handlers in the composer flip the two when needed. + const isKimi = provider?.providerType === "kimi"; + // Web search is on by default for the two providers we trust most + // for it: Anthropic (web_search_20250305 server tool, structured + // citations) and OpenAI (/v1/responses web_search, structured + // citations). Other providers stay off-by-default — OpenRouter's + // plugins shape and Kimi's $web_search builtin still work when the + // user opts in via the pill, but they're a notch less reliable so + // we don't pre-enable them. + const searchOnByDefault = + supportsBuiltinWebSearch && + (provider?.providerType === "anthropic" || + provider?.providerType === "openai"); useChatRuntimeStore.setState({ supportsReasoning: reasoningCaps.supportsReasoning, reasoningAlwaysOn: reasoningCaps.reasoningAlwaysOn, @@ -690,10 +738,22 @@ export function ChatPage(): ReactElement { reasoningEffort: nextReasoningEffort, reasoningEnabled: reasoningCaps.supportsReasoning ? reasoningCaps.supportsReasoningOff - ? state.reasoningEnabled + ? isKimi + ? true + : state.reasoningEnabled : true : state.reasoningEnabled, supportsPreserveThinking: false, + // External models never give us a local tool runtime (no Code + // execution, no python sandbox), so `supportsTools` must be + // false — that's what gates the Code pill in the composer. + // `supportsBuiltinWebSearch` is the separate flag that lets the + // Search pill light up for providers (currently just OpenAI) who + // run web_search server-side. + supportsTools: false, + supportsBuiltinWebSearch, + toolsEnabled: searchOnByDefault, + codeToolsEnabled: false, }); }, [externalProviders, inferenceParams.checkpoint]); const canCompare = useMemo(() => { @@ -810,8 +870,29 @@ export function ChatPage(): ReactElement { preferredEffort, effortLevels, ); + // Same per-provider default policy as the useEffect path above: + // Anthropic picks the highest available level, OpenAI picks + // "high", everyone else picks "medium". + const isAnthropic = selectedProvider?.providerType === "anthropic"; + const isOpenAI = selectedProvider?.providerType === "openai"; + const anthropicTopEffort = effortLevels.includes("xhigh") + ? "xhigh" + : effortLevels.includes("high") + ? "high" + : clampedEffort; + const openaiDefaultEffort = effortLevels.includes("high") + ? "high" + : effortLevels.includes("medium") + ? "medium" + : clampedEffort; const nextReasoningEffort = reasoningCaps.supportsReasoning - ? clampedEffort + ? isAnthropic + ? anthropicTopEffort + : isOpenAI + ? openaiDefaultEffort + : effortLevels.includes("medium") + ? "medium" + : clampedEffort : store.reasoningEffort; // Clear any cached router-picked openrouter/free model unless the // user is staying on openrouter/free — otherwise the chip would @@ -823,6 +904,20 @@ export function ChatPage(): ReactElement { ...store.params, checkpoint: value, }); + const supportsBuiltinWebSearch = providerSupportsBuiltinWebSearch( + selectedProvider?.providerType, + ); + // See sibling useEffect above: Kimi's k2.x default to thinking + // enabled, so the Think pill comes up clicked. Search pill stays + // off by default; mutual exclusion flips them via the composer. + const isKimi = selectedProvider?.providerType === "kimi"; + // Mirror of sibling useEffect: Anthropic and OpenAI get Search + // on-by-default since their server tools emit structured + // citations end-to-end. OpenRouter and Kimi stay off-by-default. + const searchOnByDefault = + supportsBuiltinWebSearch && + (selectedProvider?.providerType === "anthropic" || + selectedProvider?.providerType === "openai"); useChatRuntimeStore.setState({ activeGgufVariant: null, ggufContextLength: null, @@ -837,10 +932,20 @@ export function ChatPage(): ReactElement { reasoningEffort: nextReasoningEffort, reasoningEnabled: reasoningCaps.supportsReasoning ? reasoningCaps.supportsReasoningOff - ? store.reasoningEnabled + ? isKimi + ? true + : store.reasoningEnabled : true : store.reasoningEnabled, supportsPreserveThinking: false, + // External models have no local tool runtime → supportsTools=false + // keeps the Code pill greyed out. supportsBuiltinWebSearch is the + // separate flag the composer reads to light up the Search pill + // when the provider offers a server-side web_search tool. + supportsTools: false, + supportsBuiltinWebSearch, + toolsEnabled: searchOnByDefault, + codeToolsEnabled: false, ...(stillOnOpenRouterFree ? {} : { lastOpenRouterChosenModel: null }), }); return; diff --git a/studio/frontend/src/features/chat/provider-capabilities.ts b/studio/frontend/src/features/chat/provider-capabilities.ts index 0688d2d673..44da42e27b 100644 --- a/studio/frontend/src/features/chat/provider-capabilities.ts +++ b/studio/frontend/src/features/chat/provider-capabilities.ts @@ -83,6 +83,44 @@ export function clampReasoningEffortToLevels( */ export const EXTERNAL_MAX_OUTPUT_TOKENS = 32768; +/** + * Whether the external provider offers a built-in web-search tool that the + * model invokes server-side. When `true`, the chat composer's Search button + * is available for that provider and the chat-adapter forwards + * `enable_tools: true, enabled_tools: ["web_search"]` on the request — the + * backend routes the call through the provider's tool schema: + * - OpenAI: `tools: [{type: "web_search"}]` on /v1/responses + * - Anthropic: `tools: [{type: "web_search_20250305", name: "web_search", + * max_uses: 5}]` on /v1/messages + * - OpenRouter: `plugins: [{id: "web"}]` on /v1/chat/completions (the + * router's universal web-search shape; works for every + * underlying model including the `openrouter/free` router). + * - Kimi: `tools: [{type: "builtin_function", function: {name: + * "$web_search"}}]` with `thinking: {type: + * "disabled"}`. Requires a client round-trip: + * the first call returns the search args; the backend + * echoes them back as a role=tool message; the second + * call streams the answer. Handled in + * _stream_kimi_web_search on the backend. + * + * Mistral is intentionally excluded: their `web_search` connector lives on + * the Agents API (`/v1/agents` + `/v1/conversations`), not chat completions, + * and returns `"WebSearchTool connector is not supported"` if injected into + * /v1/chat/completions. Wiring it would require a dedicated Agents streaming + * path. Gemini's grounded-search can be added with the same pattern when + * matching backend translation lands. + */ +export function providerSupportsBuiltinWebSearch( + providerType: string | null | undefined, +): boolean { + return ( + providerType === "openai" || + providerType === "anthropic" || + providerType === "openrouter" || + providerType === "kimi" + ); +} + /** * Per-provider minimum on the outbound max_tokens. Kimi's docs require * `max_tokens >= 16000` whenever a thinking model is in use so the @@ -239,7 +277,7 @@ const NO_REASONING_CAPS: ReasoningCaps = { const ANTHROPIC_REASONING_MODELS = [ { prefixes: ["claude-opus-4-7"], - levels: ["none", "low", "medium", "high", "xhigh"], + levels: ["none", "low", "medium", "high", "xhigh", "max"], }, { prefixes: ["claude-opus-4-6", "claude-sonnet-4-6"], diff --git a/studio/frontend/src/features/chat/shared-composer.tsx b/studio/frontend/src/features/chat/shared-composer.tsx index ceef9f501b..18f0673433 100644 --- a/studio/frontend/src/features/chat/shared-composer.tsx +++ b/studio/frontend/src/features/chat/shared-composer.tsx @@ -304,6 +304,9 @@ export function SharedComposer({ const preserveThinking = useChatRuntimeStore((s) => s.preserveThinking); const setPreserveThinking = useChatRuntimeStore((s) => s.setPreserveThinking); const supportsTools = useChatRuntimeStore((s) => s.supportsTools); + const supportsBuiltinWebSearch = useChatRuntimeStore( + (s) => s.supportsBuiltinWebSearch, + ); const toolsEnabled = useChatRuntimeStore((s) => s.toolsEnabled); const setToolsEnabled = useChatRuntimeStore((s) => s.setToolsEnabled); const codeToolsEnabled = useChatRuntimeStore((s) => s.codeToolsEnabled); @@ -345,13 +348,31 @@ export function SharedComposer({ const reasoningLockedOn = effectiveSupportsReasoning && (effectiveReasoningAlwaysOn || !effectiveSupportsReasoningOff); + // Kimi's $web_search builtin mandates thinking=disabled per the docs at + // https://platform.kimi.ai/docs/guide/use-web-search. Both pills stay + // clickable for Kimi, but turning one on flips the other off — the + // click handlers below enforce this mutual exclusion so the visible + // state always matches what the backend actually sends. + const isKimiExternal = selectedExternalProvider?.providerType === "kimi"; const effectiveReasoningEnabled = reasoningLockedOn ? true : reasoningEnabled; const effectiveReasoningVisualEnabled = effectiveReasoningEnabled && reasoningEffort !== "none"; const reasoningDisabled = !modelLoaded || !effectiveSupportsReasoning; const showReasoningControl = effectiveSupportsReasoning || effectiveReasoningAlwaysOn; - const toolsDisabled = !modelLoaded || !supportsTools; + // Two-pill gating: Search pill lights up when the runtime has either + // a local tool runtime (supportsTools, gives us our Code/python + local + // web_search) OR a server-side web_search the provider runs for us + // (supportsBuiltinWebSearch, currently just OpenAI's /v1/responses). + // Code pill is gated on `supportsTools` only — external providers + // never give us code execution, so the pill must stay disabled even + // when Search is available. + const searchDisabled = + !modelLoaded || !(supportsTools || supportsBuiltinWebSearch); + const codeDisabled = !modelLoaded || !supportsTools; + // Backwards-compatible alias for any other call site that may still + // reference `toolsDisabled` (rare; both pills used it before). + const toolsDisabled = codeDisabled; const setPendingAudioStore = useChatRuntimeStore((s) => s.setPendingAudio); const clearPendingAudioStore = useChatRuntimeStore((s) => s.clearPendingAudio); @@ -766,6 +787,11 @@ export function SharedComposer({ setReasoningEffort(level); setReasoningEnabled(true); applyQwenThinkingParams(true); + // Mutual exclusion: turning thinking on for a + // Kimi model forces the web_search builtin off. + if (isKimiExternal && toolsEnabled) { + setToolsEnabled(false); + } }} > {formatReasoningEffortLabel(level, externalSelection?.modelId)} @@ -789,6 +815,12 @@ export function SharedComposer({ const next = !reasoningEnabled; setReasoningEnabled(next); applyQwenThinkingParams(next); + // Mutual exclusion: Kimi's $web_search builtin + // requires thinking off, so turning thinking on flips + // the Search pill off (and vice versa). + if (isKimiExternal && next && toolsEnabled) { + setToolsEnabled(false); + } }} className={cn( "flex items-center gap-1.5 rounded-full px-2.5 py-1 text-xs font-medium transition-colors", @@ -845,10 +877,22 @@ export function SharedComposer({ )} <button type="button" - disabled={toolsDisabled} - onClick={() => setToolsEnabled(!toolsEnabled)} + disabled={searchDisabled} + onClick={() => { + const next = !toolsEnabled; + setToolsEnabled(next); + // Kimi's $web_search builtin requires thinking=disabled + // (https://platform.kimi.ai/docs/guide/use-web-search). + // Toggle the Think pill off when Search comes on, and + // back on when Search goes off — mutual exclusion that + // mirrors what the backend enforces. + if (isKimiExternal) { + setReasoningEnabled(!next); + applyQwenThinkingParams(!next); + } + }} className="composer-pill-btn" - data-active={toolsEnabled && !toolsDisabled ? "true" : "false"} + data-active={toolsEnabled && !searchDisabled ? "true" : "false"} aria-label={toolsEnabled ? "Disable web search" : "Enable web search"} > <GlobeIcon className="size-3.5" /> @@ -856,10 +900,10 @@ export function SharedComposer({ </button> <button type="button" - disabled={toolsDisabled} + disabled={codeDisabled} onClick={() => setCodeToolsEnabled(!codeToolsEnabled)} className="composer-pill-btn" - data-active={codeToolsEnabled && !toolsDisabled ? "true" : "false"} + data-active={codeToolsEnabled && !codeDisabled ? "true" : "false"} aria-label={codeToolsEnabled ? "Disable code execution" : "Enable code execution"} > <CodeToggleIcon className="size-3.5" /> diff --git a/studio/frontend/src/features/chat/stores/chat-runtime-store.ts b/studio/frontend/src/features/chat/stores/chat-runtime-store.ts index 92ce1441ae..4cbed5899b 100644 --- a/studio/frontend/src/features/chat/stores/chat-runtime-store.ts +++ b/studio/frontend/src/features/chat/stores/chat-runtime-store.ts @@ -229,6 +229,16 @@ type ChatRuntimeStore = { supportsPreserveThinking: boolean; preserveThinking: boolean; supportsTools: boolean; + /** + * Whether the active external provider exposes a server-side + * web_search tool (OpenAI's /v1/responses today). Distinct from + * `supportsTools` — that flag governs the local tool runtime (Code, + * python sandbox, our DuckDuckGo web_search). This one only enables + * the chat composer's Search pill for external models and leaves + * the Code pill disabled, because external providers do not give + * us code execution. Local models keep `supportsTools` only. + */ + supportsBuiltinWebSearch: boolean; toolsEnabled: boolean; codeToolsEnabled: boolean; toolStatus: string | null; @@ -320,6 +330,7 @@ export const useChatRuntimeStore = create<ChatRuntimeStore>((set) => ({ supportsPreserveThinking: false, preserveThinking: loadBool(PRESERVE_THINKING_KEY, false), supportsTools: false, + supportsBuiltinWebSearch: false, toolsEnabled: false, codeToolsEnabled: false, toolStatus: null, @@ -430,6 +441,7 @@ export const useChatRuntimeStore = create<ChatRuntimeStore>((set) => ({ reasoningEffortLevels: ["low", "medium", "high"], supportsPreserveThinking: false, supportsTools: false, + supportsBuiltinWebSearch: false, toolsEnabled: false, codeToolsEnabled: false, toolStatus: null, From e0e606a24a96d8053dbce3adbf9bf71ce2d2f70a Mon Sep 17 00:00:00 2001 From: Daniel Han <danielhanchen@gmail.com> Date: Fri, 15 May 2026 05:35:19 -0700 Subject: [PATCH 4/4] ci: make compiler-cache shim test order-independent (#5449) The shim test_compile_real_modeling_module[*] was failing on all three RMSNorm families (llama / qwen3 / gemma3) on the Core 4.57.6 matrix cell because the preceding test_compile_every_transformers_ model_type sweep already invokes unsloth_compile_transformers for every model_type, which sets modeling.__UNSLOTH_PATCHED__ = True. unsloth_zoo.compiler.unsloth_compile_transformers (zoo compiler.py :3318-3324) early-returns when that marker is already set, without re-emitting the cache file. The targeted shim test then asserts the file exists and fails with "compiler did not write" against the temp cache path. Drop the unsloth-added marker (and any leftover cache file from the sweep) before invoking the compile so the test exercises a fresh emit regardless of collection order. Marker-only fix -- transformers version-agnostic (works on 4.57.6 + 5.x); does not touch zoo internals. --- .github/workflows/consolidated-tests-ci.yml | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/.github/workflows/consolidated-tests-ci.yml b/.github/workflows/consolidated-tests-ci.yml index 2b0ce43a7c..897367de0c 100644 --- a/.github/workflows/consolidated-tests-ci.yml +++ b/.github/workflows/consolidated-tests-ci.yml @@ -1032,16 +1032,28 @@ jobs: """Spot-check on the three production-relevant families that the compile_every sweep also covers; this case verifies the emitted cache file has the model-specific RMSNorm class - attribute, not just that the file parses + imports.""" + attribute, not just that the file parses + imports. + + Note on test isolation: ``unsloth_compile_transformers`` + early-returns when ``modeling.__UNSLOTH_PATCHED__`` is set, + so once an earlier test in the same collection patches the + module the next call won't re-emit the cache file. Drop the + marker (and any stale cache file) before invoking so this + test is order-independent.""" import importlib as _il try: - _il.import_module( + modeling = _il.import_module( f"transformers.models.{model_type}.modeling_{model_type}" ) except ModuleNotFoundError: pytest.skip( f"transformers build lacks model_type={model_type}" ) + if hasattr(modeling, "__UNSLOTH_PATCHED__"): + delattr(modeling, "__UNSLOTH_PATCHED__") + combined = _CACHE / f"unsloth_compiled_module_{model_type}.py" + if combined.exists(): + combined.unlink() unsloth_compile_transformers( model_type=model_type, fast_lora_forwards=False, ) @@ -1049,7 +1061,6 @@ jobs: f"transformers.models.{model_type}.modeling_{model_type}" ) assert getattr(modeling, "__UNSLOTH_PATCHED__", False) is True - combined = _CACHE / f"unsloth_compiled_module_{model_type}.py" _verify_file(combined, must_expose=[rms_class])