From 920920592e6c5245d315573ff98e82dbce6e755c Mon Sep 17 00:00:00 2001 From: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> Date: Fri, 15 May 2026 16:29:21 +0100 Subject: [PATCH 1/2] Polish/cloud to providers (#5450) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * polish: update provider dropdown and rename cloud * fix: tighten custom provider fallback handling * fix: external provider fallback typing * studio: wire the chat Search button to OpenAI's built-in web_search tool When the active model is an OpenAI external provider and the user clicks the existing Search pill in the composer, the chat-completion request now carries the unified enable_tools shorthand: enable_tools: true enabled_tools: ["web_search"] The backend's stream_chat_completion threads enabled_tools through to _stream_openai_responses, which translates it into the Responses API tool schema: body["tools"] = [{"type": "web_search"}] per the OpenAI Responses tool spec (https://developers.openai.com/api/docs/guides/tools). OpenAI then runs the search server-side before the model replies; the search- informed answer streams back through the existing response.output_text.delta path. web_search_call lifecycle events are silently ignored for now — sources / status indicators are follow-up scope. Frontend: - provider-capabilities.ts: new providerSupportsBuiltinWebSearch() helper. Returns true only for `openai` today; Anthropic (web_search_20250305), Gemini grounded-search, and OpenRouter variants can be added later with matching backend translation. - chat-page.tsx: both model-switch paths (the onChange handler and the inferenceParams.checkpoint useEffect) set supportsTools to match the new helper, and force toolsEnabled=false on every external switch so the Search toggle is opt-in by default. - chat-adapter.ts: external branch adds enable_tools + enabled_tools=["web_search"] to the request body when the toggle is on AND the active provider supports built-in web-search. Local-model branch is unchanged — it continues to route the same shorthand through our local tool runtime. Backend: - routes/inference.py: forwards payload.enabled_tools to stream_chat_completion at the proxy site (line 1599). - external_provider.py: stream_chat_completion gains an enabled_tools parameter; _stream_openai_responses appends {"type": "web_search"} to body["tools"] when the list contains "web_search". Other tools (file_search, code_interpreter, image_generation, computer_use_preview) are easy follow-ups in the same block. Reuses the existing pydantic ChatCompletionRequest.enabled_tools field, so no schema migrations. * studio/backend: surface OpenAI server-side web_search in the chat UI When the user has the chat Search button toggled on and OpenAI's /v1/responses invokes the built-in web_search tool, _stream_openai_responses now translates the tool's lifecycle events and citation annotations into the same _toolEvent shape that local-tool calls use. The result: the chat UI shows a web_search tool-call card mid-stream, then lists the cited sources at the end of the message — identical to how local web_search renders. SSE event translation: - response.output_item.added with item.type=web_search_call -> emit _toolEvent tool_start. Carries item.action.query as args when OpenAI ships it on the added event. - response.output_item.done with item.type=web_search_call -> backfill the query if it only arrives on the done variant. The existing reasoning branch on the same event is preserved as an if/elif under a shared isinstance guard. - response.output_text.annotation.added with type=url_citation -> collect into the most-recent web_search_call.citations list. - response.output_text.delta with inline annotations[] (older API variant) -> same collection path, so both wire shapes work. - response.completed -> emit _toolEvent tool_end per call with citations formatted as Title: \nURL: <url>\nSnippet: <snippet> blocks joined by `\n---\n`. The frontend's parseSourcesFromResult already lifts this format into source content parts at end-of-stream. - response.incomplete -> close out web_search cards with whatever citations had landed, so a truncated response does not leave a perpetually "running" tool card in the UI. Both reasoning and web_search work simultaneously on the same turn — the body sends `reasoning: {effort, summary}` and `tools: [{type: "web_search"}]` independently, and the SSE handler tracks them through separate channels. Diagnostic: finally-block logger now reports per stream web_search_requested - whether the client asked for it web_search_invocations - how many calls OpenAI actually made citations - total URLs cited queries - the search queries the model issued reasoning_emitted - whether <think> content was streamed so reports of "I clicked Search and nothing happened" can be triaged from the backend log without browser devtools. * studio/backend: fix empty query + per-card '(no sources cited)' on OpenAI web_search Two display bugs on the OpenAI Responses web_search → chat-UI bridge: 1. Tool cards showed "Searching for ''" — query missing. OpenAI's response.output_item.added for web_search_call does not reliably populate action.query across API versions; the canonical place is output_item.done. The previous code emitted tool_start at added with empty args and tried to backfill at done, but the frontend's _toolEvent: tool_start is a one-shot push (no update mechanism), so the args stayed empty. Fix: defer both tool_start *and* a placeholder tool_end emission to output_item.done, where action.query is guaranteed populated. added now just initialises tracking. Frontend then renders one card per call with the right "Searching for: <query>" label. 2. Every card showed "(no sources cited)". The previous code tried to attribute url_citation annotations to individual web_search_call invocations, but OpenAI's annotations carry no link back to a specific search call — they're just URLs the model cited from the aggregated search pool. With N invocations and M annotations, the previous logic bucketed all M into the last call and stamped "(no sources cited)" on the rest. Fix: collect citations into a single shared all_url_citations list, dedup by URL. At response.completed (and response.incomplete) overwrite the *last* web_search_call's tool_end result with the aggregated Title:/URL:/Snippet: blocks. The frontend's parseSourcesFromResult already flatMaps every web_search result, so one non-empty result is enough to surface the full source-pill set at the message tail. Other tool cards get an empty result string (no '(no sources)' text). Diagnostic log unchanged in shape; total_citations now reads len(all_url_citations) directly. * studio/chat: split Code and Search pill gates so external models cannot enable Code The previous wire-up set supportsTools=true for OpenAI external models to light up the Search pill, but supportsTools also gates the Code pill, so Code became clickable for OpenAI even though external providers have no local code execution. Separate the two gates so each pill reflects what's actually available: - chat-runtime-store: new `supportsBuiltinWebSearch: boolean` flag. Distinct from supportsTools — that one still means "runtime has a local tool sandbox" (Code, python, our DuckDuckGo web_search). This one means "the active external provider exposes a server-side web_search tool we can opt into" (OpenAI's /v1/responses today). - chat-page model-switch (both code paths): for external models, supportsTools is now forced to false (no local Code path) and supportsBuiltinWebSearch follows providerSupportsBuiltinWebSearch. Local-model paths are unaffected — they only set supportsTools. - shared-composer: Search pill gates on `searchDisabled = !modelLoaded || !(supportsTools || supportsBuiltinWebSearch)`. Code pill gates on `codeDisabled = !modelLoaded || !supportsTools` — strictly the local runtime, so external models keep Code greyed out. A `toolsDisabled = codeDisabled` alias is left in place for any later-touched call site that may still reference the old name. No backend changes — chat-adapter already calls providerSupportsBuiltinWebSearch directly, independent of the store flags, so the request shape and the backend translation are unchanged. * studio/chat: default external reasoning effort to medium, not the carry-over When switching to an external model with reasoning support, the effort dropdown was inheriting whatever value the user had set on a prior model — frequently "xhigh" left over from a previous Opus/gpt-5 session. That meant every fresh OpenAI/Anthropic selection started at Extra High, burning tokens unintentionally. Both model-switch sites in chat-page (the useEffect on inferenceParams.checkpoint and the onChange callback) now pick "medium" whenever the new model's level list contains it, instead of the clamped carry-over. The clamp still fires as a fallback for the narrow case where a model doesn't expose medium (e.g. gpt-5.3-chat- latest which only has medium anyway — no change there). Users can still pick another level explicitly via the Think dropdown. * studio/chat: also light the Search pill in the welcome-screen composer There are two composers in the chat feature. shared-composer.tsx renders inside an active thread, and assistant-ui/thread.tsx has its own WebSearchToggle / CodeToolsToggle that ship the welcome-screen "Send a message…" composer (visible before the first user message). The previous fix split supportsTools and supportsBuiltinWebSearch in shared-composer but never touched the welcome-screen toggles in thread.tsx — they both still gated on supportsTools alone, so the Search pill stayed greyed on the welcome screen even for OpenAI external models that legitimately support web_search server-side. Mirror the shared-composer rule in WebSearchToggle: disabled = !modelLoaded || !(supportsTools || supportsBuiltinWebSearch) CodeToolsToggle is left as-is — its current `disabled = !(modelLoaded && supportsTools)` is correct: external models have no local code-execution sandbox, so Code stays greyed when supportsTools=false (which is what chat-page now writes for external selections). * studio/backend: wire Anthropic server-side web_search end-to-end Mirrors the OpenAI web_search integration for Anthropic's web_search_20250305 tool. When the user toggles Search on with an Anthropic model selected, the request now carries the documented tool entry: tools: [{type: "web_search_20250305", name: "web_search", max_uses: 5}] on /v1/messages, and the SSE translation surfaces tool cards + source pills in the chat UI exactly the same way as OpenAI. stream_chat_completion now forwards enabled_tools into the Anthropic branch (was only doing this for the OpenAI Responses branch). _stream_anthropic gains an enabled_tools parameter and the web_search request-body block plus three additional event handlers: - content_block_start with type=server_tool_use, name=web_search: start tracking a new call. id becomes the tool_call_id. - content_block_delta with type=input_json_delta inside a server_tool_use block: buffer the partial_json so we can read out the search query when the block closes. - content_block_start with type=web_search_tool_result: capture the per-call result list (urls + titles) that Anthropic ships inline. - content_block_stop: closes whichever block we're inside — * server_tool_use -> emit _toolEvent: tool_start with the parsed query as args. * web_search_tool_result -> emit _toolEvent: tool_end with Title:/URL: blocks the frontend's parseSourcesFromResult lifts into source pills. * thinking block -> existing </think> close. Unlike OpenAI we get per-call results directly, so no aggregated- last-call fallback is needed — each tool card carries its own citations. Diagnostic log on stream completion now reports web_search_requested / invocations / total_results / queries, matching the OpenAI shape. Frontend providerSupportsBuiltinWebSearch returns true for 'anthropic' as well, so the Search pill lights up on Claude models the same way it does on OpenAI. The existing chat-adapter external branch already sends enabled_tools=['web_search'] based on this helper — no adapter changes needed. * studio: wire OpenRouter built-in web search via :online model suffix OpenRouter exposes a universal "add web search to any model" shortcut: append `:online` to the model id and the gateway runs the search server-side, streaming citations back as annotations on text deltas. Documented at https://openrouter.ai/docs/features/web-search Hook the existing Search toggle into that path: Backend (external_provider.py, default OAI-compat branch): - When provider_type == 'openrouter' and enabled_tools contains 'web_search', rewrite body['model']: openai/gpt-4o -> openai/gpt-4o:online anthropic/claude-sonnet-4-5:free -> anthropic/claude-sonnet-4-5:online Any existing `:variant` (`:free`, `:nitro`, etc.) is replaced — OpenRouter variants are mutually exclusive. - `openrouter/free` is skipped: it's a meta-router and `:online` is not a valid suffix on it (the gateway 400s). - A one-line INFO log fires whenever the rewrite happens so the diagnostic backend log shows exactly which model id the request was promoted to. Frontend (provider-capabilities.ts): - providerSupportsBuiltinWebSearch now returns true for 'openrouter' alongside 'openai' and 'anthropic'. The Search pill lights up and the existing chat-adapter external branch already forwards enabled_tools=['web_search'] based on this helper — no adapter changes needed. No new SSE event handling: OpenRouter does not emit a separate web_search_call event the way OpenAI/Anthropic do. Citations come back as text annotations via the existing reasoning_details path the adapter already parses, so source data flows through without extra translation. A per-call tool-card UX ("Searching for: …") would require synthesizing one client-side; deferred to a follow-up if the bare-citation flow feels too minimal. * studio: wire Mistral built-in web search connector Same shape as OpenAI's web_search tool, lives on /v1/chat/completions instead of /v1/responses. When the chat Search pill is toggled on with a Mistral model selected, the backend now appends {"type": "web_search"} to body["tools"] before the request goes out. Idempotent — won't double-append if a future call site adds it first. Models in the registry allowlist that don't support the connector (codestral, devstral, ministral, mistral-tiny) will surface a 400 from upstream; the existing default-path error log captures it. Mistral's docs: https://docs.mistral.ai/capabilities/agents/connectors/websearch Frontend providerSupportsBuiltinWebSearch returns true for 'mistral' now, alongside openai / anthropic / openrouter. The Search pill lights up for Mistral models and the existing adapter branch already sends enabled_tools=['web_search'] off this helper — no adapter changes. No SSE translation yet — Mistral streams citations inline as text annotations or `references` in the final assistant content, not as a separate web_search_call event. Citations flow through to the message body as text; a per-call tool-card UX with "Searching for: …" indicators is a follow-up if needed. * studio/backend: fix OpenRouter web_search to use plugins shape + synthesize tool card Two changes against the actual OpenRouter docs at https://openrouter.ai/docs/guides/features/plugins/web-search: Request shape: The previous commit appended :online to the model id, which works on concrete model ids but rejects on meta-routers like openrouter/free — and that's exactly the model the user was testing with, so neither the request rewrite nor the diagnostic log fired. Switch to the universal plugins shape: body["plugins"] = [{"id": "web"}] Per the docs this is "exactly equivalent" to :online but works on every model id including openrouter/free and openrouter/auto. No model suffix manipulation, idempotent if added twice. Tool-card synthesis: OpenRouter doesn't emit a structured web_search_call event the way OpenAI/Anthropic do — citations come back only as `annotations` of type=url_citation on delta/message objects. To match the chat-UI tool-card UX the user expects ("Searching for: …" indicator, source pills at message tail), synthesize the events client-side in the default OAI-compat stream loop: - On stream open (after the 200 status check): yield a synthetic _toolEvent: tool_start with tool_name=web_search, fixed id "openrouter_web_search". The chat-UI then renders the running tool card before any text streams. - During the SSE loop: scan every chunk's choices[].delta and choices[].message for `annotations: [{type: "url_citation", url_citation: {url, title, content}}]` entries. Dedup by URL into a citations list. Handles both the nested-url_citation shape OpenRouter documents and the flat-on-annotation shape some upstreams ship. - On [DONE] (or stream-close without [DONE]): emit synthetic tool_end carrying the citations as Title: …\nURL: …\nSnippet: …\n---\n… blocks the existing parseSourcesFromResult lifts into source pills at message tail. Diagnostic log on completion now also reports web_search_requested + citation count alongside the existing chosen-model / event-count telemetry. * studio: drop Mistral built-in web_search — connector lives on Agents API only Mistral's web_search is exclusively on /v1/agents + /v1/conversations; sending it on /v1/chat/completions returns "WebSearchTool connector is not supported". Wiring it would require a dedicated Agents streaming path. Remove from the frontend capability map and revert the chat-completions tool injection. * studio: wire Kimi $web_search builtin via two-call round-trip Kimi's $web_search lives on /v1/chat/completions but requires a client round-trip per https://platform.kimi.ai/docs/guide/use-web-search: the first call returns tool_calls with function.arguments populated; the caller echoes those arguments back as a role=tool message; the second call streams the final answer with search results incorporated. The docs also mandate thinking=disabled while the builtin is active. Backend: new _stream_kimi_web_search helper dispatched from stream_chat_completion when provider_type=='kimi' and 'web_search' in enabled_tools. Buffers tool_calls across deltas, falls back to a plain stream if the model declines to search, and synthesizes tool_start (with parsed query) / tool_end (with any url_citation annotations) so the chat UI's web-search card behaves the same as other providers. Frontend: kimi added to providerSupportsBuiltinWebSearch so the Search pill lights up in the composer. * studio/chat: mutual exclusion of Think + Search on Kimi composer Kimi's $web_search builtin requires thinking=disabled per https://platform.kimi.ai/docs/guide/use-web-search, so the two states cannot coexist. Make the pills mutually exclusive in both composers (shared and welcome-screen): clicking Search turns Think off; clicking Think back on turns Search off. Default Think to on when a Kimi model is selected — k2.6/k2.5 ship with thinking enabled out of the box. * studio/chat: fix wrong provider var name in onChange branch selectedProvider, not provider — TS2304 in tsc -b. * studio/backend: add diagnostics to Kimi $web_search round-trip Log the actual function.arguments from the first call (so we can see the model's search query) and the second call's usage.prompt_tokens + any annotation type names that came through. prompt_tokens spiking above the input message length is direct proof the server injected search results into context. annotation_types lets us learn the shape Kimi uses for citations if/when they emit any. * studio: per-provider defaults — Anthropic xhigh + Search on, OpenAI high + Search on, Opus 4.7 gains max Anthropic: Think effort defaults to the highest level the model supports (xhigh on 4.6/4.7, high on 4.5) and Search starts on, since the web_search_20250305 tool returns structured citations end-to-end. OpenAI: Think effort defaults to 'high' (the gpt-5.x reasoning sweet spot for /v1/responses + web_search) and Search starts on. Opus 4.7: 'max' added as an effort level above 'xhigh' in both backend (_ANTHROPIC_THINKING_SPECS) and frontend (ANTHROPIC_REASONING_MODELS). Kimi diagnostics: emit tool_end immediately after tool_start so the web-search card transitions to 'complete' before the second-call answer streams, log first-call args + second-call usage/prompt_tokens + any annotation type names, request stream_options.include_usage so the second call exposes usage in SSE. * studio/backend: harden Kimi fallback path with HTTPError handler + manual aiter_lines loop Addresses PR review feedback (#5443): the no-search fallback streaming path was using `async for response.aiter_lines()` and had no `httpx.HTTPError` guard around the POST. Switch to the manual __anext__ loop pattern used elsewhere in this module (avoids the Python 3.13 + httpcore 1.0.x GeneratorExit propagation issue) and wrap the whole request in a try/except so network failures surface as a proper SSE error frame instead of a raw traceback. * feat: prompt caching frontend for openai/anthropic * studio/chat: route vLLM provider to /v1/chat/completions, not /v1/responses vLLM's /v1/responses rebuilds messages through the loaded model's chat template, which 400s on strict-alternation templates like Gemma 3 ("Conversation roles must alternate user/assistant/..."). Stop collapsing vllm -> openai in the frontend so the backend sees the real provider type and falls through to the standard chat-completions path. Register vllm as a hidden entry in PROVIDER_REGISTRY so supports_vision and provider-create validation work without surfacing it in the cloud-provider dropdown. * studio/chat: wire prompt caching for OpenAI and Anthropic external providers Backend half of the prompt_caching toggle that already exists in the chat settings panel. Scoped to OpenAI cloud (/v1/responses) and Anthropic (/v1/messages); every other provider plumbs the flag as a no-op. - Anthropic: attach cache_control={type:ephemeral} to the system block so the static prefix is reused across turns. Without the marker Anthropic caches nothing, so this is the only way to make the toggle do real work on /v1/messages. - OpenAI: opt into prompt_cache_retention="24h" — same price as the default in_memory policy per the OpenAI docs, but the cache survives ~24 hours of idle instead of ~5-10 minutes. The model picker is registry-scoped to gpt-5.x / o3 / gpt-4.5, all of which accept the parameter (gpt-5.5+ already defaults to "24h" so it's a no-op there). - Treats `enable_prompt_caching=None` as enabled to match the frontend default for both providers; pass `false` explicitly to opt out. * studio/chat: log cache token counts on OpenAI and Anthropic stream completion Surface cache usage in the existing "stream complete" info logs so prompt-caching behavior can be verified by tailing the studio backend log instead of opening the provider dashboard. - Anthropic: latch usage from message_start (input + cache_creation + cache_read counts) and message_delta (output_tokens), then include in the per-request summary. cache_read_input_tokens > 0 confirms the cache_control marker on the system block is doing its job. - OpenAI Responses: latch usage from response.completed and response.incomplete, extract usage.input_tokens_details.cached_tokens (the /v1/responses field name, not prompt_tokens_details). A non-zero value on turn N proves prompt_cache_retention="24h" let the prefix hit the cache instead of being recomputed. * studio/backend: strip temperature/top_p for Claude 4.7 family Anthropic Opus 4.7 removed temperature, top_p, and top_k as a launch breaking change ("Sampling parameters removed" in the 4.7 release notes at https://platform.claude.com/docs/en/about-claude/models/whats-new-claude-4-7). Setting any of them to a non-default value returns 400 "<param> is deprecated for this model". The existing guard only handled top_k; temperature was still being sent unconditionally and is now breaking opus-4-7 requests. Rename _ANTHROPIC_TOP_K_DEPRECATED to _ANTHROPIC_4_7_SAMPLING_REMOVED to reflect the broader scope, omit temperature from the base body on 4.7, and skip the thinking-mode temperature=1 override on 4.7 (still applied on 4.5/4.6 where it's required). Existing thinking_translation tests target 4.5/4.6 / mock the wire so they're unaffected. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * studio/chat: anchor Anthropic prompt cache on the latest message too A system-only cache_control marker is a no-op when the system prompt is empty or shorter than Anthropic's ~1024-token cache floor — caching silently does nothing (both cache_creation and cache_read return 0). Add a second cache_control breakpoint on the final block of the latest conversation message so the entire prefix (system + prior turns + new user turn) becomes eligible for caching. On turn N+1, Anthropic rehydrates everything up through turn N's marker instead of recomputing it. Up to 4 breakpoints are allowed per request; we use at most 2 (system + tail). Tail rebuild avoids mutating the caller's content list so an image-bearing turn still slots cleanly into the cached prefix. * studio/chat: gate vLLM reasoning toggle on provider config Add a "This server runs a reasoning model" checkbox on the vLLM provider config. When off (default), the chat Think pill stays hidden and no enable_thinking ever reaches vLLM. When on, the pill renders, per-turn state flows through the existing enable_thinking plumbing, and the backend proxy lifts it onto chat_template_kwargs.enable_thinking so vLLM's Jinja template honours it. * chore: clean vLLM reasoning-toggle comments * studio/chat: gate prompt_cache_retention to actual OpenAI cloud requests Addresses Codex P1 review on _stream_openai_responses. The frontend only sends enable_prompt_caching for the openai/anthropic UI provider types, so ollama/llama.cpp/"custom" requests reach this helper with the flag as None. The previous `is not False` check treated None as enabled and injected prompt_cache_retention="24h" into every request including those bound for non-OpenAI servers, which would 400 on servers that implement /v1/responses but not the retention parameter. Match the public OpenAI host (api.openai.com) on the client base_url before adding the field so it only lands on actual OpenAI cloud requests. Studio's openai picker is already registry-scoped to gpt-5.x / o3 / gpt-4.5, all of which accept the parameter. --------- Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai> Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- .../core/inference/external_provider.py | 204 ++++++++++++++-- studio/backend/core/inference/providers.py | 32 ++- studio/backend/models/inference.py | 11 + studio/backend/routes/inference.py | 1 + .../public/provider-logos/llama_cpp.svg | 1 + .../frontend/public/provider-logos/ollama.svg | 14 ++ .../frontend/public/provider-logos/vllm.svg | 1 + .../assistant-ui/model-selector.tsx | 16 ++ .../src/components/assistant-ui/thread.tsx | 4 + .../src/features/chat/api-provider-logo.tsx | 8 +- .../src/features/chat/api/chat-adapter.ts | 23 +- .../frontend/src/features/chat/chat-page.tsx | 24 +- .../features/chat/chat-providers-dialog.tsx | 225 ++++++++++++------ .../src/features/chat/chat-settings-sheet.tsx | 39 +++ .../src/features/chat/external-providers.ts | 146 +++++++++++- .../features/chat/provider-capabilities.ts | 36 ++- .../src/features/chat/shared-composer.tsx | 4 + .../frontend/src/features/chat/types/api.ts | 1 + .../src/features/settings/settings-dialog.tsx | 2 +- 19 files changed, 681 insertions(+), 111 deletions(-) create mode 100644 studio/frontend/public/provider-logos/llama_cpp.svg create mode 100644 studio/frontend/public/provider-logos/ollama.svg create mode 100644 studio/frontend/public/provider-logos/vllm.svg diff --git a/studio/backend/core/inference/external_provider.py b/studio/backend/core/inference/external_provider.py index bd14e73d16..48a5f0e879 100644 --- a/studio/backend/core/inference/external_provider.py +++ b/studio/backend/core/inference/external_provider.py @@ -24,11 +24,17 @@ import structlog # sites use printf-style positional args, which structlog accepts. logger = structlog.get_logger(__name__) -# Claude 4.7 (Opus/Sonnet/Haiku) deprecated top_k and returns 400 -# "top_k is deprecated for this model" when it is set. 3.x and 4.5/4.6 -# still accept it. Match the 4-7 line specifically so we keep the knob -# live on every other Claude generation. -_ANTHROPIC_TOP_K_DEPRECATED = re.compile(r"^claude-(?:opus|sonnet|haiku)-4-7(?:[-.]|$)") +# Claude 4.7 (Opus/Sonnet/Haiku) removed temperature, top_p, and top_k — +# the API returns 400 "<param> is deprecated for this model" if any of +# them is set to a non-default value. The "Sampling parameters removed" +# section of the 4.7 release notes is the authoritative reference: +# https://platform.claude.com/docs/en/about-claude/models/whats-new-claude-4-7 +# 3.x and 4.5/4.6 still accept all three; match the 4-7 line strictly so +# the knobs keep working on earlier families. The trailing -4-7[-.]/EOL +# anchor keeps future versions (e.g. claude-opus-5) unaffected. +_ANTHROPIC_4_7_SAMPLING_REMOVED = re.compile( + r"^claude-(?:opus|sonnet|haiku)-4-7(?:[-.]|$)" +) _OPENAI_REASONING_SUMMARY_UNSUPPORTED = re.compile(r"^o3(?:[-.]|$)") @@ -229,6 +235,7 @@ class ExternalProviderClient: enable_thinking: Optional[bool] = None, reasoning_effort: Optional[str] = None, enabled_tools: Optional[list[str]] = None, + enable_prompt_caching: Optional[bool] = None, stream: bool = True, ) -> AsyncGenerator[str, None]: """ @@ -253,6 +260,7 @@ class ExternalProviderClient: enable_thinking, reasoning_effort, enabled_tools, + enable_prompt_caching, ): yield line return @@ -272,6 +280,7 @@ class ExternalProviderClient: enable_thinking, reasoning_effort, enabled_tools, + enable_prompt_caching, ): yield line return @@ -346,6 +355,13 @@ class ExternalProviderClient: _apply_mistral_reasoning_controls( body, model, enable_thinking, reasoning_effort ) + elif self.provider_type == "vllm" and enable_thinking is not None: + # vLLM gates thinking via chat_template_kwargs.enable_thinking. + tpl_kw = body.get("chat_template_kwargs") + if not isinstance(tpl_kw, dict): + tpl_kw = {} + tpl_kw["enable_thinking"] = bool(enable_thinking) + body["chat_template_kwargs"] = tpl_kw # OpenRouter exposes a unified `reasoning` parameter on every # chat-completion request — the gateway routes it to whichever @@ -1043,6 +1059,7 @@ class ExternalProviderClient: enable_thinking: Optional[bool] = None, reasoning_effort: Optional[str] = None, enabled_tools: Optional[list[str]] = None, + enable_prompt_caching: Optional[bool] = None, ) -> AsyncGenerator[str, None]: """ Call the Anthropic Messages API and translate its SSE to OpenAI format. @@ -1112,24 +1129,79 @@ class ExternalProviderClient: else: filtered.append(msg) + # Claude 4.7 family removed temperature / top_p / top_k entirely. + # The earlier guard only handled top_k; temperature is now also + # rejected with 400 "temperature is deprecated for this model". + # Latch the match once and reuse it everywhere temperature or + # top_k would otherwise be set — including the thinking-mode + # override below, which used to force temperature=1. + sampling_removed = bool(_ANTHROPIC_4_7_SAMPLING_REMOVED.match(model)) + body: dict[str, Any] = { "model": model, "messages": filtered, "max_tokens": max_tokens or 1024, # required by Anthropic - "temperature": temperature, "stream": True, } - # top_k is deprecated on Claude 4.7 (Opus/Sonnet/Haiku) — the API - # returns 400 "top_k is deprecated for this model" when it is set. - # 3.x and 4.5/4.6 still accept it, so gate strictly on the 4.7 ids. - if ( - top_k is not None - and top_k > 0 - and not _ANTHROPIC_TOP_K_DEPRECATED.match(model) - ): + if not sampling_removed: + body["temperature"] = temperature + if top_k is not None and top_k > 0 and not sampling_removed: body["top_k"] = top_k + # Anthropic only caches a prefix when at least one cache_control + # marker is attached to it — the frontend defaults + # enable_prompt_caching to True for Anthropic, so treat `None` the + # same as True here (callers that don't set the flag still get + # caching). Pass False explicitly to opt out. + prompt_caching_enabled = enable_prompt_caching is not False + if system: - body["system"] = system + if prompt_caching_enabled: + # System block is the most stable prefix across turns, so + # it gets its own breakpoint. Skipped when system is + # empty — there's nothing to cache, and an empty marker + # is a no-op. + body["system"] = [ + { + "type": "text", + "text": system, + "cache_control": {"type": "ephemeral"}, + } + ] + else: + body["system"] = system + + if prompt_caching_enabled and filtered: + # Second breakpoint at the end of the conversation. Anthropic + # caches the longest matching prefix up to a cache_control + # marker; placing one on the latest message means turn N+1 + # rehydrates everything up through turn N from cache instead + # of recomputing it. This is what makes caching actually work + # when the system prompt is empty or shorter than Anthropic's + # ~1024-token cache floor — the conversation history carries + # the bulk of the input tokens. Anthropic allows up to 4 + # breakpoints per request; we use at most 2 (system + tail). + last_msg = filtered[-1] + content = last_msg.get("content") + if isinstance(content, str): + last_msg["content"] = [ + { + "type": "text", + "text": content, + "cache_control": {"type": "ephemeral"}, + } + ] + elif isinstance(content, list) and content: + # Don't mutate the caller's list. Rebuild the tail with + # cache_control attached to the final block so an + # upstream image-bearing turn still cleanly slots into + # the cache as part of the conversational prefix. + head = list(content[:-1]) + tail = content[-1] + if isinstance(tail, dict): + head.append({**tail, "cache_control": {"type": "ephemeral"}}) + else: + head.append(tail) + last_msg["content"] = head thinking_spec = _anthropic_thinking_spec(model) allowed_efforts = ( thinking_spec.efforts @@ -1155,13 +1227,15 @@ class ExternalProviderClient: if effort and effort != "none": # Anthropic rejects top_k whenever thinking is enabled. body.pop("top_k", None) - # Anthropic requires temperature=1 whenever thinking is enabled, - # AND forbids top_p in the same request: setting both produces + # Earlier families (4.5/4.6) require temperature=1 when + # thinking is enabled and forbid top_p in the same request: # "temperature and top_p cannot both be specified for this # model. Please use only one." - # The base body never sets top_p, but pop defensively in case - # an upstream edit ever adds it before this branch runs. - body["temperature"] = 1 + # On Claude 4.7, temperature was removed entirely — sending + # any value (including 1) returns 400 — so skip the override + # there and let the model use its default sampling. + if not sampling_removed: + body["temperature"] = 1 body.pop("top_p", None) if thinking_spec and thinking_spec.kind == "adaptive": # `display` defaults to "omitted" on Claude Opus 4.7 (per the @@ -1278,6 +1352,13 @@ class ExternalProviderClient: current_server_tool_use: Optional[dict[str, Any]] = None current_result_block: Optional[dict[str, Any]] = None web_search_calls: dict[str, dict[str, Any]] = {} + # Cache usage tracking. message_start carries the input + # accounting (incl. cache_creation_input_tokens and + # cache_read_input_tokens); message_delta carries cumulative + # output_tokens. Both are surfaced in the "stream complete" + # log so prompt caching can be verified per-request without + # opening the Anthropic dashboard. + last_usage: dict[str, Any] = {} def _content_chunk(text: str) -> str: chunk = { @@ -1352,6 +1433,16 @@ class ExternalProviderClient: key = event_type or "<unknown>" event_counts[key] = event_counts.get(key, 0) + 1 + # message_start carries the input-side usage block + # including cache_creation_input_tokens and + # cache_read_input_tokens. message_delta updates + # output_tokens (and may overwrite the input fields + # with final values). Merge both into last_usage. + if event_type == "message_start": + start_usage = (event.get("message") or {}).get("usage") + if isinstance(start_usage, dict): + last_usage.update(start_usage) + if event_type == "content_block_start": content_block = event.get("content_block") or {} block_type = content_block.get("type") @@ -1489,6 +1580,9 @@ class ExternalProviderClient: thinking_open = False elif event_type == "message_delta": + delta_usage = event.get("usage") + if isinstance(delta_usage, dict): + last_usage.update(delta_usage) stop_reason = event.get("delta", {}).get("stop_reason") if stop_reason: if thinking_open: @@ -1538,15 +1632,28 @@ class ExternalProviderClient: for sc in web_search_calls.values() if sc.get("query") ] + # cache_read_input_tokens > 0 on turn N proves the + # cache_control marker on the system block is doing + # its job — turn 1 will show cache_creation > 0 + # instead. cache_creation tokens are billed at a + # small premium; cache_read tokens are billed at a + # discount. logger.info( "Anthropic stream complete (model=%s, " "web_search_requested=%s, web_search_invocations=%s, " - "results=%s, queries=%s, events=%s)", + "results=%s, queries=%s, " + "input_tokens=%s, output_tokens=%s, " + "cache_creation_input_tokens=%s, " + "cache_read_input_tokens=%s, events=%s)", model, web_search_requested, web_search_invocations, total_results, queries, + last_usage.get("input_tokens"), + last_usage.get("output_tokens"), + last_usage.get("cache_creation_input_tokens"), + last_usage.get("cache_read_input_tokens"), event_counts, ) await response.aclose() @@ -1584,6 +1691,7 @@ class ExternalProviderClient: enable_thinking: Optional[bool], reasoning_effort: Optional[str], enabled_tools: Optional[list[str]] = None, + enable_prompt_caching: Optional[bool] = None, ) -> AsyncGenerator[str, None]: """ Call OpenAI's /v1/responses endpoint and translate its SSE stream back @@ -1685,6 +1793,27 @@ class ExternalProviderClient: if max_tokens is not None: body["max_output_tokens"] = max_tokens + # Prompt caching on /v1/responses is automatic and free, but the + # default in-memory policy only survives ~5-10 min of inactivity + # (up to ~1 hr). Opt into the 24-hour retention policy so a chat + # left idle overnight still hits the cache on the next turn. + # Pricing is identical to in_memory per OpenAI's docs. + # + # Gated on the base URL because ollama / llama.cpp / "custom" + # presets all collapse to provider_type="openai" in + # toExternalBackendProviderType, so they also land in this + # helper. Those servers expose /v1/responses-shaped routes in + # some configurations but don't implement + # prompt_cache_retention — sending the field unconditionally + # would 400 them. Match the public OpenAI host strictly so the + # field only goes to OpenAI cloud. Studio's openai model picker + # is registry-scoped to gpt-5.x / o3 / gpt-4.5, all of which + # accept this parameter (gpt-5.5+ already defaults to "24h" and + # rejects "in_memory", so it's a safe no-op there). + is_openai_cloud = "api.openai.com" in (self.base_url or "") + if is_openai_cloud and enable_prompt_caching is not False: + body["prompt_cache_retention"] = "24h" + # OpenAI server-side tools — see # https://developers.openai.com/api/docs/guides/tools # The frontend's Search button maps to the unified @@ -1731,6 +1860,12 @@ class ExternalProviderClient: done_emitted = False reasoning_open = False reasoning_emitted = False + # Latched from response.completed / response.incomplete so + # the final log can surface input_tokens_details.cached_tokens — + # the field that proves prompt_cache_retention="24h" is + # actually hitting OpenAI's cache instead of recomputing + # the prefix every turn. + last_usage: Optional[dict[str, Any]] = None # Per-call state for OpenAI's server-side web_search tool. Mapped # back into our local _toolEvent shape so the existing chat-UI # renderer surfaces web_search the same way it does for local @@ -1955,6 +2090,9 @@ class ExternalProviderClient: reasoning_emitted = True elif event_type == "response.completed": + completed_usage = (event.get("response") or {}).get("usage") + if isinstance(completed_usage, dict): + last_usage = completed_usage if reasoning_open: yield _chunk_with_text("</think>") reasoning_open = False @@ -1998,6 +2136,11 @@ class ExternalProviderClient: yield f"data: {_json.dumps(chunk)}" elif event_type == "response.incomplete": + incomplete_usage = (event.get("response") or {}).get( + "usage" + ) + if isinstance(incomplete_usage, dict): + last_usage = incomplete_usage if reasoning_open: yield _chunk_with_text("</think>") reasoning_open = False @@ -2071,16 +2214,33 @@ class ExternalProviderClient: for sc in web_search_calls.values() if sc.get("query") ] + # cached_input_tokens > 0 on turn N proves + # prompt_cache_retention="24h" is letting the previous + # turn's prefix hit the cache instead of being + # recomputed. On /v1/responses the field is nested as + # usage.input_tokens_details.cached_tokens (not + # prompt_tokens_details, which is the /v1/chat/completions + # shape). + cached_input_tokens = None + if isinstance(last_usage, dict): + details = last_usage.get("input_tokens_details") + if isinstance(details, dict): + cached_input_tokens = details.get("cached_tokens") logger.info( "OpenAI Responses stream complete (model=%s, " "web_search_requested=%s, web_search_invocations=%s, " - "citations=%s, queries=%s, reasoning_emitted=%s)", + "citations=%s, queries=%s, reasoning_emitted=%s, " + "input_tokens=%s, output_tokens=%s, " + "cached_input_tokens=%s)", model, web_search_requested, web_search_invocations, total_citations, queries, reasoning_emitted, + (last_usage or {}).get("input_tokens"), + (last_usage or {}).get("output_tokens"), + cached_input_tokens, ) await response.aclose() await lines_gen.aclose() diff --git a/studio/backend/core/inference/providers.py b/studio/backend/core/inference/providers.py index 4b6d7d6b17..143ced95f1 100644 --- a/studio/backend/core/inference/providers.py +++ b/studio/backend/core/inference/providers.py @@ -218,6 +218,28 @@ PROVIDER_REGISTRY: dict[str, dict[str, Any]] = { # are always among the top regardless of the API's order. "model_id_limit": 15, }, + "vllm": { + "display_name": "vLLM", + # User-supplied via provider_base_url; the route layer already falls + # back to the payload's base_url when the registry entry has none. + "base_url": "", + "default_models": [], + "supports_streaming": True, + "supports_vision": True, + "supports_tool_calling": True, + "auth_header": "Authorization", + "auth_prefix": "Bearer ", + # Force /v1/chat/completions in stream_chat_completion — vLLM's + # /v1/responses rebuilds messages and runs them through the loaded + # model's chat template, which 400s on strict-alternation templates + # (Gemma 3 raises "Conversation roles must alternate user/assistant + # /user/assistant/..."). The chat-completions path takes messages + # verbatim and avoids that template gauntlet. + "notes": "Self-hosted vLLM server. Always routed to /v1/chat/completions.", + # Surfaced through the frontend's CUSTOM_PROVIDER_PRESETS, not the + # /api/providers/registry dropdown — see list_available_providers. + "hidden": True, + }, "openrouter": { "display_name": "OpenRouter", "base_url": "https://openrouter.ai/api/v1", @@ -269,9 +291,17 @@ def get_base_url(provider_type: str) -> str | None: def list_available_providers() -> list[dict[str, Any]]: - """Return all registered providers (for the /registry endpoint).""" + """Return all registered providers (for the /registry endpoint). + + Hidden entries (``"hidden": True``) are filtered out — they exist in the + registry only for backend lookups (e.g. ``supports_vision`` for vLLM) and + are surfaced in the frontend via ``CUSTOM_PROVIDER_PRESETS`` instead of + the cloud-provider dropdown. + """ result = [] for provider_type, info in PROVIDER_REGISTRY.items(): + if info.get("hidden"): + continue result.append( { "provider_type": provider_type, diff --git a/studio/backend/models/inference.py b/studio/backend/models/inference.py index 013328f6c6..f2eed314ee 100644 --- a/studio/backend/models/inference.py +++ b/studio/backend/models/inference.py @@ -593,6 +593,17 @@ class ChatCompletionRequest(BaseModel): None, description = "[x-unsloth] Override base URL for the external provider.", ) + enable_prompt_caching: Optional[bool] = Field( + None, + description = ( + "[x-unsloth] Opt in to provider-side prompt caching. On Anthropic, " + "attaches cache_control={type:ephemeral} to the system block so the " + "static prefix is reused across turns. On OpenAI cloud, caching is " + "automatic for prompts >=1024 tokens and this flag is informational. " + "Ignored for every other provider (mistral, gemini, kimi, openrouter, " + "vllm, local, etc.). Treated as enabled when omitted." + ), + ) # ── Streaming response chunks ──────────────────────────────────── diff --git a/studio/backend/routes/inference.py b/studio/backend/routes/inference.py index 55d74d1cbe..b223bcf981 100644 --- a/studio/backend/routes/inference.py +++ b/studio/backend/routes/inference.py @@ -1596,6 +1596,7 @@ async def _proxy_to_external_provider( enable_thinking = payload.enable_thinking, reasoning_effort = payload.reasoning_effort, enabled_tools = payload.enabled_tools, + enable_prompt_caching = payload.enable_prompt_caching, stream = payload.stream, ) try: diff --git a/studio/frontend/public/provider-logos/llama_cpp.svg b/studio/frontend/public/provider-logos/llama_cpp.svg new file mode 100644 index 0000000000..218cc1de88 --- /dev/null +++ b/studio/frontend/public/provider-logos/llama_cpp.svg @@ -0,0 +1 @@ +<svg xmlns="http://www.w3.org/2000/svg" xml:space="preserve" viewBox="0 0 512 512"><path d="m356.4 201.3-32.8 58.3c-43.3-33.3-107.4-38.2-150.7-2.4-69.8 57.6-64.9 190.8 43.7 191.6 30.4 0 56.2-14.3 83.9-23.8l14.6 58.1c-24.6 11.4-49.6 23.1-76.6 26.7-246 33.5-231.9-321.6-9.5-340.1 46.7-3.9 87.8 8.3 127.6 31.6zm-169.9-55.9c-37.4 11.2-72.2 31.8-98.5 60.8-4.9-58.8 8.3-177.7 73.7-201 9.7-3.4 43-11.9 42.1 5.3-1 17.3-24.1 46.9-29.7 63-9.7 28.2-.7 47.6 12.6 72.2zm92.4 252.8h-36.5v-41.3h-41.3v-34h37.7l3.6-3.6v-40.1h36.5V323h38.9v34h-38.9zm133.7-41.3v41.3h-36.5v-41.3h-38.9v-34h38.9v-43.8h36.5v40.1l3.6 3.6h37.7v34h-41.3zM305.4 31.4c4.9 7.3-22.6 38.7-27 46.7-12.6 23.8-4.1 37.4 5.3 60-27.5-4.1-53-.7-80.2 2.4C209.6 88.3 239 12.2 305.4 31.4" style="fill:#ff8236"/></svg> diff --git a/studio/frontend/public/provider-logos/ollama.svg b/studio/frontend/public/provider-logos/ollama.svg new file mode 100644 index 0000000000..d3b6a42dd7 --- /dev/null +++ b/studio/frontend/public/provider-logos/ollama.svg @@ -0,0 +1,14 @@ +<svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="294 159 1405.09 1857.06"> +<g clip-path="url(#clip0_1758_1066)"> +<path d="M599.877 159.522C582.544 162.322 561.744 171.388 547.077 182.588C502.677 216.322 468.277 287.922 453.744 377.122C448.277 410.855 444.544 457.655 444.544 493.388C444.544 535.522 449.477 589.388 456.544 626.589C458.144 634.855 458.944 642.188 458.277 642.722C457.744 643.255 451.211 648.588 443.877 654.455C418.811 674.455 390.144 705.255 370.411 733.388C332.544 787.122 308.011 848.188 297.744 914.322C293.744 940.455 292.677 993.255 295.877 1019.39C302.944 1079.66 321.077 1130.59 352.144 1177.26L362.277 1192.32L359.344 1197.26C338.544 1232.19 320.811 1282.72 312.544 1331.26C306.011 1369.66 305.211 1379.92 305.211 1431.39C305.211 1483.26 305.877 1493.52 312.011 1529.39C319.344 1572.32 334.277 1617.79 350.944 1648.06C356.411 1657.92 369.744 1678.46 371.344 1679.52C371.877 1679.79 370.277 1684.72 367.744 1690.46C348.544 1732.46 332.144 1788.32 325.344 1835.39C320.544 1867.66 319.877 1878.06 319.877 1912.06C319.877 1955.39 322.277 1976.46 331.344 2010.99L332.677 2016.06H389.744H446.944L443.211 2008.99C420.144 1966.32 418.011 1887.12 437.877 1808.06C446.944 1771.52 457.211 1744.72 476.411 1707.79L487.877 1685.39V1671.66C487.877 1658.86 487.611 1657.39 483.477 1648.99C480.277 1642.59 476.011 1637.12 468.411 1629.66C455.477 1617.12 446.144 1603.92 438.677 1587.66C405.877 1516.46 399.477 1410.72 422.544 1320.59C432.144 1282.99 448.011 1249.52 464.677 1231.26C476.011 1218.72 481.877 1204.72 481.877 1190.19C481.877 1175.12 476.544 1162.72 464.544 1149.79C430.144 1112.99 408.944 1068.19 401.344 1016.06C390.544 941.788 410.144 860.855 454.677 796.722C498.277 733.788 559.477 693.388 627.877 682.589C643.211 680.055 671.877 680.455 687.877 683.388C705.344 686.455 716.277 685.522 727.477 680.188C741.344 673.655 748.277 665.522 756.411 646.855C763.611 630.188 769.211 621.122 784.277 602.322C802.411 579.788 819.877 564.455 847.877 545.922C879.877 524.988 916.277 509.788 952.544 502.455C965.744 499.788 971.877 499.388 996.544 499.388C1021.21 499.388 1027.34 499.788 1040.54 502.455C1093.74 513.255 1146.54 540.722 1188.68 579.655C1197.74 588.055 1219.48 614.988 1226.41 626.188C1229.08 630.588 1233.74 639.922 1236.68 646.855C1244.81 665.522 1251.74 673.655 1265.61 680.188C1276.41 685.388 1287.74 686.455 1304.54 683.655C1331.08 679.122 1351.48 679.522 1377.48 684.855C1466.01 702.722 1543.08 775.655 1577.21 873.388C1606.94 959.122 1598.54 1048.86 1554.28 1117.39C1546.81 1128.99 1539.34 1138.32 1528.54 1149.79C1505.21 1174.72 1505.21 1205.66 1528.41 1231.26C1566.54 1272.99 1590.41 1375.66 1583.21 1466.19C1578.41 1525.92 1563.08 1579.39 1542.01 1609.66C1538.28 1614.99 1530.54 1624.06 1524.68 1629.66C1517.08 1637.12 1512.81 1642.59 1509.61 1648.99C1505.48 1657.39 1505.21 1658.86 1505.21 1671.66V1685.39L1516.68 1707.79C1535.88 1744.72 1546.14 1771.52 1555.21 1808.06C1574.81 1886.06 1573.08 1963.66 1550.68 2007.79C1548.81 2011.52 1547.21 2014.99 1547.21 2015.39C1547.21 2015.79 1572.68 2016.06 1603.88 2016.06H1660.41L1661.88 2010.32C1662.68 2007.26 1664.01 2002.59 1664.68 1999.92C1666.14 1994.06 1669.08 1976.72 1671.48 1960.06C1673.74 1943.26 1673.74 1881.39 1671.48 1862.72C1662.94 1794.99 1648.68 1741.26 1625.34 1690.46C1622.81 1684.72 1621.21 1679.79 1621.74 1679.52C1622.41 1679.12 1626.14 1673.79 1630.14 1667.79C1659.21 1623.79 1677.08 1568.46 1686.14 1495.39C1688.54 1475.26 1688.54 1388.72 1686.14 1369.39C1679.74 1319.52 1672.01 1285.66 1659.21 1251.39C1653.88 1237.12 1639.74 1206.99 1633.74 1197.26L1630.81 1192.32L1640.94 1177.26C1672.01 1130.59 1690.14 1079.66 1697.21 1019.39C1700.41 993.255 1699.34 940.455 1695.34 914.322C1684.94 848.055 1660.54 787.255 1622.68 733.388C1602.94 705.255 1574.28 674.455 1549.21 654.455C1541.88 648.588 1535.34 643.255 1534.81 642.722C1534.14 642.188 1534.94 634.855 1536.54 626.589C1552.68 542.455 1552.14 437.522 1535.21 355.522C1520.54 284.055 1493.88 227.255 1459.48 194.455C1432.01 168.322 1404.01 157.122 1370.41 159.255C1293.34 163.788 1231.21 252.455 1206.68 392.188C1202.68 414.722 1199.21 441.122 1199.21 448.322C1199.21 451.122 1198.68 453.388 1198.01 453.388C1197.34 453.388 1192.14 450.722 1186.54 447.388C1127.08 412.188 1060.94 393.388 996.544 393.388C932.144 393.388 866.011 412.188 806.544 447.388C800.944 450.722 795.744 453.388 795.077 453.388C794.411 453.388 793.877 451.122 793.877 448.322C793.877 440.855 790.277 413.655 786.411 392.188C764.144 266.722 713.077 183.655 645.211 162.722C635.877 159.922 609.344 158.055 599.877 159.522ZM622.544 268.055C641.744 283.255 663.077 326.722 675.344 375.388C677.611 384.188 680.011 394.322 680.677 398.055C681.211 401.655 682.677 409.788 683.877 416.055C689.077 444.322 691.477 474.855 691.744 512.055L691.877 548.722L682.677 562.322L673.477 576.055H652.011C626.944 576.055 602.011 579.255 578.144 585.655C569.611 587.788 561.344 589.922 559.744 590.322C557.211 590.855 556.811 590.055 555.344 579.122C547.477 519.788 547.877 454.055 556.544 399.388C566.144 338.455 588.544 283.255 610.411 266.988C615.611 263.122 616.544 263.255 622.544 268.055ZM1382.81 267.122C1396.01 276.855 1410.54 302.722 1421.34 335.788C1443.08 401.922 1449.21 492.722 1437.74 579.122C1436.28 590.055 1435.88 590.855 1433.34 590.322C1431.74 589.922 1423.48 587.788 1414.94 585.655C1391.08 579.255 1366.14 576.055 1341.08 576.055H1319.61L1310.41 562.322L1301.21 548.722L1301.34 512.055C1301.61 460.322 1306.41 419.922 1317.88 374.988C1330.01 326.722 1351.48 283.255 1370.54 268.055C1376.54 263.255 1377.48 263.122 1382.81 267.122Z" fill="black"/> +<path d="M975.877 938.189C946.944 940.989 939.077 942.055 925.21 944.855C902.677 949.522 872.544 959.922 851.61 970.189C778.81 1005.79 728.677 1065.12 713.344 1133.79C710.277 1147.39 709.877 1151.92 709.877 1174.86C709.877 1197.52 710.277 1202.46 713.21 1215.39C733.61 1305.12 816.277 1371.39 923.21 1383.52C946.41 1386.06 1046.68 1386.06 1069.88 1383.52C1155.74 1373.79 1229.61 1327.26 1262.81 1261.92C1271.61 1244.46 1275.88 1233.12 1279.88 1215.39C1282.81 1202.46 1283.21 1197.52 1283.21 1174.86C1283.21 1151.92 1282.81 1147.39 1279.74 1133.79C1257.48 1034.06 1160.68 955.522 1042.01 940.589C1026.54 938.722 986.01 937.122 975.877 938.189ZM1025.74 1010.72C1065.34 1014.99 1105.21 1029.12 1137.21 1050.46C1154.41 1061.92 1178.68 1085.92 1189.08 1101.66C1201.88 1121.12 1209.21 1140.99 1212.54 1165.12C1214.01 1176.19 1213.21 1184.59 1209.21 1202.46C1202.94 1229.12 1183.48 1256.99 1157.21 1276.46C1144.94 1285.39 1119.48 1298.32 1103.88 1303.39C1074.28 1312.86 1054.94 1314.59 985.877 1314.06C940.81 1313.66 932.81 1313.26 919.877 1310.86C875.744 1302.59 840.81 1284.99 815.477 1258.19C794.944 1236.59 785.61 1216.86 780.544 1184.99C778.277 1170.19 782.544 1145.66 791.21 1124.99C801.744 1099.79 828.944 1068.46 855.877 1050.46C887.077 1029.66 928.144 1014.86 965.877 1010.86C980.41 1009.26 1011.21 1009.26 1025.74 1010.72Z" fill="black"/> +<path d="M945.61 1108.06C935.477 1113.52 928.41 1127.39 930.543 1137.66C932.943 1148.72 942.677 1159.92 957.877 1169.12C966.01 1174.06 966.543 1174.72 966.943 1179.66C967.21 1182.59 966.143 1190.99 964.677 1198.46C963.077 1205.79 961.877 1213.52 961.877 1215.66C962.01 1221.39 967.343 1230.72 972.943 1235.26C977.877 1239.26 978.81 1239.39 992.677 1239.79C1005.34 1240.19 1008.01 1239.92 1013.08 1237.52C1026.14 1231.12 1029.48 1219.39 1024.68 1196.86C1020.68 1178.06 1021.48 1175.12 1031.48 1169.39C1042.01 1163.26 1053.21 1152.46 1056.54 1145.12C1062.94 1131.12 1057.08 1115.26 1042.94 1107.92C1039.48 1106.19 1035.21 1105.39 1028.94 1105.39C1019.21 1105.39 1012.94 1107.66 1001.48 1114.99L994.943 1119.12L990.81 1116.59C973.877 1106.59 970.81 1105.39 960.543 1105.52C953.21 1105.52 949.21 1106.19 945.61 1108.06Z" fill="black"/> +<path d="M621.878 953.255C598.278 960.722 580.678 978.055 571.611 1002.72C567.211 1014.46 565.078 1032.99 566.945 1042.99C571.345 1066.86 590.945 1088.59 613.211 1094.59C641.211 1101.92 662.145 1097.12 680.678 1078.72C691.478 1068.19 697.345 1058.99 703.211 1044.06C707.478 1033.52 707.745 1031.66 707.745 1016.72L707.878 1000.72L702.278 989.255C693.345 971.122 677.211 957.655 658.545 952.722C648.011 950.055 631.078 950.189 621.878 953.255Z" fill="black"/> +<path d="M1334.01 952.855C1315.74 957.789 1299.48 971.389 1290.81 989.255L1285.21 1000.72L1285.34 1016.72C1285.34 1031.66 1285.61 1033.52 1289.88 1044.06C1295.74 1058.99 1301.61 1068.19 1312.41 1078.72C1330.94 1097.12 1351.88 1101.92 1379.88 1094.59C1396.01 1090.32 1412.14 1076.72 1419.88 1060.86C1426.54 1047.39 1428.14 1037.66 1426.01 1022.32C1421.08 987.255 1400.54 961.789 1370.01 952.855C1361.08 950.189 1343.74 950.189 1334.01 952.855Z" fill="black"/> +</g> +<defs> +<clipPath id="clip0_1758_1066"> +<rect width="5849.33" height="2016" fill="transparent"/> +</clipPath> +</defs> +</svg> diff --git a/studio/frontend/public/provider-logos/vllm.svg b/studio/frontend/public/provider-logos/vllm.svg new file mode 100644 index 0000000000..0c8a13de01 --- /dev/null +++ b/studio/frontend/public/provider-logos/vllm.svg @@ -0,0 +1 @@ +<svg version="1.1" viewBox="0.0 0.0 96.0 96.0" fill="none" stroke="none" stroke-linecap="square" stroke-miterlimit="10" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg"><clipPath id="g31e21232314_0_33.0"><path d="m0 0l96.0 0l0 96.0l-96.0 0l0 -96.0z" clip-rule="nonzero"/></clipPath><g clip-path="url(#g31e21232314_0_33.0)"><path fill="#d9d9d9" d="m41.04961 80.271324l1.8897629 0l0 2.3307114l-1.8897629 0z" fill-rule="evenodd"/><path fill="#d9d9d9" d="m42.221855 81.45145l1.8897629 0l0 2.3307037l-1.8897629 0z" fill-rule="evenodd"/><g filter="url(#shadowFilter-g31e21232314_0_33.1)"><use xlink:href="#g31e21232314_0_33.1" transform="matrix(1.0 0.0 0.0 1.0 0.0 2.0)"/></g><defs><filter id="shadowFilter-g31e21232314_0_33.1" filterUnits="userSpaceOnUse"><feGaussianBlur in="SourceAlpha" stdDeviation="2.0" result="blur"/><feComponentTransfer in="blur" color-interpolation-filters="sRGB"><feFuncR type="linear" slope="0" intercept="0.0"/><feFuncG type="linear" slope="0" intercept="0.0"/><feFuncB type="linear" slope="0" intercept="0.0"/><feFuncA type="linear" slope="0.0" intercept="0"/></feComponentTransfer></filter></defs><g id="g31e21232314_0_33.1"><path fill="#d9d9d9" d="m42.22417 28.470434l0 55.307083l-27.653543 -55.307083z" fill-rule="evenodd"/></g><g filter="url(#shadowFilter-g31e21232314_0_33.2)"><use xlink:href="#g31e21232314_0_33.2" transform="matrix(1.0 0.0 0.0 1.0 0.0 2.0)"/></g><defs><filter id="shadowFilter-g31e21232314_0_33.2" filterUnits="userSpaceOnUse"><feGaussianBlur in="SourceAlpha" stdDeviation="2.0" result="blur"/><feComponentTransfer in="blur" color-interpolation-filters="sRGB"><feFuncR type="linear" slope="0" intercept="0.0"/><feFuncG type="linear" slope="0" intercept="0.0"/><feFuncB type="linear" slope="0" intercept="0.0"/><feFuncA type="linear" slope="0.0" intercept="0"/></feComponentTransfer></filter></defs><g id="g31e21232314_0_33.2"><path fill="#d9d9d9" d="m42.223038 83.77752l21.729656 0l18.653545 -70.385826l-25.574802 13.461943z" fill-rule="evenodd"/></g><g filter="url(#shadowFilter-g31e21232314_0_33.3)"><use xlink:href="#g31e21232314_0_33.3" transform="matrix(1.0 0.0 0.0 1.0 0.0 2.0)"/></g><defs><filter id="shadowFilter-g31e21232314_0_33.3" filterUnits="userSpaceOnUse"><feGaussianBlur in="SourceAlpha" stdDeviation="2.0" result="blur"/><feComponentTransfer in="blur" color-interpolation-filters="sRGB"><feFuncR type="linear" slope="0" intercept="0.0"/><feFuncG type="linear" slope="0" intercept="0.0"/><feFuncB type="linear" slope="0" intercept="0.0"/><feFuncA type="linear" slope="0.0" intercept="0"/></feComponentTransfer></filter></defs><g id="g31e21232314_0_33.3"><path fill="#fdb515" d="m41.0477 27.293962l0 55.30709l-27.653542 -55.30709z" fill-rule="evenodd"/></g><g filter="url(#shadowFilter-g31e21232314_0_33.4)"><use xlink:href="#g31e21232314_0_33.4" transform="matrix(1.0 0.0 0.0 1.0 0.0 2.0)"/></g><defs><filter id="shadowFilter-g31e21232314_0_33.4" filterUnits="userSpaceOnUse"><feGaussianBlur in="SourceAlpha" stdDeviation="2.0" result="blur"/><feComponentTransfer in="blur" color-interpolation-filters="sRGB"><feFuncR type="linear" slope="0" intercept="0.0"/><feFuncG type="linear" slope="0" intercept="0.0"/><feFuncB type="linear" slope="0" intercept="0.0"/><feFuncA type="linear" slope="0.0" intercept="0"/></feComponentTransfer></filter></defs><g id="g31e21232314_0_33.4"><path fill="#30a2ff" d="m41.046566 82.60105l21.72966 0l18.653545 -70.385826l-25.574806 13.461943z" fill-rule="evenodd"/></g></g></svg> diff --git a/studio/frontend/src/components/assistant-ui/model-selector.tsx b/studio/frontend/src/components/assistant-ui/model-selector.tsx index b4f7dd08d2..dc8bffb2b7 100644 --- a/studio/frontend/src/components/assistant-ui/model-selector.tsx +++ b/studio/frontend/src/components/assistant-ui/model-selector.tsx @@ -10,10 +10,12 @@ import { } from "@/components/ui/popover"; import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs"; import { usePlatformStore } from "@/config/env"; +import { isCustomProviderType } from "@/features/chat/external-providers"; import { cn } from "@/lib/utils"; import { ArrowDown01Icon, CloudIcon, + DashboardSquare01Icon, FolderSearchIcon, Logout01Icon, Search01Icon, @@ -40,6 +42,9 @@ const PROVIDER_LOGO_EXT: Record<string, "svg" | "png" | "jpg"> = { kimi: "jpg", qwen: "png", openrouter: "svg", + vllm: "svg", + ollama: "svg", + llama_cpp: "svg", }; function providerLogoSrc(providerType: string | undefined): string | undefined { @@ -59,6 +64,17 @@ function ExternalProviderLogo({ title?: string; }) { const src = providerLogoSrc(providerType); + if (!src && isCustomProviderType(providerType)) { + return ( + <span title={title} aria-hidden={true} className="inline-flex shrink-0"> + <HugeiconsIcon + icon={DashboardSquare01Icon} + className={cn("shrink-0", className)} + /> + </span> + ); + } + if (!src) return null; return ( <img diff --git a/studio/frontend/src/components/assistant-ui/thread.tsx b/studio/frontend/src/components/assistant-ui/thread.tsx index 4988cd5a46..0579f84896 100644 --- a/studio/frontend/src/components/assistant-ui/thread.tsx +++ b/studio/frontend/src/components/assistant-ui/thread.tsx @@ -510,6 +510,10 @@ const ReasoningToggle: FC = () => { ? getExternalReasoningCapabilities( selectedExternalProvider?.providerType, effectiveExternalModelId, + { + isReasoningProvider: + selectedExternalProvider?.isReasoningModel === true, + }, ) : null; const effectiveReasoningStyle = diff --git a/studio/frontend/src/features/chat/api-provider-logo.tsx b/studio/frontend/src/features/chat/api-provider-logo.tsx index bd4f05b2ff..09794e3acf 100644 --- a/studio/frontend/src/features/chat/api-provider-logo.tsx +++ b/studio/frontend/src/features/chat/api-provider-logo.tsx @@ -4,6 +4,7 @@ import { cn } from "@/lib/utils"; import { DashboardSquare01Icon } from "@hugeicons/core-free-icons"; import { HugeiconsIcon } from "@hugeicons/react"; +import { isCustomProviderType } from "./external-providers"; /** * Registry logos live at `public/provider-logos/{provider_type}.{ext}` where `provider_type` @@ -19,6 +20,9 @@ const PROVIDER_LOGO_EXT: Record<string, "svg" | "png" | "jpg"> = { kimi: "jpg", qwen: "png", openrouter: "svg", + vllm: "svg", + ollama: "svg", + llama_cpp: "svg", }; export function apiProviderLogoSrc( @@ -42,7 +46,8 @@ interface ApiProviderLogoProps { * OpenAI's asset is black-on-transparent; it is inverted in dark mode for contrast. */ export function ApiProviderLogo({ providerType, className, title }: ApiProviderLogoProps) { - if (providerType === "custom") { + const src = apiProviderLogoSrc(providerType); + if (!src && isCustomProviderType(providerType)) { return ( <span title={title} aria-hidden className="inline-flex shrink-0"> <HugeiconsIcon icon={DashboardSquare01Icon} className={cn("shrink-0", className)} /> @@ -50,7 +55,6 @@ export function ApiProviderLogo({ providerType, className, title }: ApiProviderL ); } - const src = apiProviderLogoSrc(providerType); if (!src) return null; return ( <img diff --git a/studio/frontend/src/features/chat/api/chat-adapter.ts b/studio/frontend/src/features/chat/api/chat-adapter.ts index 30b256f8ba..29c999678d 100644 --- a/studio/frontend/src/features/chat/api/chat-adapter.ts +++ b/studio/frontend/src/features/chat/api/chat-adapter.ts @@ -28,6 +28,8 @@ import { getExternalProviderApiKey, loadExternalProviders, parseExternalModelId, + supportsProviderPromptCaching, + toExternalBackendProviderType, } from "../external-providers"; import { EXTERNAL_MAX_OUTPUT_TOKENS, @@ -742,13 +744,13 @@ export function createOpenAIStreamAdapter(): ChatModelAdapter { if (isExternalRequest && !externalProvider) { toast.error("External provider not found.", { - description: "Open API Providers and re-add this provider.", + description: "Open Connections and re-add this provider.", }); throw new Error("External provider not found."); } if (isExternalRequest && !externalApiKey) { toast.error("Missing API key for selected external provider.", { - description: "Open API Providers and set the API key again.", + description: "Open Connections and set the API key again.", }); throw new Error("Missing external provider API key."); } @@ -929,10 +931,9 @@ export function createOpenAIStreamAdapter(): ChatModelAdapter { supportsPreserveThinking, preserveThinking, } = runtime; - const externalBackendProviderType = - externalProvider?.providerType === "custom" - ? "openai" - : externalProvider?.providerType; + const externalBackendProviderType = toExternalBackendProviderType( + externalProvider?.providerType, + ); const externalCapabilities = getProviderCapabilities( externalProvider?.providerType, ); @@ -943,6 +944,10 @@ export function createOpenAIStreamAdapter(): ChatModelAdapter { ? getExternalReasoningCapabilities( externalProvider.providerType, externalSelection.modelId, + { + isReasoningProvider: + externalProvider.isReasoningModel === true, + }, ) : { supportsReasoning, @@ -1028,6 +1033,12 @@ export function createOpenAIStreamAdapter(): ChatModelAdapter { forceRefreshPublicKey, ), provider_base_url: externalProvider.baseUrl || null, + ...(supportsProviderPromptCaching(externalProvider.providerType) + ? { + enable_prompt_caching: + externalProvider.enablePromptCaching ?? true, + } + : {}), ...(externalReasoningCaps.supportsReasoning ? externalReasoningCaps.reasoningStyle === "reasoning_effort" ? externalReasoningEnabled diff --git a/studio/frontend/src/features/chat/chat-page.tsx b/studio/frontend/src/features/chat/chat-page.tsx index 3fa294d465..2ebd8f45f0 100644 --- a/studio/frontend/src/features/chat/chat-page.tsx +++ b/studio/frontend/src/features/chat/chat-page.tsx @@ -550,6 +550,7 @@ export function ChatPage(): ReactElement { const settingsOpen = useChatRuntimeStore((s) => s.settingsPanelOpen); const setSettingsOpen = useChatRuntimeStore((s) => s.setSettingsPanelOpen); const externalProviders = useExternalProvidersStore((s) => s.providers); + const setExternalProviders = useExternalProvidersStore((s) => s.setProviders); useEffect(() => { const threadId = search.thread; @@ -629,14 +630,16 @@ export function ChatPage(): ReactElement { const reasoningStyle = useChatRuntimeStore((s) => s.reasoningStyle); const reasoningEffort = useChatRuntimeStore((s) => s.reasoningEffort); const supportsReasoningOff = useChatRuntimeStore((s) => s.supportsReasoningOff); - const activeExternalProviderType = useMemo(() => { + const activeExternalProvider = useMemo(() => { const selection = parseExternalModelId(inferenceParams.checkpoint); if (!selection) return null; - const provider = externalProviders.find( - (p) => p.id === selection.providerId, + return ( + externalProviders.find( + (p) => p.id === selection.providerId, + ) ?? null ); - return provider?.providerType ?? null; }, [externalProviders, inferenceParams.checkpoint]); + const activeExternalProviderType = activeExternalProvider?.providerType ?? null; const activeProviderCapabilities = useMemo(() => { const selection = parseExternalModelId(inferenceParams.checkpoint); if (!selection) return null; @@ -671,6 +674,7 @@ export function ChatPage(): ReactElement { const reasoningCaps = getExternalReasoningCapabilities( provider?.providerType, selection.modelId, + { isReasoningProvider: provider?.isReasoningModel === true }, ); const state = useChatRuntimeStore.getState(); const preferredEffort = state.reasoningEffort; @@ -863,6 +867,10 @@ export function ChatPage(): ReactElement { const reasoningCaps = getExternalReasoningCapabilities( selectedProvider?.providerType, selectedExternal?.modelId, + { + isReasoningProvider: + selectedProvider?.isReasoningModel === true, + }, ); const preferredEffort = store.reasoningEffort; const effortLevels = reasoningCaps.reasoningEffortLevels; @@ -1426,6 +1434,14 @@ export function ChatPage(): ReactElement { onParamsChange={setInferenceParams} isExternalModel={isExternalModel} providerCapabilities={activeProviderCapabilities} + activeExternalProvider={activeExternalProvider} + onExternalProviderChange={(updatedProvider) => { + setExternalProviders( + externalProviders.map((provider) => + provider.id === updatedProvider.id ? updatedProvider : provider, + ), + ); + }} externalProviderType={activeExternalProviderType} onReloadModel={() => { const state = useChatRuntimeStore.getState(); diff --git a/studio/frontend/src/features/chat/chat-providers-dialog.tsx b/studio/frontend/src/features/chat/chat-providers-dialog.tsx index 3297e6a992..0127369eb4 100644 --- a/studio/frontend/src/features/chat/chat-providers-dialog.tsx +++ b/studio/frontend/src/features/chat/chat-providers-dialog.tsx @@ -15,7 +15,9 @@ import { Label } from "@/components/ui/label"; import { Select, SelectContent, + SelectGroup, SelectItem, + SelectSeparator, SelectTrigger, SelectValue, } from "@/components/ui/select"; @@ -23,7 +25,6 @@ import { Spinner } from "@/components/ui/spinner"; import { Textarea } from "@/components/ui/textarea"; import { ArrowLeft02Icon, - DashboardSquare01Icon, Delete02Icon, Edit03Icon, PlusSignIcon, @@ -47,9 +48,19 @@ import { } from "./api/providers-api"; import type { ExternalProviderConfig } from "./external-providers"; import { + CUSTOM_BACKEND_PROVIDER_TYPE, + CUSTOM_PROVIDER_PRESETS, + customProviderBaseUrlPlaceholder, + customProviderDisplayName, + customProviderModelIdsPlaceholder, getExternalProviderApiKey, + isCustomProviderType, + LEGACY_CUSTOM_PROVIDER_TYPE, removeExternalProviderApiKey, setExternalProviderApiKey, + supportsProviderPromptCaching, + supportsProviderReasoningToggle, + toExternalBackendProviderType, } from "./external-providers"; /** Matches navbar / thread layout easing (see index.css --ease-out-quart) */ @@ -57,12 +68,11 @@ const PROVIDER_FORM_EASE: [number, number, number, number] = [ 0.165, 0.84, 0.44, 1, ]; const PROVIDER_FORM_DURATION = 0.2; -const CUSTOM_PROVIDER_TYPE = "custom"; -const CUSTOM_BACKEND_PROVIDER_TYPE = "openai"; const CUSTOM_PROVIDER_MISSING_KEY_MESSAGE = "No API key found, please make sure API key is added and valid for this provider."; const ANTHROPIC_DATED_SNAPSHOT_SUFFIX = /-\d{8}$/; const OPENAI_DEPRECATED_MODELS = new Set(["gpt-5.3"]); +const HIDDEN_PROVIDER_TYPES = new Set(["qwen"]); const OPENROUTER_EXCLUDED_MODELS = new Set([ "google/chirp-3", "kwaivgi/kling-v3.0-pro", @@ -82,37 +92,37 @@ function resolveUiProviderTypeFromConfig( registryRows: ProviderRegistryEntry[], existingProviderType: string | undefined, ): string { - if (existingProviderType === CUSTOM_PROVIDER_TYPE) { - return CUSTOM_PROVIDER_TYPE; + if (existingProviderType && isCustomProviderType(existingProviderType)) { + return existingProviderType; } if (configProviderType !== CUSTOM_BACKEND_PROVIDER_TYPE) { return configProviderType; } + const displayName = (configDisplayName ?? "").trim().toLowerCase(); + const matchingCustomPreset = CUSTOM_PROVIDER_PRESETS.find( + (preset) => preset.displayName.toLowerCase() === displayName, + ); + if (matchingCustomPreset) { + return matchingCustomPreset.providerType; + } const openAiRegistry = registryRows.find( (entry) => entry.provider_type === CUSTOM_BACKEND_PROVIDER_TYPE, ); if (!openAiRegistry) { return configProviderType; } - const displayName = (configDisplayName ?? "").trim().toLowerCase(); const openAiDisplayName = openAiRegistry.display_name.trim().toLowerCase(); if (displayName.length > 0 && displayName !== openAiDisplayName) { - return CUSTOM_PROVIDER_TYPE; + return LEGACY_CUSTOM_PROVIDER_TYPE; } const configUrl = normalizeUrl(configBaseUrl ?? ""); const defaultUrl = normalizeUrl(openAiRegistry.base_url ?? ""); if (configUrl.length > 0 && configUrl !== defaultUrl) { - return CUSTOM_PROVIDER_TYPE; + return LEGACY_CUSTOM_PROVIDER_TYPE; } return configProviderType; } -function toBackendProviderType(uiProviderType: string): string { - return uiProviderType === CUSTOM_PROVIDER_TYPE - ? CUSTOM_BACKEND_PROVIDER_TYPE - : uiProviderType; -} - function parseManualModelIds(text: string): string[] { const seen = new Set<string>(); const out: string[] = []; @@ -175,22 +185,22 @@ export function ChatProvidersSettings({ const [manualModelIds, setManualModelIds] = useState(""); const [modelSearchQuery, setModelSearchQuery] = useState(""); const [customProviderName, setCustomProviderName] = useState("Custom"); + const [isReasoningModel, setIsReasoningModel] = useState(false); const reduceMotion = useReducedMotion(); - const isCustomProvider = providerType === CUSTOM_PROVIDER_TYPE; + const isCustomProvider = isCustomProviderType(providerType); + const showReasoningToggle = supportsProviderReasoningToggle(providerType); const registryByType = useMemo( () => new Map(registry.map((entry) => [entry.provider_type, entry])), [registry], ); - const hasCustomInRegistry = registryByType.has(CUSTOM_PROVIDER_TYPE); - const isCuratedModelList = useMemo(() => { return registryByType.get(providerType)?.model_list_mode === "curated"; }, [registryByType, providerType]); const isManualModelList = isCustomProvider || isCuratedModelList; const modelsPanelKey = isCustomProvider - ? "custom" + ? providerType || "custom" : isCuratedModelList ? "curated" : "remote"; @@ -225,7 +235,12 @@ export function ChatProvidersSettings({ useEffect(() => { if (!providerType || editingProviderId) return; const entry = registryByType.get(providerType); - if (!entry) return; + if (!entry) { + if (isCustomProviderType(providerType)) { + setCustomProviderName(customProviderDisplayName(providerType)); + } + return; + } // Seed the registry's default_models for every provider — curated and // remote alike. For remote-mode providers, loadModels() will replace // this with the union of defaults + the live /models response once the @@ -297,6 +312,12 @@ export function ChatProvidersSettings({ baseUrl: config.base_url ?? "", models: existingModels, availableModels: existing?.availableModels ?? [], + enablePromptCaching: supportsProviderPromptCaching(uiProviderType) + ? (existing?.enablePromptCaching ?? true) + : undefined, + isReasoningModel: supportsProviderReasoningToggle(uiProviderType) + ? existing?.isReasoningModel === true + : undefined, createdAt: existing?.createdAt ?? createdAt, updatedAt, }; @@ -328,7 +349,8 @@ export function ChatProvidersSettings({ setSelectedModelIds([]); setManualModelIds(""); setModelSearchQuery(""); - setCustomProviderName("Custom"); + setCustomProviderName(customProviderDisplayName(providerType)); + setIsReasoningModel(false); } function openAddProvider() { @@ -384,7 +406,7 @@ export function ChatProvidersSettings({ const trimmed = input.trim(); if (!trimmed) { if (required) { - throw new Error("Base URL is required for custom providers."); + throw new Error("Base URL is required for this connection."); } return null; } @@ -397,7 +419,7 @@ export function ChatProvidersSettings({ return; } if (isCustomProvider) { - toast.info("Custom providers use manual model IDs."); + toast.info("This connection uses manual model IDs."); return; } if (isCuratedModelList) { @@ -458,10 +480,10 @@ export function ChatProvidersSettings({ toast.error("Choose a provider first."); return; } - const backendProviderType = toBackendProviderType(providerType); + const backendProviderType = toExternalBackendProviderType(providerType); const selectedRegistryEntry = registryByType.get(backendProviderType); const displayName = isCustomProvider - ? customProviderName.trim() || "Custom" + ? customProviderName.trim() || customProviderDisplayName(providerType) : (selectedRegistryEntry?.display_name ?? providerType); if (!isCustomProvider && !apiKey.trim()) { toast.error("API key is required."); @@ -511,17 +533,21 @@ export function ChatProvidersSettings({ const updatedAt = Number.isFinite(Date.parse(created.updated_at)) ? Date.parse(created.updated_at) : Date.now(); + const uiProviderType = isCustomProvider + ? providerType + : created.provider_type; const provider: ExternalProviderConfig = { id: created.id, - providerType: isCustomProvider - ? CUSTOM_PROVIDER_TYPE - : created.provider_type, + providerType: uiProviderType, name: created.display_name, baseUrl: created.base_url ?? "", models: modelsToSave, availableModels: manualModels ? [] : pruneProviderModelIds(providerType, availableModels), + isReasoningModel: supportsProviderReasoningToggle(uiProviderType) + ? isReasoningModel + : undefined, createdAt, updatedAt, }; @@ -553,7 +579,7 @@ export function ChatProvidersSettings({ return; } const isEditingCustomProvider = - existing.providerType === CUSTOM_PROVIDER_TYPE; + isCustomProviderType(existing.providerType); if (!isEditingCustomProvider && !apiKey.trim()) { toast.error("API key is required."); return; @@ -597,7 +623,8 @@ export function ChatProvidersSettings({ ); const updated = await updateProviderConfig(editingProviderId, { displayName: isEditingCustomProvider - ? customProviderName.trim() || "Custom" + ? customProviderName.trim() || + customProviderDisplayName(existing.providerType) : existing.name, baseUrl, }); @@ -620,6 +647,11 @@ export function ChatProvidersSettings({ availableModels: manualModels ? [] : pruneProviderModelIds(existing.providerType, availableModels), + isReasoningModel: supportsProviderReasoningToggle( + existing.providerType, + ) + ? isReasoningModel + : undefined, updatedAt, } : provider, @@ -640,12 +672,19 @@ export function ChatProvidersSettings({ setEditingProviderId(provider.id); setPage("form"); setProviderType(provider.providerType); - setCustomProviderName(provider.name || "Custom"); + setCustomProviderName( + provider.name || customProviderDisplayName(provider.providerType), + ); setApiKey(getExternalProviderApiKey(provider.id)); setShowApiKey(false); setBaseUrlDraft(provider.baseUrl); setModelSearchQuery(""); - if (provider.providerType === CUSTOM_PROVIDER_TYPE) { + setIsReasoningModel( + supportsProviderReasoningToggle(provider.providerType) + ? provider.isReasoningModel === true + : false, + ); + if (isCustomProviderType(provider.providerType)) { setAvailableModels([]); setSelectedModelIds([]); setManualModelIds(provider.models.join("\n")); @@ -696,7 +735,7 @@ export function ChatProvidersSettings({ async function testProvider(provider: ExternalProviderConfig) { const savedKey = getExternalProviderApiKey(provider.id).trim(); if (!savedKey) { - if (provider.providerType === CUSTOM_PROVIDER_TYPE) { + if (isCustomProviderType(provider.providerType)) { await editProvider(provider); toast.info(CUSTOM_PROVIDER_MISSING_KEY_MESSAGE); return; @@ -707,7 +746,9 @@ export function ChatProvidersSettings({ } try { const result = await testProviderConnection({ - providerType: toBackendProviderType(provider.providerType), + providerType: + toExternalBackendProviderType(provider.providerType) ?? + provider.providerType, apiKey: savedKey, baseUrl: provider.baseUrl || null, }); @@ -715,7 +756,7 @@ export function ChatProvidersSettings({ toast.success(result.message); } else { if ( - provider.providerType === CUSTOM_PROVIDER_TYPE && + isCustomProviderType(provider.providerType) && result.message.includes("Illegal header value b'Bearer '") ) { toast.error(CUSTOM_PROVIDER_MISSING_KEY_MESSAGE); @@ -726,7 +767,7 @@ export function ChatProvidersSettings({ } catch (error) { const message = error instanceof Error ? error.message : "Unknown error"; if ( - provider.providerType === CUSTOM_PROVIDER_TYPE && + isCustomProviderType(provider.providerType) && message.includes("Illegal header value b'Bearer '") ) { toast.error(CUSTOM_PROVIDER_MISSING_KEY_MESSAGE); @@ -753,7 +794,7 @@ export function ChatProvidersSettings({ </Button> <div className="flex min-w-0 items-center gap-2 leading-none"> <span className="text-xs font-medium text-muted-foreground"> - Cloud + Connections </span> <span className="size-1 rounded-full bg-muted-foreground/35" /> <span className="truncate text-xs font-medium text-muted-foreground"> @@ -774,7 +815,7 @@ export function ChatProvidersSettings({ Provider </Label> <p className="text-xs leading-snug text-muted-foreground"> - Supported registry or Custom. + Supported registry or local OpenAI-compatible connection. </p> </div> <Select @@ -786,6 +827,9 @@ export function ChatProvidersSettings({ setSelectedModelIds([]); setManualModelIds(""); setModelSearchQuery(""); + if (isCustomProviderType(value)) { + setCustomProviderName(customProviderDisplayName(value)); + } }} > <SelectTrigger @@ -796,32 +840,46 @@ export function ChatProvidersSettings({ <SelectValue placeholder="Choose a provider" /> </SelectTrigger> <SelectContent> - {registry.map((entry) => ( - <SelectItem - key={entry.provider_type} - value={entry.provider_type} - > - <span className="flex items-center gap-2"> - <ApiProviderLogo - providerType={entry.provider_type} - className="size-4" - title={entry.display_name} - /> - {entry.display_name} - </span> - </SelectItem> - ))} - {hasCustomInRegistry ? null : ( - <SelectItem value={CUSTOM_PROVIDER_TYPE}> - <span className="flex items-center gap-2"> - <HugeiconsIcon - icon={DashboardSquare01Icon} - className="size-4" - /> - Custom - </span> - </SelectItem> - )} + <SelectGroup> + {CUSTOM_PROVIDER_PRESETS.map((preset) => ( + <SelectItem + key={preset.providerType} + value={preset.providerType} + > + <span className="flex items-center gap-2"> + <ApiProviderLogo + providerType={preset.providerType} + className="size-4" + title={preset.displayName} + /> + {preset.displayName} + </span> + </SelectItem> + ))} + </SelectGroup> + <SelectSeparator /> + <SelectGroup> + {registry + .filter( + (entry) => + !HIDDEN_PROVIDER_TYPES.has(entry.provider_type), + ) + .map((entry) => ( + <SelectItem + key={entry.provider_type} + value={entry.provider_type} + > + <span className="flex items-center gap-2"> + <ApiProviderLogo + providerType={entry.provider_type} + className="size-4" + title={entry.display_name} + /> + {entry.display_name} + </span> + </SelectItem> + ))} + </SelectGroup> </SelectContent> </Select> </div> @@ -902,11 +960,35 @@ export function ChatProvidersSettings({ type="text" value={baseUrlDraft} onChange={(event) => setBaseUrlDraft(event.target.value)} - placeholder="https://my-vllm-server.com/v1" + placeholder={customProviderBaseUrlPlaceholder(providerType)} className="h-9 text-sm" /> </div> ) : null} + + {showReasoningToggle ? ( + <div className="grid grid-cols-[minmax(150px,0.8fr)_minmax(260px,1.2fr)] items-center gap-4 px-4 py-3 max-sm:grid-cols-1"> + <Label + htmlFor="provider-is-reasoning" + className="text-sm font-medium" + > + Reasoning model + </Label> + <label + htmlFor="provider-is-reasoning" + className="flex cursor-pointer items-center gap-2 text-sm" + > + <Checkbox + id="provider-is-reasoning" + checked={isReasoningModel} + onCheckedChange={(checked) => + setIsReasoningModel(checked === true) + } + /> + This server runs a reasoning model + </label> + </div> + ) : null} </div> </section> @@ -952,7 +1034,7 @@ export function ChatProvidersSettings({ } title={ isCustomProvider - ? "Custom providers use manual model IDs" + ? "This connection uses manual model IDs" : isCuratedModelList ? "Full catalog is not fetched for this provider" : undefined @@ -986,7 +1068,7 @@ export function ChatProvidersSettings({ onChange={(event) => setManualModelIds(event.target.value) } - placeholder={"gpt-4o-mini\nQwen/Qwen3-14B"} + placeholder={customProviderModelIdsPlaceholder(providerType)} rows={5} className="min-h-[100px] resize-y font-mono text-sm" /> @@ -1197,9 +1279,9 @@ export function ChatProvidersSettings({ <div className="flex min-h-0 flex-col gap-6"> <header className="flex flex-col gap-1 pr-8"> <div className="flex min-w-0 flex-col gap-1"> - <h1 className="font-heading text-lg font-semibold">Cloud</h1> + <h1 className="font-heading text-lg font-semibold">Connections</h1> <p className="text-xs leading-relaxed text-muted-foreground"> - Manage cloud provider connections for chat through the Studio proxy. + Manage model provider connections for chat through the Studio proxy. </p> </div> </header> @@ -1237,7 +1319,8 @@ export function ChatProvidersSettings({ const detail = provider.baseUrl || registryEntry?.base_url || ""; const providerLabel = - registryEntry?.display_name ?? provider.providerType; + registryEntry?.display_name ?? + customProviderDisplayName(provider.providerType); const modelSummary = formatModelSummary(provider.models); return ( <div @@ -1346,9 +1429,9 @@ export function ChatProvidersDialog({ className="flex max-h-[90dvh] w-[96vw] flex-col gap-0 overflow-y-auto p-8 sm:max-w-none md:max-w-[44rem]" > <DialogHeader className="sr-only"> - <DialogTitle>Cloud</DialogTitle> + <DialogTitle>Connections</DialogTitle> <DialogDescription> - Manage external model providers for chat. + Manage external model connections for chat. </DialogDescription> </DialogHeader> <ChatProvidersSettings diff --git a/studio/frontend/src/features/chat/chat-settings-sheet.tsx b/studio/frontend/src/features/chat/chat-settings-sheet.tsx index 1eb6fc35d7..74e4943b3c 100644 --- a/studio/frontend/src/features/chat/chat-settings-sheet.tsx +++ b/studio/frontend/src/features/chat/chat-settings-sheet.tsx @@ -64,6 +64,10 @@ import { Fragment, type ReactNode } from "react"; import { useEffect, useMemo, useRef, useState } from "react"; import { toast } from "sonner"; import { useChatRuntimeStore } from "./stores/chat-runtime-store"; +import { + type ExternalProviderConfig, + supportsProviderPromptCaching, +} from "./external-providers"; import { applyPresetParams, BUILTIN_PRESET_NAMES, @@ -517,6 +521,8 @@ interface ChatSettingsPanelProps { * per-param visibility in the sampling section. */ providerCapabilities?: ProviderCapabilities | null; + activeExternalProvider?: ExternalProviderConfig | null; + onExternalProviderChange?: (provider: ExternalProviderConfig) => void; /** * Backend provider type for the active external model (e.g. "kimi", * "anthropic", "openai"), or `null` for local models. Drives the @@ -533,6 +539,8 @@ export function ChatSettingsPanel({ onParamsChange, isExternalModel = false, providerCapabilities = null, + activeExternalProvider = null, + onExternalProviderChange, externalProviderType = null, onReloadModel, }: ChatSettingsPanelProps) { @@ -662,6 +670,11 @@ export function ChatSettingsPanel({ Boolean(currentCheckpoint) && modelRequiresTrustRemoteCode && !(params.trustRemoteCode ?? false); + const showPromptCachingControl = + activeExternalProvider != null && + supportsProviderPromptCaching(activeExternalProvider.providerType); + const promptCachingEnabled = + activeExternalProvider?.enablePromptCaching !== false; function set<K extends keyof InferenceParams>(key: K) { return (v: InferenceParams[K]) => { @@ -1145,6 +1158,32 @@ export function ChatSettingsPanel({ </div> </CollapsibleSection> + {showPromptCachingControl && activeExternalProvider ? ( + <CollapsibleSection label="Provider" defaultOpen={true}> + <div className="flex items-center justify-between gap-3 pt-1"> + <div className="flex min-w-0 items-center gap-1.5"> + <span className="min-w-0 text-[13px] font-medium leading-[1.25] tracking-nav text-nav-fg"> + Prompt caching + </span> + <InfoHint> + Reuse compatible prompt prefixes for lower latency and cost. + </InfoHint> + </div> + <Switch + className="panel-switch shrink-0" + checked={promptCachingEnabled} + onCheckedChange={(checked) => { + onExternalProviderChange?.({ + ...activeExternalProvider, + enablePromptCaching: checked, + }); + }} + aria-label="Enable prompt caching" + /> + </div> + </CollapsibleSection> + ) : null} + <CollapsibleSection label="System Prompt" defaultOpen={true}> <button type="button" diff --git a/studio/frontend/src/features/chat/external-providers.ts b/studio/frontend/src/features/chat/external-providers.ts index 867d0461cc..5f645042bf 100644 --- a/studio/frontend/src/features/chat/external-providers.ts +++ b/studio/frontend/src/features/chat/external-providers.ts @@ -14,10 +14,147 @@ export interface ExternalProviderConfig { models: string[]; /** Cached available model ids from the provider's /models response. */ availableModels?: string[]; + /** Whether to ask supported hosted providers to use prompt caching. */ + enablePromptCaching?: boolean; + /** User-pinned: the loaded vLLM model supports `enable_thinking`. */ + isReasoningModel?: boolean; createdAt: number; updatedAt: number; } +const PROMPT_CACHING_PROVIDER_TYPES = new Set(["openai", "anthropic"]); + +export function supportsProviderPromptCaching( + providerType: string | null | undefined, +): boolean { + return providerType != null && PROMPT_CACHING_PROVIDER_TYPES.has(providerType); +} + +// Provider types that expose the connection-level "reasoning model" +// toggle. vLLM's OpenAI-compat endpoint doesn't advertise this per model. +const REASONING_TOGGLE_PROVIDER_TYPES = new Set(["vllm"]); + +export function supportsProviderReasoningToggle( + providerType: string | null | undefined, +): boolean { + return ( + providerType != null && REASONING_TOGGLE_PROVIDER_TYPES.has(providerType) + ); +} + +export const CUSTOM_BACKEND_PROVIDER_TYPE = "openai"; +export const LEGACY_CUSTOM_PROVIDER_TYPE = "custom"; + +export const CUSTOM_PROVIDER_PRESETS = [ + { + providerType: "llama_cpp", + displayName: "llama.cpp", + baseUrlPlaceholder: "http://localhost:8080/v1", + modelIdsPlaceholder: "gpt-oss-20b\nqwen3-14b", + }, + { + providerType: "vllm", + displayName: "vLLM", + baseUrlPlaceholder: "https://my-vllm-server.com/v1", + modelIdsPlaceholder: "openai/gpt-oss-20b\nQwen/Qwen3-14B", + }, + { + providerType: "ollama", + displayName: "Ollama", + baseUrlPlaceholder: "http://localhost:11434/v1", + modelIdsPlaceholder: "gpt-oss:20b\nqwen3:14b", + }, +] as const; + +const CUSTOM_PROVIDER_LABELS: Record<string, string> = { + [LEGACY_CUSTOM_PROVIDER_TYPE]: "Custom", + ...Object.fromEntries( + CUSTOM_PROVIDER_PRESETS.map((preset) => [ + preset.providerType, + preset.displayName, + ]), + ), +}; + +const CUSTOM_PROVIDER_BASE_URL_PLACEHOLDERS: Record<string, string> = { + [LEGACY_CUSTOM_PROVIDER_TYPE]: "https://my-vllm-server.com/v1", + ...Object.fromEntries( + CUSTOM_PROVIDER_PRESETS.map((preset) => [ + preset.providerType, + preset.baseUrlPlaceholder, + ]), + ), +}; + +const CUSTOM_PROVIDER_MODEL_IDS_PLACEHOLDERS: Record<string, string> = { + [LEGACY_CUSTOM_PROVIDER_TYPE]: "openai/gpt-oss-20b\nQwen/Qwen3-14B", + ...Object.fromEntries( + CUSTOM_PROVIDER_PRESETS.map((preset) => [ + preset.providerType, + preset.modelIdsPlaceholder, + ]), + ), +}; + +export function isCustomProviderType( + providerType: string | null | undefined, +): boolean { + if (!providerType) return false; + return providerType in CUSTOM_PROVIDER_LABELS; +} + +export function customProviderDisplayName( + providerType: string | null | undefined, +): string { + if (!providerType) return "Custom"; + return CUSTOM_PROVIDER_LABELS[providerType] ?? providerType; +} + +export function customProviderBaseUrlPlaceholder( + providerType: string | null | undefined, +): string { + if (!providerType) { + return CUSTOM_PROVIDER_BASE_URL_PLACEHOLDERS[LEGACY_CUSTOM_PROVIDER_TYPE]; + } + return ( + CUSTOM_PROVIDER_BASE_URL_PLACEHOLDERS[providerType] ?? + CUSTOM_PROVIDER_BASE_URL_PLACEHOLDERS[LEGACY_CUSTOM_PROVIDER_TYPE] + ); +} + +export function customProviderModelIdsPlaceholder( + providerType: string | null | undefined, +): string { + if (!providerType) { + return CUSTOM_PROVIDER_MODEL_IDS_PLACEHOLDERS[LEGACY_CUSTOM_PROVIDER_TYPE]; + } + return ( + CUSTOM_PROVIDER_MODEL_IDS_PLACEHOLDERS[providerType] ?? + CUSTOM_PROVIDER_MODEL_IDS_PLACEHOLDERS[LEGACY_CUSTOM_PROVIDER_TYPE] + ); +} + +export function toExternalBackendProviderType(providerType: string): string; +export function toExternalBackendProviderType( + providerType: null | undefined, +): undefined; +export function toExternalBackendProviderType( + providerType: string | null | undefined, +): string | undefined; +export function toExternalBackendProviderType( + providerType: string | null | undefined, +): string | undefined { + if (!providerType) return undefined; + // vLLM's /v1/responses applies the loaded model's chat template, which + // 400s on strict-alternation templates (e.g. Gemma 3). Pass the actual + // type through so the backend routes vLLM to /v1/chat/completions instead + // of the OpenAI Responses path used for gpt-5.x. + if (providerType === "vllm") return "vllm"; + return isCustomProviderType(providerType) + ? CUSTOM_BACKEND_PROVIDER_TYPE + : providerType; +} + const EXTERNAL_PROVIDERS_KEY = "unsloth_chat_external_providers"; const EXTERNAL_PROVIDER_KEYS_KEY = "unsloth_chat_external_provider_keys"; const EXTERNAL_MODEL_PREFIX = "external::"; @@ -71,9 +208,10 @@ function mapLegacyPresetToProviderType(presetId: string): string { } function normalizeProvider(raw: ExternalProviderConfig): ExternalProviderConfig { + const providerType = raw.providerType.trim(); return { ...raw, - providerType: raw.providerType.trim(), + providerType, name: raw.name.trim(), baseUrl: raw.baseUrl.trim(), models: raw.models @@ -82,6 +220,12 @@ function normalizeProvider(raw: ExternalProviderConfig): ExternalProviderConfig availableModels: (raw.availableModels ?? []) .map((model) => model.trim()) .filter((model) => model.length > 0), + enablePromptCaching: supportsProviderPromptCaching(providerType) + ? raw.enablePromptCaching !== false + : undefined, + isReasoningModel: supportsProviderReasoningToggle(providerType) + ? raw.isReasoningModel === true + : undefined, }; } diff --git a/studio/frontend/src/features/chat/provider-capabilities.ts b/studio/frontend/src/features/chat/provider-capabilities.ts index 44da42e27b..1c40a25773 100644 --- a/studio/frontend/src/features/chat/provider-capabilities.ts +++ b/studio/frontend/src/features/chat/provider-capabilities.ts @@ -221,10 +221,13 @@ const PROVIDER_CAPABILITIES: Record<string, ProviderCapabilities> = { // OpenRouter silently drops params the target model does not support, so we // surface every knob and let the gateway handle the per-model fan-out. openrouter: ALL_SUPPORTED, - // Custom providers are assumed OpenAI-compatible by the backend; users who - // point at vLLM/Ollama backends often want top_k / min_p / repetition, - // so be permissive. + // Local OpenAI-compatible connections are proxied through the OpenAI backend + // path, but vLLM/Ollama/llama.cpp users often want top_k / min_p / + // repetition controls, so be permissive. custom: ALL_SUPPORTED, + vllm: ALL_SUPPORTED, + ollama: ALL_SUPPORTED, + llama_cpp: ALL_SUPPORTED, }; const DEFAULT_EXTERNAL_CAPABILITIES = OPENAI_COMPAT_BASE; @@ -420,6 +423,25 @@ function resolveMistralReasoningCapabilities(modelId: string): ExternalReasoning return withEnableThinkingStyle(); } +export interface ExternalReasoningResolveOptions { + /** vLLM connection flagged as a reasoning model in provider config. */ + isReasoningProvider?: boolean; +} + +// vLLM has no per-model reasoning signal on OpenAI-compat — pin via user toggle. +function resolveConnectionLevelReasoning( + normalizedProvider: string, + options: ExternalReasoningResolveOptions | undefined, +): ExternalReasoningCapabilities | null { + if (normalizedProvider === "vllm" && options?.isReasoningProvider) { + return withEnableThinkingStyle({ + supportsReasoning: true, + supportsReasoningOff: true, + }); + } + return null; +} + /** * resolve external-model thinking capabilities. * provider-specific matching lives in the OpenAI/Anthropic resolvers. @@ -428,9 +450,17 @@ function resolveMistralReasoningCapabilities(modelId: string): ExternalReasoning export function getExternalReasoningCapabilities( providerType: string | null | undefined, modelId: string | null | undefined, + options?: ExternalReasoningResolveOptions, ): ExternalReasoningCapabilities { const normalizedModel = modelId?.trim().toLowerCase() ?? ""; const normalizedProvider = providerType?.trim().toLowerCase() ?? ""; + const connectionLevel = resolveConnectionLevelReasoning( + normalizedProvider, + options, + ); + if (connectionLevel) { + return connectionLevel; + } if (!normalizedModel) { return withEnableThinkingStyle(); } diff --git a/studio/frontend/src/features/chat/shared-composer.tsx b/studio/frontend/src/features/chat/shared-composer.tsx index 18f0673433..da32dee3a4 100644 --- a/studio/frontend/src/features/chat/shared-composer.tsx +++ b/studio/frontend/src/features/chat/shared-composer.tsx @@ -330,6 +330,10 @@ export function SharedComposer({ ? getExternalReasoningCapabilities( selectedExternalProvider?.providerType, effectiveExternalModelId, + { + isReasoningProvider: + selectedExternalProvider?.isReasoningModel === true, + }, ) : null; const isExternalOpenAIReasoning = diff --git a/studio/frontend/src/features/chat/types/api.ts b/studio/frontend/src/features/chat/types/api.ts index 3fa8fcd3a7..18270a7620 100644 --- a/studio/frontend/src/features/chat/types/api.ts +++ b/studio/frontend/src/features/chat/types/api.ts @@ -224,6 +224,7 @@ export interface OpenAIChatCompletionsRequest { external_model?: string; encrypted_api_key?: string; provider_base_url?: string | null; + enable_prompt_caching?: boolean | null; } export interface OpenAIChatDelta { diff --git a/studio/frontend/src/features/settings/settings-dialog.tsx b/studio/frontend/src/features/settings/settings-dialog.tsx index 4e0764c5f2..1ece504d6f 100644 --- a/studio/frontend/src/features/settings/settings-dialog.tsx +++ b/studio/frontend/src/features/settings/settings-dialog.tsx @@ -45,7 +45,7 @@ const TABS: TabDef[] = [ { id: "profile", label: "Profile", icon: UserIcon }, { id: "appearance", label: "Appearance", icon: PaintBrush02Icon }, { id: "chat", label: "Chat", icon: Message01Icon }, - { id: "connections", label: "Cloud", icon: CloudIcon, badge: "New" }, + { id: "connections", label: "Connections", icon: CloudIcon, badge: "New" }, { id: "api-keys", label: "API", icon: Globe02Icon, badge: "New" }, { id: "about", label: "Help", icon: HelpCircleIcon }, ]; From c7c3840b5f720c000ea94c5b1cbdaf8f2c5d984d Mon Sep 17 00:00:00 2001 From: Daniel Han <danielhanchen@gmail.com> Date: Fri, 15 May 2026 09:37:26 -0700 Subject: [PATCH 2/2] ci: cap each compiler-sweep iteration with SIGALRM + log progress (#5456) Core (HF=latest + TRL=latest) (transformers >=5,<6, trl >=1,<2) hangs 30+ minutes in the compiler-sweep test under the new shim layout, exceeding the 35-min job timeout and showing up as cancelled with no log of which model_type wedged. unsloth_compile_transformers does real source rewriting + torch.compile decoration and can deadlock inside a single problem model on a new transformers point release. Per-model SIGALRM cap (60s) so one infinite-loop model_type cannot wedge the whole sweep. Print sweep progress every 25 models so the log surfaces the slow model_type the next time this regresses -- crucial for finding the upstream/transformers compile bug. Timeout errors land in the same KNOWN / NEW_FAILURES bucket as any other compile exception, so the matrix still surfaces real regressions instead of silently absorbing them. --- .github/workflows/consolidated-tests-ci.yml | 75 +++++++++++++-------- 1 file changed, 47 insertions(+), 28 deletions(-) diff --git a/.github/workflows/consolidated-tests-ci.yml b/.github/workflows/consolidated-tests-ci.yml index de928e83ee..3f4e054b00 100644 --- a/.github/workflows/consolidated-tests-ci.yml +++ b/.github/workflows/consolidated-tests-ci.yml @@ -977,40 +977,59 @@ jobs: skipped -> no `modeling_<x>.py` file (expected for some umbrella packages like `auto`, `deprecated`) known -> in KNOWN_BROKEN_COMPILE; tracked for follow-up. - Any uncaught failure fails the cell.""" + Any uncaught failure fails the cell. + + Per-model SIGALRM cap so one infinite-looping model_type + cannot wedge the whole sweep + nuke the job timeout + (observed on transformers >=5,<6 -- 30+ min hang before + this guard landed).""" import importlib as _il + import signal ok = 0 skipped = [] known = [] new_failures = [] - for model_type in _all_model_types(): - modeling_path = f"transformers.models.{model_type}.modeling_{model_type}" - try: - _il.import_module(modeling_path) - except (ModuleNotFoundError, ImportError): - skipped.append((model_type, "no modeling file")) - continue - try: - unsloth_compile_transformers( - model_type=model_type, fast_lora_forwards=False, - ) - except Exception as e: - msg = f"{type(e).__name__}: {str(e)[:200]}" + models = _all_model_types() + def _on_timeout(signum, frame): + raise TimeoutError("compile exceeded per-model budget") + prev_handler = signal.signal(signal.SIGALRM, _on_timeout) + try: + for i, model_type in enumerate(models): + if i % 25 == 0: + print(f" sweep progress: {i}/{len(models)} -> {model_type}", flush=True) + modeling_path = f"transformers.models.{model_type}.modeling_{model_type}" + try: + _il.import_module(modeling_path) + except (ModuleNotFoundError, ImportError): + skipped.append((model_type, "no modeling file")) + continue + signal.alarm(60) + try: + unsloth_compile_transformers( + model_type=model_type, fast_lora_forwards=False, + ) + except Exception as e: + signal.alarm(0) + msg = f"{type(e).__name__}: {str(e)[:200]}" + if model_type in KNOWN_BROKEN_COMPILE: + known.append((model_type, msg)) + else: + new_failures.append((model_type, msg)) + continue + signal.alarm(0) if model_type in KNOWN_BROKEN_COMPILE: - known.append((model_type, msg)) - else: - new_failures.append((model_type, msg)) - continue - if model_type in KNOWN_BROKEN_COMPILE: - # Came back green unexpectedly -- that's GOOD news, - # the bug was fixed. Surface it so we can drop the - # entry from KNOWN_BROKEN_COMPILE. - print( - f" UNEXPECTED-OK {model_type}: was in " - "KNOWN_BROKEN_COMPILE, now compiles cleanly. " - "Drop the entry." - ) - ok += 1 + # Came back green unexpectedly -- that's GOOD news, + # the bug was fixed. Surface it so we can drop the + # entry from KNOWN_BROKEN_COMPILE. + print( + f" UNEXPECTED-OK {model_type}: was in " + "KNOWN_BROKEN_COMPILE, now compiles cleanly. " + "Drop the entry." + ) + ok += 1 + finally: + signal.alarm(0) + signal.signal(signal.SIGALRM, prev_handler) print(f"\nCompile sweep: ok={ok} skipped={len(skipped)} " f"known-broken={len(known)} new-failures={len(new_failures)}") for m, r in known: