Detect a DiffusionGemma GGUF (general.architecture=diffusion-gemma) and serve
it through the diffusion runner instead of llama-server: launch the OpenAI-compat
shim (unsloth_zoo.diffusion_studio.shim, or UNSLOTH_DG_SHIM) driving the on-device
visual decoder, resolving the visual-server binary from DG_VISUAL_BIN or next to
llama-server. Surface is_diffusion to the UI and auto-render the per-step denoising
canvas artifact for DiffusionGemma (no toggle). install_llama_prebuilt + setup.sh/ps1
place the visual-server binary in the install tree best-effort. Other models unchanged.
• fix: handle empty responses tool output
Normalize empty Responses `function_call_output.output` values before converting them into Chat Completions `role="tool"` messages. Empty strings, whitespace-only strings, and empty arrays now use the existing no-output sentinel, while non-empty text and content arrays are preserved.
Add regression coverage for empty tool outputs, image payloads outside `output`, content-array serialization, validator round trips, and preserving non-empty text.
---------
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: Tai An <antai12232931@outlook.com>
Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
* studio: import MCP servers from a config file
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* import config' on the add-server form
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: defensively handle MCP config imports
* fix: address MCP import review follow-ups
* fix: preserve apostrophes in Windows MCP commands
* fix: preserve apostrophe-wrapped Windows MCP args
* fix: align Windows MCP parsing with list2cmdline
* fix: preserve explicit MCP remote transport intent
* fix: trim MCP remote URLs before transport checks
---------
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* fix(studio): surface live step with null loss through the SSE progress stream
The metric histories skip non-finite steps, so during a NaN stretch the
SSE live loop and final complete event replayed the last finite
step/loss pair. Follow the live progress step when it is ahead of the
history tail and report its loss honestly (null until recovery).
Completes the NaN honesty fix for the SSE consumer flagged in review.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Apply live-step handling to inactive streams and clear the UI loss on null for PR #6206
Fresh /progress connections after a finished run took the inactive branch
which still replayed the last finite step and loss pair; apply the same
live-step correction there. On the frontend, applyProgress kept the stale
currentLoss when a payload advanced the step with a null loss; clear it so
the display shows -- until the loss recovers. Widen the runtime state type
to number | null, which the view layer already handles.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: report the real llama-server context window and add an opt-in overflow policy for OpenAI-compatible serving
A community report showed OpenCode failing tool calls every few minutes
against Studio's OpenAI-compatible API while the same GGUF was stable on
LM Studio. Root cause: Studio advertises the requested context length, but
llama-server can allocate less (memory-fit step on small GPUs, --parallel
slot split), so clients budget against a window that does not exist. Their
generations truncate mid tool call at the real wall (finish_reason=length
with cut JSON arguments) and eventually the prompt itself exceeds the real
window, returning a 400 that agentic clients treat as non-retryable.
Changes:
- After llama-server health, read default_generation_settings.n_ctx from
/props and adopt it whenever it is below Studio's computed context, with
a warning. The load response, status route, UI value, and the passthrough
max_tokens ceiling all become honest automatically.
- Expose context_length and max_context_length on /v1/models so clients can
budget against the enforced window.
- Accept empty role=tool content (commands with no output are routine in
agentic loops; OpenAI and llama-server both accept it) instead of a 400.
- Add context_overflow=truncate_middle (per request, or server-wide via
UNSLOTH_CONTEXT_OVERFLOW=truncate_middle): on exceed_context_size_error
the passthrough drops whole middle turn-groups (system prompt, first turn,
and recent turns kept; tool calls stay paired with their results), clips
oversized contents middle-out when group-dropping is not enough, clamps
max_tokens to the generation headroom, and retries. Default stays 'error'
with code=context_length_exceeded so clients running their own compaction
keep full control.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: allocate the requested context for real (kv-unified, fit-ctx floor)
Two launch-flag gaps caused the advertised vs allocated divergence at the
source:
- llama-server enables --kv-unified only when the slot count is auto; Studio
always passes --parallel N, which silently splits -c into per-slot windows
of -c/N. Pass --kv-unified when N > 1 so a single request can use the full
advertised window (same total KV memory, shared pool).
- with --fit on the fit step may set ctx as low as 4096; pass
--fit-ctx <requested> for explicit requests so fit offloads or fails into
the existing --fit off retry instead of silently shrinking the window.
Both flags are gated on --help capability probing so older builds keep the
current behavior, where the /props readback remains the backstop. Verified
live: -c 98304 --parallel 4 now serves per-slot n_ctx 98304 (was 24576),
48k-token requests pass through the passthrough, and the readback warning no
longer fires.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: surface the llama.cpp update affordance when MTP is disabled
When a model asks for MTP (auto on an MTP model, or forced mtp / mtp+ngram)
but it gets disabled, the load already degrades gracefully and serves without
speculative decoding. Until now the UI gave no hint why, or that an update
would fix it.
Record why MTP was dropped on the backend (spec_fallback_reason): the probe
found no mtp token (binary_no_mtp), the spawn aborted with an outdated-arch /
context-build error such as a prebuilt that predates the Gemma drafter
(binary_outdated), or the current build could not run it, e.g. a CUDA kernel
limit (runtime_error). Expose it in the inference status. In the chat
Speculative Decoding section, show a short note and, for the two update-fixable
reasons, an inline Update llama.cpp button that reuses the existing update flow.
A runtime_error gets the note without an update push, since a newer build may
not fix it.
Backend tests cover the reason being set / cleared. Frontend typechecks.
* Address review: tighten the update hint to genuinely outdated binaries
Reserve binary_outdated (which surfaces the Update llama.cpp affordance) for an
unknown-architecture abort, which proves the prebuilt predates the model;
classify the generic memory/context build failures as runtime_error, where an
update may not help. Frontend: only append the "Update llama.cpp to enable it"
sentence when an update is actually available, so the text never points at an
action the UI is not offering.
The export route's load_checkpoint waits for the training subprocess to
exit by calling time.sleep(0.5) in a loop (up to 30s) inside an async
function. time.sleep blocks the whole event loop, so every other request
to the server stalls for that duration. Use await asyncio.sleep(0.5),
matching the async pattern already used elsewhere in this file
(asyncio.to_thread, await asyncio.sleep).
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* Studio: enable MTP for sub-3B Gemma separate-drafter GGUFs
The sub-3B auto-drop to ngram-mod was tuned for an embedded draft head
(Qwen), whose per-token cost regresses below 3B. Gemma ships the head as a
separate root mtp-*.gguf drafter, a tiny standalone model that is cheap
enough to win below 3B: B200 Q4_K_XL bench, draft-mtp n=2 vs spec-off,
gemma-4-E2B (2B) = 1.21x (accept ~0.65) while ngram-mod is 1.00x.
Exempt a separate drafter from the sub-3B gate everywhere the threshold is
applied: the resolver (_mtp_too_small), the auto-fit VRAM reserve, the
drafter auto-download decision, and the reload-skip mirror via a
has_separate_drafter flag on _auto_mode_drops_mtp. Embedded sub-3B heads
(Qwen) still drop to ngram-mod. A drafter the binary cannot build (older
prebuilt, or a CUDA kernel limit) still aborts the spawn and the load
retries once without speculative decoding.
Adds the full Qwen3.5 + Gemma-4 (regular and QAT) auto/off/forced resolver
matrix, plus explicit sub-3B exemption tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Always compare the separate drafter in the reload-skip mirror
The sub-3B wrapper around the drafter compare could skip it when the drafter
was deleted out from under a running sub-3B server (detected None, stored set),
leaving a stale launch. The resolved-path compare is cheap and already handles
every case, so drop the _auto_mode_drops_mtp guard (and its now-unused imports)
and always compare when the mode can use a drafter and the user does not own
--spec-type. Addresses review feedback on #6191.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: offer the in-app llama.cpp update for source-build (markerless) installs
Source-build installs have no UNSLOTH_PREBUILT_INFO.json marker, so freshness
reported supported=False and the Update button never showed (notably on macOS,
where the fork shipped no prebuilt before b9585 and setup fell back to a source
build). When an install has no marker but an official prebuilt now exists for
the host, surface the update and let one click swap it in place.
- install_llama_prebuilt.py: published_repo_for_host() (the setup.sh host->repo
rule in Python) and a --resolve-prebuilt mode that reports whether a prebuilt
exists for this host without downloading.
- llama_cpp_update.py: markerless branch in get_update_status/start_update,
version-suppressed so source builds already newer than latest are not nagged;
fail-open throughout.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: run llama update detection off the event loop, expose source_build
The markerless source-build check probes the host and reads GitHub, so run
get_update_status and start_update in a worker thread to keep the API
responsive. Expose source_build in the status response so the banner can label
the source-build switch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the llama-route auth stub out of sys.modules for the rest of the suite
test_llama_route.py replaced sys.modules['auth.authentication'] with a
bare stub at collection time and never restored it, so every later test
importing create_access_token got the stub: 17 failures across
test_desktop_auth, test_middleware, test_openai_tool_passthrough and
test_rag_preview on all four Backend CI Python versions. Import the
real module when its deps are available and only stub in minimal envs,
popping the stubs after the standalone route load either way.
* Studio: address review on the source-build update path
- published_repo_for_host: route CPU-only Windows to ggml-org too (mirrors
setup.ps1; the fork ships no win-cpu bundle), macOS always the fork.
- markerless detection compares/display the upstream llama_tag, not a possible
fork wrapper release_tag, so a source build is not wrongly judged newer.
- do not offer when there is no resolvable install root (a pinned
LLAMA_SERVER_PATH outside a managed dir): an apply would not take effect.
* Ignore version probes in the update tests' subprocess capture
The status polls in these tests trigger the new source-build detection,
which shells out to llama-server --version through the same patched
subprocess.run. On slow runners that probe lands after the installer
call and clobbers the single captured argv, failing the flag
assertions (seen on the 3.10/3.11 Backend CI jobs). Skip probe calls
in all three fakes so only the installer invocation is captured.
* Skip markerless re-detection while the update job is swapping the tree
On a source-build install the frontend polls update-status every 3s
during an apply, and each poll ran _source_build_status, which execs
the very llama-server binary the job is concurrently replacing. On
Windows that exec can hold the exe long enough to fail the installer's
os.replace; everywhere it is a per-poll subprocess spawn for a status
the poller does not read (it only consumes job progress). Gate the
markerless branch on the job not running; the marked path is probe-free
and still returns the live job state.
* Studio: tighten source-build update root, repo routing, and downgrade guard
Only manage a markerless install when the active binary lives under a
resolvable llama.cpp root (marker dir, UNSLOTH_LLAMA_CPP_PATH it sits in,
or a llama.cpp ancestor); a pinned LLAMA_SERVER_PATH or a PATH/system
binary is left alone so an apply cannot install where it would not take
effect. Gate start_update on the same suppression as detection so a
direct POST cannot downgrade a source build newer than the latest
prebuilt. Route Linux hosts with AMD tooling (rocminfo/amd-smi/hipconfig/
hipinfo) to the fork in --resolve-prebuilt, matching setup.sh, so a HIP
source build is not offered an upstream CPU prebuilt.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: cover inactive env root and pinned llama.cpp checkout in update root tests
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: forward preserve_thinking + reasoning_effort on the OpenAI passthrough
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Provide _request_reasoning_kwargs on the responses passthrough test backend mock
The OpenAI passthrough body builder now asks the active backend for
capability-gated reasoning kwargs. The responses stream adapter test fakes
the backend with a bare SimpleNamespace, so give it the same method a
non-reasoning template would expose (returns None, keeping
chat_template_kwargs out of the captured body).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shorten the backend mock comment
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Adds an in-app "Update llama.cpp" banner and button to Unsloth Studio. When the installed prebuilt is behind the latest published release, a non-invasive banner appears; clicking Update downloads the latest prebuilt for this host and swaps it in place in the background, with no restart.
Detection reuses the freshness check from #5529. The update re-runs install_llama_prebuilt.py the same way setup.sh and setup.ps1 do after #5963: it forwards the published repo and the AMD gfx target derived from the install marker, and does not pass the removed --simple-policy or the arm64-only --cpu-fallback.
While the installer swaps binaries the backend enters a maintenance state (flag set under the serial load lock, active server unloaded) so a concurrent load cannot start a server from a half-swapped binary; the next load uses the new build. The banner also handles refused responses and jobs started in another tab so it never sticks on "Updating...".
Verified end to end on an NVIDIA B200: installed b9493, detected the update, applied it, and confirmed the binary at the same path advanced to b9585 in the same process. Hermetic backend tests and the frontend type-check pass.
* Studio: sync detected model capabilities into models[] after load
The chat composer gates audio upload on activeModel.hasAudioInput, but
/api/models/list omits audio fields for default and active-GGUF entries
and the single chat load path never wrote the load response's
capability flags back into the store. Audio-capable models such as the
Gemma 4 GGUFs therefore never unlocked audio input in the main chat,
while the compare composer (which does sync) worked.
Add syncModelCapabilities and call it after a successful load and after
the status fetch in refresh, so the flags also survive F5 and are not
clobbered by stale catalog data.
* Studio: merge audio upload into the Add photos & files picker
Remove the separate Upload audio row from the composer plus menu and
register an AudioAttachmentAdapter in the shared attachment pipeline,
so the standard picker and drag-drop accept wav, mp3, m4a, ogg, flac
and webm directly. Gating matches images: the picker always lists
audio and models without audio input get a toast at add() time. The
50MB limit is kept and the file shows as a normal attachment chip.
On send the adapter emits an audio content part on the attachment and
findLatestUserAudioBase64 now also scans attachment content, so the
request still carries audio_base64 exactly as before.
* Studio: extract AudioAttachmentAdapter into its own module
Move the adapter out of runtime-provider.tsx so it is importable in
isolation, export the audio send-path and capability-sync helpers for
tests, and guard attachment id generation for non-secure contexts
(crypto.randomUUID is undefined over plain HTTP on a LAN, matching the
existing guard in startCompare).
* Studio: do not claim .webm by extension in the audio adapter
A video/webm file would match the .webm extension entry and route to
the audio adapter. Real audio webm (MediaRecorder output) always
reports the audio/webm MIME, so matching webm by MIME only keeps video
files out while keeping recorded audio working.
* Studio: only send audio from the newest user message
audio_base64 switches the backend onto the audio generation path
(generate_whisper_response ignores chat messages entirely and
generate_audio_input_response bypasses the normal streaming path), so
replaying audio from an older turn hijacked text-only follow-ups:
Whisper would retranscribe the stale clip instead of erroring cleanly,
and audio VLMs lost tools and streaming. Stop the scan at the newest
user message, matching the consumed-on-send semantics of the legacy
pendingAudio path. Regenerating the audio turn itself still resends
its audio since it is the newest user message in that run.
Also guard extractAudioPartBase64 against null parts in deserialized
history content.
* Studio: forward audio input to llama-server for GGUF models (#6096)
* Studio: forward audio input to llama-server for GGUF models
* Studio: harden GGUF audio input handling (multi-format decode, size cap)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: carry GGUF audio in the message list so it works with tools
* Studio: bound decoded audio length and make the soundfile decoder optional
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Handle audio attachment edge cases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate audio file picker by loaded model capability (#6142)
* Gate audio attachments by loaded model
* Use conditional spread for audio attachment adapter
* Preserve audio fallback while filtering picker
---------
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
Co-authored-by: oobabooga <oobabooga4@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: support separate-file MTP GGUF drafters (Gemma 4)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix review findings for separate-file MTP drafters
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: pair local MTP drafters by name and include them in reload dedup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: manage --model-draft in extras and reject MTP/ copies as models
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: training survives a non-writable HF datasets cache
A shared HF datasets cache can contain subtrees owned by another user
(for example populated by an earlier root-run job). datasets then dies
with "[Errno 13] Permission denied: ..._builder.lock" while locking
the cached builder and the training run fails. load_dataset in the
training worker and trainer now goes through a wrapper that catches the
EACCES and rebuilds the dataset in a Studio-owned cache under
cache_root()/hf-datasets, logging the fallback.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope the HF_DATASETS_CACHE override to the fallback load
* Route non-streaming dataset preview loads through the cache-safe wrapper
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Adds a self-contained RAG stack to Studio: knowledge bases with chunked indexing, hybrid (dense + lexical) retrieval, and an automatic first-pass context inject into chat. Embeddings run through a local llama-server GGUF backend (default unsloth/bge-small-en-v1.5-GGUF) with a sentence-transformers fallback. The chat tool loop gains a search_knowledge_base tool, a per-turn re-search cap, and source citation, layered on top of the shared ToolLoopController.
* Studio: merge developer prompts for GGUF chat
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: fix OpenAI- and Anthropic-compatible API spec compliance
* Studio: fix API spec-compliance gaps on passthrough and streaming paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: carry context_length_exceeded through the OpenAI passthrough error path
* Studio: count tool-schema tokens in the Anthropic server-tool stream, and small stream-handling guards
* Studio: guard message_delta usage against None and normalize developer role before proxying
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: honor max_completion_tokens on the external-provider proxy path
* Studio: forward llama-server cached_tokens into OpenAI prompt_tokens_details
* Studio: sanitize messages in count_tokens to match the /v1/messages prompt
* Studio: report max_tokens for truncated tool calls and guard null usage in metadata events
* Studio: drop the request-id middleware (headers aren't declared in either spec)
* Studio: include the required request_id field in Anthropic error bodies
* Studio: honor max_completion_tokens on the audio (TTS / audio-input) paths
* Studio: add the _effective_max_tokens helper and route all max-token sites through it
* Studio: align API compatibility edge cases
* Studio: clarify multi-choice chat support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: clarify logprobs chat support
* Studio: opt the local chat UI into the streaming usage chunk so the context bar and tok/s repopulate
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: forward seed to llama-server, and fix Anthropic server-tool stop_reason, tool_result id correlation, and parallel-tool execution cap
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: align OpenAI chat completion spec edge cases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: align backend API compatibility tests
* Studio: honor tool caps and internal stream usage
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: coerce nullable stream usage counts
* Studio: preserve system prompts with developer messages
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Unify the Studio local tool-call loop (GGUF + safetensors) behind a shared ToolLoopController: ordered preface-then-tool-card rendering, duplicate-call de-looping with a forced final answer, XML-leak containment, and a parser fix that accepts closed <function=...> calls followed by trailing prose. Includes backend tests for the controller, strict parser, and GGUF route cursor reset.
* Studio: make Helper LLM startup pre-cache opt in
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: fix recipe dataset preview
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Trim and tighten code comments and docstrings across studio/ Python. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.
* Studio: stop leaking internal exceptions to API clients; harden sandbox path
Security hardening for the FastAPI backend.
Error exposure (CodeQL py/stack-trace-exposure): many route handlers returned
raw caught-exception text to clients via HTTPException detail / response bodies,
which can leak internal filesystem paths and stack detail. Add shared helpers in
utils/utils.py (safe_error_detail, log_and_http_error) that log the full
exception server-side and return a generic message, and sweep the route layer
(inference, models, export, training, datasets, chat_history, providers,
mcp_servers, settings, data_recipe/{jobs,seed,validate,mcp}) to use them.
Intentionally user-facing validation messages, the existing _friendly_error SSE
paths, and upstream-service body passthrough (llama-server / OpenAI) are kept;
absolute server paths echoed in models.py browse/read errors are redacted.
Path injection (CodeQL py/path-injection): serve_sandbox_file already does
basename + realpath containment; add a strict filename allowlist
(^[A-Za-z0-9._-]{1,255}$) before the path is built as defense-in-depth and to
give the analyzer a clear sanitizer.
No behavior change beyond error-message text; status codes preserved.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: keep curated error messages, fix remaining load leak
- inference.py /load non-native path: redact str(e) instead of leaking it
(matched the native branch which already redacted).
- llama_extra_args validation: return the curated, path-redacted message
instead of the generic fallback so users see the offending flag.
- sandbox file serving: allowlist now forbids only separators/control chars
via fullmatch, so generated images like 'loss curve.png' render again
while traversal is still blocked by basename + extension + realpath.
- Add safe_curated_detail() for domain/validation exceptions whose message
is intentionally user-facing; apply it to data_recipe job/validate,
chat conflict, provider test, and MCP probe paths (these were collapsing
to 'An internal error occurred', and 'connection' even mis-mapped to an
upstream-service message). Generic Exception paths keep safe_error_detail.
- log_and_http_error: tolerate stdlib loggers (no structlog kwargs).
- delete_openai_container: log transport errors with exc_info like list/create.
- Drop helper/HTTPException imports this change left unused.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* log_and_http_error: log original error traceback on stdlib-logger fallback
* Tidy error-helper and sandbox comments for PR #6072
* Trim redundant comments in studio error-hardening routes for PR #6072
* Re-trigger CI now that unsloth-zoo #727 is merged (Core pulls zoo main)
* Address PR #6072 review feedback
- inference.py: keep the actionable NativePathLeaseError detail (path-redacted)
instead of collapsing it to the generic message, matching the other curated
validation paths in this file.
- utils.py: log via a single formatted log.error(exc_info=error) call that works
for structlog and stdlib loggers; drop the now-unneeded try/except helper.
- models.py: use Path.name instead of os.path.basename(str(current)).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: emit usage and timings for MLX generation speed stats
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make MLX generation stats request scoped
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix: persist chat presets with fast mode
* Add schema drift guard test for chat inference settings (#5862)
Asserts ChatInferenceSettings declares every InferenceParams field the
frontend persists (all but checkpoint). With extra="forbid", a field
present in the UI but missing here 400s PUT /api/chat/settings, which is
exactly how fastMode regressed. Catches the next occurrence at CI time.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix: enable GGUF tools with vision inputs
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: GGUF vision tool routing
* Dedupe system messages on GGUF vision tool path for PR #6009
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: enable audio input for Gemma 4 GGUF models
Audio file upload was disabled for Gemma 4 vision+audio GGUFs (e.g.
gemma-4-12b-it-GGUF) even though their mmproj carries an audio encoder
(clip.has_audio_encoder, gemma4ua). Two causes:
- Audio-input detection only matched Gemma 3n's <audio_soft_token>;
Gemma 4 uses <|audio|>, so audio_vlm was never detected.
- The GGUF load/status responses hardcoded has_audio_input=False, so the
flag was dropped even when audio_vlm was detected (affected Gemma 3n
GGUFs too).
Changes:
- Recognize <|audio|> alongside <audio_soft_token> in the llama-server
token probe and the tokenizer-config pattern.
- Read clip.has_audio_encoder from the mmproj as an independent,
model-agnostic signal (read_mmproj_audio_capability).
- Emit the computed has_audio_input on the GGUF load/status responses.
- Tests for the new pattern and the mmproj reader.
* Studio: default chat model and dataset helper to Qwen3.5-4B-MTP
Switch the auto-loaded chat default and the dataset-analysis helper GGUF
from gemma-4-E2B-it to unsloth/Qwen3.5-4B-MTP-GGUF (UD-Q4_K_XL).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Addresses review feedback on #5971. _reset_password_command() already
shell-quotes the launcher path on POSIX (shlex.quote), so wrapping the result in
another pair of single quotes in the error string produced a mangled hint for
installs / home dirs containing spaces, e.g.
Run ''/tmp/Unsloth Studio/.../unsloth' studio reset-password' in your terminal
which a shell mis-parses. Drop the outer quotes and put the command at the end of
the message so it is unambiguous and copy-pasteable in every case:
Incorrect password. To reset it, run this in your terminal: <cmd>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The Studio login error rewrote the backend's PATH-based command into a relative Windows path (.\unsloth_studio\Scripts\unsloth.exe ...) that only resolves from inside the Studio home dir and fails with CommandNotFoundException elsewhere. Removes the Windows-only rewrite and the now-unused usePlatformStore import so the backend's unsloth studio reset-password command is shown as-is on all platforms.
* studio: cap training dataset uploads
* studio: clean up failed dataset uploads
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: raise upload limits to 500MB
* studio: make upload limit configurable
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: stream upload routes
* studio: split recipe upload caps
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten upload limit handling
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: import settings router directly
* studio: polish upload cap setting control
* studio: cap settings request bodies
* studio: stub settings route in desktop auth test
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: align project sidebar UX with ChatGPT
* feat: align project sidebar UX with ChatGPT
* feat(chat): load stored project list
* feat(chat): add project sidebar workflows
* fix: stabilize project page navigation
* fix: projects chat loading
* fix: show project chat thread
* style: sidebar project spacing and hover clipping
* style: add expandable project chat history and move-to-project submenu
* feat: polish project sidebar
* feat: persist project sandbox paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: only create sandbox project workspace dir
* feat: add optional project workspace deletion from delete dialog
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: stabilize chat projects CI failures
* fix: polish project chat navigation
* Studio: manage chat history with projects
Group chats into projects with a dedicated projects page and route.
Sidebar shows recents with per-row actions and a vertical more-vertical
menu, and the sidebar scrollbar stays hidden so rows never shift on
hover. Includes chat settings and composer refinements.
* Studio: projects sidebar and breadcrumb polish
Sidebar:
- Remove the Compare nav item.
- Widen the sidebar to match the projects layout.
- Replace the scroll-gated bottom fade with a static fade pinned above
the profile box, so it no longer attaches to Recents or lags the
collapse and expand animation.
Topbar breadcrumb (chat-page):
- On a project landing show "Projects" linking to the projects list.
- Inside a project chat show the project name and chat title, with the
project name linking back to that specific project page.
- Drop the divider between the model selector and the breadcrumb.
* Studio: make project workspace delete test cross-platform
test_chat_project_delete_files_removes_workspace rooted the project under
pytest tmp_path, which resolves to /private/tmp on macOS. The workspace
delete guard refuses paths under the system denylist by design, so the
test passed on Linux CI but failed on macOS.
Add a workspace_projects_home fixture that keeps tmp_path on Linux and
Windows (CI unchanged) and falls back to a home subdir only when the temp
root is on the platform denylist. Derive the workspace path from the
created project so it tracks the projects home.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: satisfy import-hoist check for new path re-exports
documents_root and project_workspaces_root are re-exported from
utils.paths but only referenced as __all__ string literals, which the
import-hoist safety net does not count as a use. It flagged the two newly
added re-exports as unused imports and failed Source lint.
Name-load both via a module-level _REEXPORTED tuple so the check sees
them used. No behaviour change; consumers still import them from
utils.paths.
* fix: avoid projects empty-state flash
* fix: batch chat search indexing
* Studio: polish chat sidebar, run settings, and search
- Use the native OS scrollbar for the chat sidebar, Run settings panel, and chat search list instead of a custom scrollbar
- Highlight the active run in the sidebar and keep chat search available during training
- Stop the training log view from replaying when navigating back to a run
- Rename the chat settings panel to Run settings and align its toggle icon and position
- Tighten heading and sidebar letter spacing and lighten the Train and Recents labels
- Match the search dialog corner style across light and dark and drop the stray border
- Make the MCP Servers section header plain text instead of a link
- Remove a stray .orig backup file
* studio/frontend: restore Compare entry point in the sidebar
The chat-projects sidebar redesign dropped the Compare nav item and moved
it to thread-sidebar.tsx, which is not imported or rendered anywhere. That
left no way for a user to start a new model comparison (enterCompare only
fired from the guided tour and the training handoff), and broke the
Compare/Recipes/Export UI smoke test that clicks [data-tour="chat-compare"].
Re-add the Compare NavItem to the New Chat / Search group, carrying
data-tour="chat-compare" and the same new-comparison navigation as before.
* studio/frontend: use Unsloth green for the fallback profile avatar
Switch the initials-avatar background from blue to #14b789 so the sidebar
and edit-profile avatar match the Unsloth brand colour.
* studio/frontend: turn project breadcrumb into a project switcher dropdown
* studio/frontend: stop project card kebab clicks from opening the project
* studio/frontend: hide project switcher outside projects
* studio/frontend: stabilize project switcher loading
* style: project switcher alignment
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: shimmyshimmer <107991372+shimmyshimmer@users.noreply.github.com>
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Improve the rejection message when an MCP server address is not an http(s) URL. It now points to the expected http(s):// form with an example, and only mentions that local commands are disabled when the value contains whitespace (a reliable command signal), since a lone token may just be a scheme-less URL. Wording is host-scoped rather than desktop-only because self-hosted hosts can opt in via an env var. Backend only, with tests; accepted input is unchanged.
* Studio: harden stdio MCP gating and fix transport edge cases
- Gate the Data Recipe stdio path behind UNSLOTH_STUDIO_ALLOW_STDIO_MCP so a hosted deployment cannot spawn local processes through recipes
- Enforce the gate inside _client() so the transport sink cannot spawn when disabled
- keep_alive=False so stdio probes/calls do not leave orphan subprocesses
- Force OAuth off for stdio servers on create and update
- Drop stored headers when a server switches transport type
- Reject a command whose first token is a URL scheme
- Add MCP gate and improvement tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests: skip Data Recipe stdio tests when data_designer is absent
The data_designer plugin is only installed in the Studio test job, so guard
the two build_mcp_providers tests with importorskip so the core matrix skips
them instead of failing on ModuleNotFoundError.
---------
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* added remote MCP server support
* trim
* added tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* increased timeout
* disabling MCP chat toggle
* Fix MCP OpenAI function-name validation + cancel propagation for PR #5750
OpenAI requires function.name to match ^[a-zA-Z0-9_-]{1,64}$ before
streaming starts. The existing 64-char length check is necessary but
not sufficient: MCP servers can return tool names containing '.', '/',
spaces, etc. that would 400 the whole chat request. Validate the
composed mcp__<server_id>__<tool> name against the regex, skip + warn
on miss, and drop duplicate tool names from the same server (which
would also 400 the request as "duplicates").
Also propagate the agentic-loop cancel_event into MCP tool execution
so a /cancel POST during a long-running MCP call (e.g. GitHub MCP
search across a large repo) actually interrupts the in-flight HTTP
call instead of waiting out the 300 s timeout. The watcher polls the
threading.Event at 50 ms cadence inside the asyncio loop (matches
routes/inference.py's existing cancel-watcher cadence) and races
against the call task with asyncio.wait FIRST_COMPLETED.
Tests added:
- test_mcp_specs_skip_invalid_openai_function_names: drops bad chars
- test_mcp_specs_skip_empty_tool_name
- test_mcp_specs_drops_duplicate_names
- test_call_tool_sync_respects_pre_set_cancel_event
Also fix test_desktop_auth.py's router stub that listed every existing
router but missed mcp_servers_router, so importing main.py fails after
this PR adds it to routes/__init__.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* PR #5750 round 2: OAuth cleanup on delete/url-change + mcp_enabled standalone
Round 2 of cross-platform validation surfaced two more P1 findings:
1. OAuth tokens never get cleared. fastmcp keys tokens by MCP URL, not by
server row, and delete / URL change / use_oauth toggle only updated
the SQLite row. Re-registering the same URL would silently reuse the
old account's credentials. Adds clear_oauth_tokens_async() in
mcp_client.py and calls it from the delete + put route handlers when
the row had use_oauth=True and either the URL changes or OAuth is
turned off.
2. mcp_enabled=true was ignored unless the caller also sent
enable_tools=true. The frontend always sends both together so the UI
path was fine, but a direct API caller sending only mcp_enabled would
silently get no MCP tools, which contradicts the field's documented
"append tools from every enabled MCP server" behavior. Loosens the
use_tools gate in both the GGUF and safetensors paths so mcp_enabled
opens the tool loop on its own; when the caller did not also opt
into built-ins, the built-in list starts empty.
Tests added:
- test_clear_oauth_tokens_async_no_op_safe
- test_delete_server_calls_oauth_cleanup_when_oauth_was_on
- test_delete_server_skips_oauth_cleanup_when_oauth_off
- test_update_server_clears_oauth_on_url_change
- test_update_server_clears_oauth_when_oauth_disabled
26 backend MCP tests pass; full studio/backend suite 1710 passed locally.
Cross-platform CI (Linux, macOS, Windows) green on staging fork.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* PR #5750 round 3: reject null bool updates + /test surfaces 400
Round 3 of cross-platform validation:
1. PUT /api/mcp/servers/<id> would 500 with TypeError when the body
explicitly set is_enabled or use_oauth to null. Pydantic accepts
None for an Optional[bool] and _changes_from_payload then passed
None into mcp_servers_db.update_server, which int(None)d. Reject
explicit null at the validation layer with 400 instead.
2. POST /api/mcp/servers/test caught HTTPException under
"except Exception", so an invalid URL came back as HTTP 200 with
{"ok": false, "error": "400: ..."} instead of a real 400. The
create + update paths return 400 for the same input. Move
validation outside the transport try/except so it surfaces 400.
Tests added:
- test_changes_from_payload_rejects_null_is_enabled
- test_changes_from_payload_rejects_null_use_oauth
- test_test_endpoint_surfaces_url_validation_as_400
* PR #5750 round 4: hyphenated MCP tool names + empty-tool-list gate
Round 4 surfaces two more interaction bugs between the new MCP path
and existing safetensors tool plumbing:
1. OpenAI accepts ^[a-zA-Z0-9_-]{1,64}$ for function.name, and round 1
widened the MCP regex to that set, so MCP tools can now be advertised
as `mcp__srv__list-issues`. But the XML tool-call parser in
tool_call_parser.py used `\w+` (no hyphen), so the model could call
the tool but Studio could not parse the call. Same in
routes/inference.py's `_TOOL_XML_RE` stripper, which would leave
hyphenated tool-call XML in the visible content. Both regexes now
use `[\w-]+`.
2. safetensors_agentic treats `tools=[]` as "allow all" (documented
contract, exercised by test_empty_tools_list_does_not_enforce_allowlist).
When a caller sends `enable_tools=true` + `enabled_tools=[]` +
`mcp_enabled=true` and MCP discovery returns 0, the resolved tool
list is genuinely empty and built-in tools (web_search / python /
terminal) could execute via the model's emitted call. Fix at the
route gate instead of breaking the documented contract: set
`use_tools=False` when the resolved list is empty, in both GGUF and
safetensors paths. Existing callers who omit `enabled_tools` still
get ALL_TOOLS and are unaffected.
Tests added (32 total):
- test_tool_xml_parser_handles_hyphenated_function_names
- test_tool_xml_strip_handles_hyphenated_function_names
- test_safetensors_agentic_empty_allowlist_still_means_allow_all
(documents the contract round 4 preserved)
1716 passed locally; cross-platform CI on staging fork still green.
* PR #5750 round 5: GGUF allow-list + CLI policy + hyphenated params + cancel race
Round 5 of parallel-reviewer aggregation surfaced six additional
findings; five are real and fixed here:
1. Hyphenated MCP parameter names (`<parameter=issue-number>`) were
dropped by the XML parser's `\w+` regex. Extended to `[\w-]+` in
both core/inference/tool_call_parser.py and core/tool_healing.py.
The latter is GGUF's own copy of the parser/strip patterns and was
missed by round 4.
2. core/tool_healing.py's `strip_tool_call_markup` still used
`<function=\w+>` so hyphenated MCP tool-call XML leaked into the
GGUF visible content even after round 4 fixed the shared parser.
3+4. `mcp_enabled` re-opened the tool loop even when the operator
passed `unsloth run --disable-tools` (CLI policy False). Round 2's
`(_tools_on or payload.mcp_enabled)` gate ignored the raw process
policy. Now reads `state.tool_policy.get_tool_policy()` and gates
mcp_enabled on `_cli_policy is not False`. Applied to both GGUF
and safetensors paths.
5. GGUF's agentic loop called `execute_tool(tool_name, ...)` without
checking the model-emitted name against the per-request tool list,
while the safetensors loop already enforces this. Added the same
allow-list check so a model that hallucinates a filtered MCP name
or a built-in the caller opted out of returns "not enabled" instead
of executing.
Bonus P2 fixes:
- `call_tool_sync` now checks `cancel_event.is_set()` BEFORE
creating the call task, so a pre-set cancellation does not open
the HTTP transport.
- `clear_oauth_tokens_async` moved the OAuth import + construction
inside the protected try block; a fastmcp.client.auth load error
used to escape and 500 the delete / update route.
NOT fixed (verified false or out of scope):
- finding #10 "structured_content vs structuredContent": fastmcp's
CallToolResult dataclass uses snake_case (verified live against
structured-only tool result; fields are
`dict_keys(['content', 'structured_content', 'meta', 'data', 'is_error'])`).
- finding #11 "asyncio.run from running loop": call_tool_sync is
invoked from `asyncio.to_thread` worker threads which have no
event loop; asyncio.run() is safe there.
Tests added (37 total): hyphenated param names, tool_healing strip,
GGUF allow-list gate, cancel pre-set short-circuit, OAuth cleanup
constructor-error swallowing. 1721 passed locally, no regressions.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Studio: add Gemini provider with web_search, code_execution, prompt caching, and Nano Banana image generation
Wires Google's native Gemini API into Studio's external-provider stack
so users can pick gemini-2.5-pro / gemini-2.5-flash / gemini-2.5-flash-image
(Nano Banana) alongside the existing OpenAI / Anthropic / OpenRouter
providers. Gemini does not speak OpenAI Chat Completions on its primary
endpoint; the new `_stream_gemini` async generator translates between
the two shapes the same way `_stream_anthropic` handles the Messages API.
Backend:
- New `_stream_gemini` translator in external_provider.py. Converts
OpenAI messages -> Gemini `contents` + `systemInstruction`; maps
generationConfig (temperature / topP / topK / maxOutputTokens);
forwards `tools: [{googleSearch: {}}]` for web_search and
`{codeExecution: {}}` for code_execution; passes `cachedContent`
through for prompt caching; sets `responseModalities=[TEXT, IMAGE]`
for Nano Banana image generation.
- Translates streamed `GenerateContentResponse` SSE frames back into
OpenAI chat.completion.chunk frames (text deltas, function_call ->
tool_calls deltas, inlineData -> image_b64 tool_end envelope, usage
chunk before [DONE]).
- Registry entry switched to native base URL
`https://generativelanguage.googleapis.com/v1beta` with
`openai_compatible: False` and the `x-goog-api-key` auth header.
Model lineup curated to current 2.5 / 2.0 family + Nano Banana.
Frontend:
- Provider-capability matrix: Gemini supports temperature, top_p, top_k,
presence_penalty (matches generationConfig); min_p / repetition_penalty
hidden because the API does not accept them.
- `providerSupportsBuiltinWebSearch` / `providerSupportsBuiltinCodeExecution`
/ `providerSupportsBuiltinImageGeneration` extended for Gemini.
- Prompt caching toggle now also lit on Gemini.
Tests:
- 21 new tests in `test_gemini_provider.py` using httpx.MockTransport.
Cover request body shape conversion, URL/header wiring, web_search
forwarded as googleSearch, function-call translation both directions,
prompt caching passthrough, image generation emitting image_b64,
grounded-search citations -> tool_end, finish_reason mapping, and
vision data URL -> inlineData translation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: forward presence_penalty to Gemini and recover function name from tool_call_id
Two follow-up fixes for the Gemini provider:
* Thread presence_penalty into _stream_gemini and set
generationConfig.presencePenalty when non-zero. The OpenAI-side
capability matrix already exposes the slider for Gemini, so the
value was being collected and silently dropped on the way out.
* When an OpenAI role=tool message omits 'name' and only carries
'tool_call_id', recover the function name from the matching
functionCall on the prior assistant turn. Gemini 400s on an empty
functionResponse name.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: surface Gemini code execution parts as code_execution tool events
The Gemini stream parser only handled text/functionCall/inlineData
parts, so when the user toggled the Code pill on a Gemini model the
sandbox output (executableCode + codeExecutionResult parts) was
dropped on the floor while adjacent text reached the UI. Reviewers
flagged this as the headline feature being silently broken.
Translate both parts into the existing code_execution tool envelope
that CodeExecutionToolUI already consumes for OpenAI / Anthropic:
* executableCode -> tool_start with kind=code_execution and the
source code under arguments.code. We mint a tool_call_id and
stash it so the matching result block can pair to it.
* codeExecutionResult -> tool_end on that id with the stdout under
result. Non-OK outcomes (OUTCOME_FAILED / OUTCOME_DEADLINE_EXCEEDED)
are prefixed onto the text so the failure is visible.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: native Gemini model catalog, function-call ids, and honest cache claim
Three follow-ups to the Gemini provider PR after the codex pass:
* list_models() now translates Gemini's native /v1beta/models
payload ({models[{name, baseModelId, displayName,
supportedGenerationMethods}]}) into the OpenAI-compatible shape
Studio expects. Without this the picker stayed empty for Gemini
and fell back to hardcoded defaults. Embedding-only models are
filtered out.
* Forward the OpenAI tool_call id into Gemini's functionCall.id
and mirror it onto functionResponse.id. Two parallel calls to
the same function name can now be paired unambiguously on the
follow-up turn.
* Drop Gemini from the prompt-caching capability set. The wire
flow requires a separate cachedContents POST first and the
boolean Studio emits today is a no-op; the toggle should not
advertise a feature it cannot apply. Leaves a pointer to the
docs for the eventual two-step orchestration.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: distinct tool_calls index per emitted Gemini function call
Codex flagged that the Gemini stream parser hardcoded
tool_calls[0].index to 0 on every emitted functionCall. OpenAI
reassemblers key tool_calls by index when joining deltas, so two
parallel function calls in one assistant turn collapsed onto a
single slot and the second call's arguments overwrote the first.
Track the running count via len(emitted_function_call_ids) - 1
and emit it as the per-call index. The dedupe guard above (skip
when fc_id already in the set) means the index is monotonic and
stable for the lifetime of the stream. Regression test asserts
[0, 1] across two parallel calls in one candidate parts list.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: surface Gemini 3.5/3.1/3 + Nano Banana 2/Pro and plumb thinking budget
`gemini-2.0-flash` / `gemini-2.0-flash-exp` were retired by Google in 2026
(`/v1beta/models/gemini-2.0-flash:streamGenerateContent` returns HTTP 404
"no longer available to new users"), and the picker had nothing past the
2.x family. Verified against the live ListModels catalog: drop the retired
ids from `default_models` + allowlist and surface the chat-capable
3.5 / 3.1 / 3 families plus the Nano Banana image trio.
Also plumb `enable_thinking` / `reasoning_effort` into Gemini's
`generationConfig.thinkingConfig`. Without this, Gemini 3.5 Flash,
gemini-pro-latest, and the 3.x previews silently spend the caller's
`max_tokens` budget on hidden "thoughts" before emitting any visible
answer -- the chat shows a truncated stub like "The capital of" and
streams stop. Mapping:
- enable_thinking=False / reasoning_effort=none -> thinkingBudget=0
(Flash tier; Pro tier coerces to a small positive budget because
the API 400s on 0 with "This model only works in thinking mode")
- minimal/low/medium/high -> 512/2048/8192/24576 budget tokens
- max/xhigh -> -1 (dynamic)
- default (neither knob set) -> thinkingConfig omitted, model decides
Frontend `getExternalReasoningCapabilities` now surfaces a
`reasoning_effort` picker for every Gemini chat id (Pro tier hides the
"none" option; image-tier ids stay knob-less). Adds 6 unit tests
covering Flash/Pro effort mapping, the off-toggle coercion on Pro,
default omission, and the nano-banana-pro-preview alias routing
through the image modalities path. 28 -> 34 tests in
`test_gemini_provider.py`, all green; full backend suite still passes
(1459/1460; the unrelated test_help_output flake is pre-existing and
not in any file this PR touches).
Live verification against generativelanguage.googleapis.com on
2026-05-24 with `_stream_gemini` directly:
text gemini-3.5-flash single PASS multi PASS
text gemini-3.1-pro-preview single PASS multi PASS
text gemini-3.1-flash-lite single PASS multi PASS
text gemini-3-pro-preview single PASS multi PASS
text gemini-3-flash-preview single PASS multi PASS
text gemini-2.5-pro single PASS multi PASS
text gemini-2.5-flash single PASS multi PASS
text gemini-2.5-flash-lite single PASS multi PASS
text gemini-flash-latest single PASS multi PASS
text gemini-flash-lite-latest single PASS multi PASS
text gemini-pro-latest single PASS multi PASS
image gemini-2.5-flash-image PASS (1082 KB png returned)
image gemini-3.1-flash-image-preview PASS (Nano Banana 2)
image gemini-3-pro-image-preview PASS (Nano Banana Pro)
tool web_search PASS
tool code_execution PASS
-> 16/16 e2e through the actual ExternalProviderClient code path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten Gemini provider after review (PR #5720)
Fixes a batch of bugs surfaced by a second-pass review on top of the
3.5/3.1/3 + Nano Banana 2/Pro additions in c6724dbd.
Backend (external_provider.py):
- Constructor normalises legacy /v1beta/openai base URLs to /v1beta so
Gemini providers saved before the native switch keep working without
a manual re-config.
- Skip thinkingConfig, googleSearch, and codeExecution on image-tier
models (-image / nano-banana). The image responseModalities path is
mutually exclusive with text-tool wiring and stale UI state would
otherwise 400 the turn.
- _PRO_THINKING_PREFIXES now includes gemini-3.5-pro and uses anchored
prefix matching (exact id or "<prefix>-...") so the image-tier
gemini-3-pro-image-preview cannot accidentally match the pro guard.
- Gemini 3 functionCall thoughtSignature is round-tripped through the
tool_calls envelope via extra_content.google.thought_signature on
emit, and replayed as a sibling of functionCall on the next request.
- finishReason swaps STOP -> tool_calls when any functionCall was
emitted on the same turn so OAI clients trigger tool execution
(matches the OpenAI Chat Completions contract).
- usageMetadata.thoughtsTokenCount is rolled into output_tokens and
surfaced on output_tokens_details.reasoning_tokens so total_tokens
reflects the full billable spend instead of dropping the hidden
reasoning slice.
Registry (providers.py):
- Drop gemini-3-pro-preview from default_models. Google shut it down
on 2026-03-09 and auto-redirects to gemini-3.1-pro-preview; we
surface the canonical id only.
- Add model_id_deny_exact = ("gemini-3-pro-preview",) so the live
ListModels fetch does not re-surface the redirect alias.
Route schema (models/inference.py):
- enable_prompt_caching widened to Optional[Union[bool, str]] so the
/v1/chat/completions caller can pass a Gemini cachedContent resource
name (e.g. cachedContents/abc123). Without this widening _stream_gemini
s string cachedContent passthrough was unreachable from the public
route (bool_parsing 422). stream_chat_completion signature mirrors.
Frontend (provider-capabilities.ts, chat-page.tsx, chat-adapter.ts):
- providerSupportsBuiltinImageGeneration now also recognises
nano-banana ids (nano-banana-pro-preview was hidden from the image
pill before).
- providerSupportsBuiltinWebSearch takes the model id so Gemini image
models hide the Search pill (mirrors the backend skip).
- providerSupportsBuiltinCodeExecution uses the same isGeminiImageModel
guard for nano-banana ids.
- GEMINI_THINKING_PRO_PREFIXES gains gemini-3.5-pro; gemini-3-pro
tightened to gemini-3-pro-preview to avoid the image-id overlap.
- Updated 3 callers of providerSupportsBuiltinWebSearch to thread the
selected model id through.
Tests (test_gemini_provider.py): 34 -> 42, all green
- test_image_models_skip_thinking_config
- test_image_models_drop_text_only_tools
- test_gemini_35_pro_recognized_as_pro_thinking
- test_legacy_openai_base_url_normalized
- test_finish_reason_swaps_to_tool_calls_when_function_call_emitted
- test_thought_signature_round_trips_into_gemini_function_call
- test_thought_signature_emitted_in_tool_call_delta
- test_usage_chunk_includes_thoughts_tokens
Verification:
- Backend pytest 1518/1519 passing (one unrelated Qwen3.5 flash-attn
test fails on main as well; nothing in this PR touches that path).
- Frontend npx tsc -b clean.
- Live e2e 16/16 against generativelanguage.googleapis.com through the
patched _stream_gemini code path (all 11 chat models single + multi
turn, all 3 image models returned image bytes, web_search and
code_execution tools both emit the expected envelope).
- Live /api/providers/models against the patched backend surfaces 16
ids (gemini-3-pro-preview correctly filtered via deny_exact).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address second-pass review findings on Gemini (PR #5720)
Round-2 reviewer.py flagged a phantom web_search card on image
turns (12/12 reviewers), route-layer stripping of tool_calls /
tool_call_id / name, an over-narrow image-mode tool guard, and
silent safety blocks. This patch fixes all four.
Backend (external_provider.py):
- web_search_active is now derived from the outbound tools_array
(whether googleSearch was actually forwarded), not the raw
enabled_tools intent. Image-mode turns dropped the tool above so
the inbound stream no longer emits a phantom "search complete"
tool_start / tool_end on those turns.
- text_tools_allowed now uses is_image_model (covers both `-image`
/ `nano-banana` picker models AND text models that requested
`image_generation` via enabled_tools). Verified against the live
Gemini API which rejects both googleSearch and codeExecution
alongside responseModalities=["TEXT","IMAGE"] with explicit 400s
("Search as tool is not enabled for this model", "Code execution
is not enabled for this model").
- promptFeedback.blockReason is surfaced as a 400 content-filter
error chunk instead of returning an empty successful assistant
response. The streaming loop closes the response before exiting.
Route (routes/inference.py):
- _build_external_messages now propagates tool_calls (assistant),
tool_call_id, and name (tool result) through every code path
(string content, multimodal content, non-vision fallback). Without
this Gemini 3 function-call round trips lost their thoughtSignature
+ tool_call_id at the route boundary, and functionResponse.name
arrived empty on the second turn.
- Assistant messages with content=None and tool_calls populated are
preserved as a synthetic empty-string content turn so the
Gemini translator can rebuild the functionCall part.
Tests (test_gemini_provider.py): 42 -> 45, all green
- test_image_models_suppress_phantom_web_search_card
- test_image_generation_tool_drops_text_tools
- test_prompt_feedback_block_reason_surfaces_as_error
Verification:
- Backend pytest 1736 / 1736 (the two pre-existing unrelated fails
on main, test_help_output and Qwen3.5 flash-attn pin, are skipped).
- Frontend npx tsc -b clean.
- Live e2e 16/16 against generativelanguage.googleapis.com:
11 chat models single + multi turn, 3 image models returning
image bytes, web_search and code_execution both PASS.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix third-pass Gemini findings (PR #5720)
Round 3 review follow-ups:
Backend (studio/backend/core/inference/external_provider.py):
- Close response AND aiter_lines iterator in a finally so normal,
prompt-block, and cancellation exits all clean up (eliminates the
RuntimeWarning about aclose never being awaited).
- Pair the synthetic web_search tool_start with a tool_end on the
promptFeedback.blockReason path so the UI does not leave a stuck
"searching..." spinner after the error toast.
- Preserve native id and thoughtSignature on executableCode and
codeExecutionResult tool events under google.native_part, and pair
the tool_end on the code-exec id so multi-turn code-execution
replays do not lose Gemini-required history.
- Carry part-level thoughtSignature on text deltas via
delta.extra_content.google.thought_signature and on inline image
tool_end via google.thought_signature so Gemini 3 image editing
and tool turns round-trip the signature on the next request.
- Guess remote image_url MIME from the URL path so PNG / WebP / GIF
inputs are not silently relabeled as JPEG.
- Roll usageMetadata.toolUsePromptTokenCount into translated input
tokens and surface thoughtsTokenCount as
completion_tokens_details.reasoning_tokens in _build_usage_chunk.
- Only normalize the Google-hosted /v1beta/openai legacy base URL;
custom proxies whose paths happen to end in /openai are left
untouched.
- Forward ChatCompletionRequest.tools and tool_choice through
stream_chat_completion into _stream_gemini, translating to
tools[].functionDeclarations and toolConfig.functionCallingConfig.
Frontend:
- chat-adapter: when Gemini image-generation is enabled for the turn,
also disable Search and Code so the request, builder, and active
pills agree with what the backend actually sends (the backend
already strips text tools when image_generation is in enabled_tools).
- chat-adapter: consume OpenAI-shape delta.tool_calls chunks so
Gemini function-call deltas without text surface as tool-call parts.
- shared-composer: disable Search and Code pills while Gemini image
mode is active so the UI matches the request.
Tests (studio/backend/tests/test_gemini_provider.py): adds coverage
for proxy base-url gating, remote image MIME inference,
toolUsePromptTokenCount, reasoning_tokens propagation, prompt-block
web_search tool_end pairing, native code-exec id/thoughtSignature
metadata, inline image thoughtSignature, text-chunk extra_content,
OpenAI tools/tool_choice translation, and image-model tool drop.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: Gemini 3 thinkingLevel + image-model Search grounding (PR #5720)
Gemini 3.x migrated to a string `thinkingConfig.thinkingLevel`
(MINIMAL/LOW/MEDIUM/HIGH) and rejects `thinkingBudget`+`thinkingLevel`
in the same request. Gemini 3 also cannot turn thinking fully off, so
the lowest position is "minimal" (Flash) or "low" (Pro rejects
"minimal").
- external_provider._stream_gemini: split thinking translation by
family. Gemini 3.x (3 / 3.1 / 3.5 + gemini-pro-latest /
gemini-flash-latest / gemini-flash-lite-latest) emits
thinkingConfig.thinkingLevel; effort none/off coerces to "low" on
Pro and "minimal" on Flash. Gemini 2.5 stays on thinkingBudget.
- external_provider._stream_gemini: allow `tools: [{googleSearch: {}}]`
on the Gemini 3 image family (gemini-3-pro-image-preview,
gemini-3.1-flash-image-preview, nano-banana-pro). Google's docs
document Search grounding on these. codeExecution stays blocked
on image mode (still mutually exclusive with responseModalities).
- provider-capabilities.ts: mirror the Gemini 3 effort ladders in
resolveGeminiReasoningCapabilities (Pro: low/medium/high; Flash:
minimal/low/medium/high; 2.5 Flash keeps the off-position).
- provider-capabilities.ts: providerSupportsBuiltinWebSearch now
returns true on the documented Gemini 3 image models so the pill
is reachable; older image ids (gemini-2.5-flash-image) still hide.
Tests: splits the existing thinkingBudget cases by family (Gemini 3
checks thinkingLevel; Gemini 2.5 keeps thinkingBudget), adds positive
googleSearch coverage for Gemini 3 image models and negative
googleSearch coverage for legacy image models.
References:
- https://ai.google.dev/gemini-api/docs/thinking
- https://ai.google.dev/gemini-api/docs/gemini-3
- https://ai.google.dev/gemini-api/docs/models/gemini-3-pro-image-preview
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: attach Gemini code_execution inline images to the code card (PR #5720)
When a text Gemini turn wires codeExecution and the sandbox produces a
matplotlib plot, the inline image part ships right after the
codeExecutionResult. Previously this surfaced as a separate empty
image_generation card. Track the most recent code_execution
tool_call_id + result text and, when an inline image follows with
code_execution active, emit a second tool_end on the same id that
appends the image as a data: URI under the `__IMAGES__:` marker the
chat-adapter already understands.
Image-picker turns (`-image` / `nano-banana`) keep the standalone
image_generation envelope so Nano Banana outputs render the same way.
Tests: covers the merged code-execution card emission with no
standalone image_generation event when code_execution is the active
tool.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix fourth-pass Gemini findings (PR #5720)
Round 4 review follow-ups:
Backend:
- `_is_openai_compatible` + `_auth_headers` detect Gemini connections
pointed at a custom OpenAI-compatible proxy (non-Google host whose
path ends in `/openai`) and route them through the OpenAI-compat
surface with `Authorization: Bearer ...` instead of the native
`_stream_gemini` translator + `x-goog-api-key`. Google-hosted Gemini
keeps the native dispatch path it migrated to in this PR.
- `_stream_gemini` thinkingLevel handling for Gemini 3 Pro now coerces
both "minimal" and "medium" effort to "low" / "high" respectively
(Pro tier only accepts low/high per
https://ai.google.dev/gemini-api/docs/thinking).
- `providers.py` `default_models` restores the advertised
`gemini-3.5-pro` and the rolling `gemini-pro-latest` /
`gemini-flash-latest` / `gemini-flash-lite-latest` aliases that the
allowlist already admits.
Frontend:
- chat-adapter: lean on `providerSupportsBuiltinWebSearch` (which
already encodes the Gemini 3 image-model Search allowance) instead
of blanket-disabling Search whenever Gemini image mode is active.
Code execution stays blocked because Gemini image mode rejects it.
- shared-composer: mirror the same gate -- only the Code pill is
unconditionally disabled in Gemini image mode; the Search pill is
driven by `supportsBuiltinWebSearch`.
- provider-capabilities: Gemini 3 Pro reasoning levels now expose only
"low" and "high" (no Medium pill) to match the API.
Tests: covers the Gemini 3 Pro medium / minimal coercion, the custom
proxy OAI-compat dispatch + Authorization Bearer auth, and the
native-vs-proxy detection. Also closes the mocked httpx.AsyncClient
inside the test event loop so the Python 3.13 `aclose was never
awaited` warning no longer fires.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix fifth-pass Gemini findings (PR #5720)
Round 5 review follow-ups:
Backend:
- `_is_openai_compatible` + `_auth_headers` now treat ANY non-Google
Gemini base URL as OpenAI-compat (LiteLLM / custom OAI gateways /
OpenAI-compat vLLM routers), not just paths ending in `/openai`.
Pre-existing saved Gemini proxies on `/v1` keep working.
- Gemini 3 thinkingLevel coercion narrowed to the documented
inconsistencies: only "minimal" is coerced to "low" on Pro tier.
"medium" passes through (Gemini 3.1 Pro accepts it per
https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/3-1-pro).
- `_stream_gemini` only flips `responseModalities=[TEXT,IMAGE]` when
the selected model is image-capable. A stale
`enabled_tools=["image_generation"]` on a text model is silently
dropped instead of producing an invalid Gemini request.
- `_stream_gemini` validates the model id against
`[A-Za-z0-9._-]+` before URL interpolation so a model like
`../cachedContents/x` cannot redirect the request to an unintended
endpoint with the configured API key attached.
- Empty-text Gemini parts that still carry `thoughtSignature` emit a
content-free delta with `extra_content.google.thought_signature` so
Gemini 3 turns that end with a signature-only fragment do not lose
the replay state.
- ConnectError / ReadTimeout / generic HTTPError paths in
`_stream_gemini` now close the synthetic web_search tool_start
with a matching tool_end before the error chunk so the UI does not
leave a stuck "searching..." card on transport failure.
- `providers.py` default_models drop the non-existent
`gemini-3.5-pro` (Google launched only `gemini-3.5-flash` at
I/O 2026; Pro tier remains `gemini-3.1-pro-preview`).
- `routes/inference.py` only forwards `payload.top_k` when the caller
explicitly set it on the request (Pydantic `model_fields_set`).
Omitted top_k stays omitted, restoring the pre-PR behavior where
Gemini uses its server default.
- `ChatCompletionRequest.enable_prompt_caching` adds a `mode="before"`
validator that coerces the canonical string literals "true"/"false"
back to bool so historical opt-out callers keep working after the
field widened to `Union[bool, str]` for Gemini cache resource names.
Frontend:
- `providerSupportsBuiltinWebSearch` / Code / Image now accept the
saved connection `baseUrl` and return false for custom OAI-compat
Gemini proxies. Backend skips `_stream_gemini` for those bases, so
native tool envelopes never reach them; hiding the pills keeps the
request, builder, and UI consistent.
- `provider-capabilities.ts` Gemini 3 Pro effort ladder restores
`["low", "medium", "high"]` to match Google's documented levels.
- Call sites in `chat-page.tsx` and `chat-adapter.ts` pass through
`provider.baseUrl` so the proxy gate fires.
Tests: covers Gemini 3 Pro medium pass-through, custom proxy dispatch
on `/v1` and `/openai` bases, path-traversal model id rejection,
top_k omission when not explicit, text-model image_generation drop,
empty-text + thoughtSignature surfacing, and
enable_prompt_caching string coercion.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix sixth-pass Gemini findings (PR #5720)
Round 6 review follow-ups:
Frontend:
- chat-adapter `delta.tool_calls` accumulates fragments by `id` /
`index` instead of pushing a new tool-call card per chunk. The
standard OpenAI Chat Completions stream contract sends `id`/`name`
on the first chunk and partial `function.arguments` on subsequent
chunks; our previous handler parsed each fragment as a standalone
tool call. Local llama.cpp and OAI-compat providers that stream
fragments now reassemble into a single function-call part.
- chat-adapter also preserves `extra_content` on streamed tool-call
deltas so Gemini 3 `thoughtSignature` survives to the next turn.
- provider-capabilities Gemini 3 Pro restores "medium" in the
reasoning-effort ladder (Google's official Gemini API thinking
doc lists low/medium/high for Gemini 3.1 Pro; my earlier round 4
coercion was wrong).
- provider-capabilities orders `gemini-2.5-flash-lite` ahead of the
broader `gemini-2.5-flash` prefix so Flash-Lite falls into the
"no native thinking knob" branch as documented.
* Studio: round-trip Gemini tool_calls and tool results (PR #5720)
Recurring round 3-6 P1: the chat-adapter renders Gemini function-call
parts and code-execution events but `toOpenAIMessage` only serialized
text + image content, so the next turn lost the assistant
`tool_calls[]` (including Gemini 3's required
`extra_content.google.thought_signature`) and the matching
`role="tool"` result. Gemini 3 multi-turn function calling and code
execution failed validation on the second turn.
Frontend:
- types/api.ts widens OpenAIChatMessage to permit `role="tool"`,
`tool_calls`, `tool_call_id`, `name`, and `content: null`. Adds
OpenAIToolCallPart with `extra_content` for the Gemini round-trip.
- chat-adapter: new `toOpenAIMessages` expands an assistant turn with
tool-call parts into [assistant w/ tool_calls + extra_content,
role=tool result, ...]. tool result content is JSON-serialized so
the backend translator can rebuild Gemini's `functionResponse`
shape.
- chat-adapter outbound history now uses `flatMap(toOpenAIMessages)`
so each assistant tool-call round-trips through the standard OAI
shape the backend's `_stream_gemini` already understands.
* Studio: replay Gemini code_execution and image native parts on history (PR #5720)
Multi-turn Gemini history previously lost the native executableCode,
codeExecutionResult, and inlineData parts because the outbound
translator regenerated a generic functionCall for every assistant
tool_call. Stow the native dict on tool_end (frontend) and replay it
verbatim with thoughtSignature (backend) so follow-up turns preserve
the prior execution and image generation state. Skip role="tool"
fan-out for server-side builtin tools so Gemini does not 400 on a
functionResponse with no matching user-declared function.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: complete Gemini built-in tool replay round-trip (PR #5720)
Round 7 follow-up to the multi-turn native-part work. Three asymmetric
storage/consume gaps remained between the backend translator and the
chat adapter, so realistic Gemini follow-up turns degraded to generic
functionCalls instead of native history.
- Frontend collectAssistantToolCalls now drops web_search outright,
drops code_execution / image_generation when the native part is
missing, and promotes args.google to extra_content.google so the
backend native_part replay branch actually fires.
- Backend image_generation tool_end now emits google.native_part
with the inlineData (mimeType + base64) and thoughtSignature so the
follow-up image-edit turn can replay the prior image as a native
Gemini model part.
- Backend code-execution plot tool_end now stows google.native_part
with the inlineData so the merged code-exec card can round-trip
executableCode + codeExecutionResult + inlineData on the same id.
- Added regression tests for image-gen native-part replay and the
code-exec plot native_part stow.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 8 Gemini follow-ups (PR #5720)
- Text-part thoughtSignature: stow on the assistant message during
streaming and replay onto the last text part on the next turn so
Gemini 3 strict function-calling does not reject history.
- Function declarations: recursively strip Gemini-unsupported OpenAPI
keys (additionalProperties, $schema, $defs, strict, etc.) so OpenAI
strict tools stop 400ing as INVALID_ARGUMENT on Gemini.
- OpenAI-compat fallback: forward tools/tool_choice so custom Gemini
proxies (LiteLLM, gateways) keep function-calling.
- enable_prompt_caching: cover the Pydantic v1 legacy off/on/f/n/t/y
string set so explicit opt-outs stay opt-out (Gemini was sending
cachedContent: "off" otherwise).
- Frontend collectAssistantToolCalls / collectToolResultMessages: use
google.native_part + result presence to disambiguate provider
builtins from same-named user-declared functions.
- Added regression tests for text-signature replay and schema
sanitization.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 9 Gemini follow-ups (PR #5720)
Two round-9 convergent finds across the 12 reviewers:
- Server-side web_search was leaking onto the next turn as a fake
user functionCall/functionResponse. The previous heuristic (skip
builtin only when no native_part AND no result) let it through
because the synthetic tool card has a non-empty result string.
Always skip web_search by name on both serializers, accept that a
user-declared function literally named "web_search" must use a
different name.
- Assistant `extra_content` was dropped by ChatMessage validation
before _stream_gemini could replay text-part thought signatures.
Add the field to ChatMessage and forward it through
_build_external_messages so the multi-turn signature path actually
carries data.
Includes a regression test for the ChatMessage round-trip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 10 Gemini follow-ups (PR #5720)
Three convergent round-10 reviewer findings closed:
- Tag synthetic provider-side builtins with `args._server_tool=True`
via a central helper that runs in every `_emit_tool_event` /
`_emit_synthetic_tool_event` path. The frontend filter now skips
on that marker instead of on the public tool name, so local
llama.cpp `web_search` and OpenAI function tools literally named
`web_search` / `code_execution` / `image_generation` round-trip
cleanly while Gemini grounding / hosted code-exec / hosted image
cards stay skipped.
- Gate Gemini image-mode (responseModalities=[TEXT,IMAGE]) on the
Images pill (enabled_tools containing `image_generation`).
Selecting an image-capable model with the pill off no longer forces
image output the UI says is disabled.
- Frontend missing-key guard now exempts custom Gemini OAI-compat
proxies (LiteLLM, gateways) the same way the backend already
does, so a saved Gemini connection on `http://localhost:4000/v1`
with no API key stops being blocked.
Existing tests updated to pass `enabled_tools=["image_generation"]`
on image-mode capture paths.
* Studio: round 11 Gemini follow-ups (PR #5720)
Four round-11 findings closed:
- Kimi _stream_kimi_web_search's local _synthetic_chunk helper now
runs through _stamp_server_tool_marker so Kimi search history is
not replayed as a fake user functionCall on the next turn (was an
asymmetric miss after the round-10 tagging work).
- OpenAI Responses path (/v1/responses for gpt-5.x) forwards
caller-supplied tools / tool_choice, translating the Chat
Completions function-tool shape into the Responses native shape.
Without this, standard OpenAI tools silently dropped on
Responses-routed traffic.
- Decoupled the Gemini image-tier model-id guards (text-tool /
thinking strip) from the Images pill flip
(responseModalities=[TEXT,IMAGE]). gemini-2.5-flash-image with
Search/Code on and the Images pill OFF no longer forwards
googleSearch + thinkingConfig (Gemini 400s on those for legacy
image ids).
- Gemini-only extra_content is now forwarded by
_build_external_messages only when provider_type=="gemini" so
Google's thought_signature does not leak into OpenAI / Mistral /
Kimi / OpenRouter request bodies as an unknown field.
Added a regression test for the image-tier strict-guard split and
extended the extra_content test to cover the non-Gemini suppression.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 12 Gemini follow-ups (PR #5720)
Three round-12 convergent findings closed:
- extra_content leak to custom Gemini OAI-compat proxies (8/12
reviewers). _build_external_messages now gates extra_content on
the native generativelanguage.googleapis.com host, not just
provider_type=="gemini", so LiteLLM / custom gateways routed
through /chat/completions do not get an unknown top-level field.
- OpenAI Responses function-tool round-trip (5/12 reviewers). I
added user `tools` forwarding in round 11 but did not parse the
matching response.output_item.done items of type=function_call.
The parser now translates them into Chat Completions
delta.tool_calls and the terminal chunk reports
finish_reason="tool_calls" when the model invoked a user
function.
- Image-tier model with Images pill OFF (2/12). Google's image
models default to text+image when responseModalities is omitted,
so the previous fix silently still billed image output. Force
responseModalities=["TEXT"] when the Images pill is off and the
selected model is image-capable.
Updated the two pre-existing tests that pinned the synthetic-tool
arguments shape to include the new `_server_tool: True` marker, and
added a regression test for the Responses function-call output
translation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 13 Gemini/Responses follow-ups (PR #5720)
Three round-13 convergent findings closed:
- OpenAI Responses function_call indices: my round-12 translator
hardcoded every emitted tool_calls[*].index to 0, so parallel
function calls collapsed for index-keyed clients. Track and
increment function_call_index per emit (mirrors the Gemini
branch's distinct-index pattern). 10/12 reviewers flagged.
- _SERVER_SIDE_BUILTIN_TOOL_NAMES now includes web_fetch so
Anthropic-hosted web_fetch cards carry the _server_tool marker
and the frontend history serializer doesn't replay them as fake
user functions. 4 reviewers flagged.
- OpenAI Responses follow-up tool results now serialize as
Responses-shape function_call / function_call_output items keyed
by call_id, instead of Chat Completions role="tool" content.
Skips assistant tool_calls tagged with _server_tool so hosted
builtins don't round-trip as user functions. 2 reviewers flagged.
Updated the Anthropic code_execution and web_fetch test argument
pins to include the new _server_tool marker, and added two
regression tests (distinct indices on parallel function_call,
function_call_output round-trip).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 14 Gemini follow-ups (PR #5720)
Three round-14 findings closed:
- Remote `image_url` translation (5 reviewers convergent). Public
HTTPS image URLs can't be sent as `fileData.fileUri` -- Gemini
reserves that path for Files API URIs and YouTube. Fetch the
bytes server-side and inline them as base64 `inlineData`,
mirroring the pre-PR OpenAI-compat behaviour. YouTube URLs and
generativelanguage.googleapis.com/v1beta/files/* stay as
`fileData`.
- Nullable JSON Schema type arrays. OpenAI strict tools commonly
use `"type": ["string", "null"]`; the Gemini sanitizer now
flattens that to `"type": "string", "nullable": true` so strict
function tools stop 400ing.
- Parallel functionResponses now ride on one user content block
with multiple `functionResponse` parts, matching Google's
parallel tool docs. Consecutive `role="tool"` messages merge
into the previous user turn instead of splitting into separate
Gemini user turns.
Three regression tests added (remote URL fetch + inline, Files
API / YouTube fileData preservation, schema nullable flattening,
parallel-tool grouping).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: SSRF harden Gemini remote image fetch (PR #5720)
Round 15 convergent finding (12/12 reviewers). My round-14 fix to
download user-controlled image URLs for inlineData inlining was an
SSRF / data-exfiltration path: no scheme check, no private-host
guard, no size cap, no Content-Type validation, redirects could
bounce to internal services, and the full URL was logged.
Replace the inline fetch with `_safe_fetch_image_for_gemini`:
- Require https:// (reject http, file, data, ftp, etc).
- Resolve the hostname via socket.getaddrinfo and reject if ANY
resolved address is private / loopback / link-local / multicast /
reserved / unspecified (covers 127.0.0.0/8, 10/8, 172.16/12,
192.168/16, ::1, 169.254/16 metadata, RFC 6890).
- Block IP-literal URLs that resolve into those same ranges.
- Cap response body at 10 MB (Content-Length pre-check + streamed
byte counter).
- Require Content-Type to start with `image/`.
- Disable redirect following so a 302 to a private host can't slip
past the address check.
- Use a short 15s timeout and a tiny connection pool dedicated to
these fetches.
- Log only the host name + error class -- no full URL, no signed
querystring leak.
If the guard rejects, the image part is silently dropped (instead
of forwarding raw bytes or a fileData fallback). Files API URIs
and YouTube URLs still ride as `fileData.fileUri` unchanged.
Tests: replaced the live-fetch test with a `_safe_fetch_image_for_gemini`
monkeypatch, added four new SSRF-guard tests (non-https rejected,
loopback / private IP literals rejected, hostnames that resolve to
private IPs rejected).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 16 Gemini follow-ups (PR #5720)
- IP-pinned image fetch (`_safe_fetch_image_for_gemini`): reuse the
validated-once-then-pin pattern from `tools._fetch_page_text` via
`asyncio.to_thread`, so DNS rebinding between validation and the
HTTP connect cannot redirect us at a private/metadata address.
Catch malformed-bracketed IPv6 urlparse errors. Follow up to 4
redirect hops with per-hop SSRF re-validation.
- Replace contains-substring detection of Gemini Files API + YouTube
URLs with parsed scheme/host/path checks, so attacker URLs like
`https://evil.example/path/youtube.com/x.png` no longer skip the
safe-fetch path and serialize as `fileData.fileUri`.
- `_build_external_messages`: strip per-tool-call `extra_content`
for non-native-Gemini providers; the Gemini-only
`thought_signature` payload was leaking through `tool_calls[]`
into /chat/completions on OpenAI, Anthropic, and custom Gemini
OAI-compat gateways.
- `_server_tool` marker now gated on the function name being one of
the canonical builtin names (`web_search`, `web_fetch`,
`code_execution`, `image_generation`) AND the marker being set,
so a user function whose schema happens to define an
`_server_tool` field is no longer dropped. Frontend filter mirrors
the same gate, plus a backward-compat fallback for pre-PR
persisted server-tool cards (no marker) routed via name +
native_part / web-tool heuristic.
- Gemini schema sanitizer collapses `anyOf: [{X}, {"type":"null"}]`
to `{X, "nullable": true}` so Optional[X] tool args from
OpenAI/Pydantic schemas no longer 400 the Gemini request.
- Frontend tool-result serializer emits `{"result":""}` for empty
string outputs so the ChatMessage validator does not reject
`role="tool"` with empty content.
- Coerce `medium` thinkingLevel to `high` for legacy
`gemini-3-pro*` / `gemini-3-pro-preview*` (only low/high
documented; shut down 2026-03-09); 3.1+ Pro still passes through.
- Hide Gemini native thinking ladder on custom OAI-compat Gemini
gateways by routing `getExternalReasoningCapabilities` through
`isGeminiCustomOpenAICompatBase(baseUrl)`; thread baseUrl through
all four call sites.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 17 Gemini follow-ups (PR #5720)
- Frontend `collectAssistantToolCalls` and `collectToolResultMessages`
no longer drop unmarked `web_search` / `web_fetch` cards by name
alone: a user-defined function with one of those names must
round-trip. Pre-PR persisted `code_execution` / `image_generation`
cards still get filtered via a shape heuristic (kind/command/code/
prompt fields) instead of bare name.
- `_build_external_messages._filter_tool_calls` now drops marked
server-side builtin `tool_calls` entirely for non-native-Gemini
providers, not just their `extra_content`. An assistant turn whose
only payload was a marked builtin is dropped completely so the
receiving provider does not see an orphan tool_call.
- `_stream_anthropic` translates OpenAI top-level `tool_calls` into
Anthropic native `{type:"tool_use", id, name, input}` content
blocks, and translates `role="tool"` follow-ups into `role:"user"`
messages carrying a `tool_result` block. Anthropic's native
Messages API rejects the OpenAI shapes.
- `_safe_fetch_image_for_gemini_sync` factors URL validation through
`_safe_parse_https`, so malformed `port` access (e.g.
`https://host:bad/x.png`) and malformed redirect targets (e.g. a
302 to `https://[bad/x.png`) drop the image instead of raising mid-
request.
- `tool_choice="none"` now disables hosted builtins (Gemini
googleSearch / codeExecution and OpenAI Responses web_search /
shell / image_generation), not just user function declarations.
- Schema sanitizer handles multi-type `anyOf` with null
(`Union[str, int, None]`): keep the slim non-null anyOf and add
`nullable: true` so Gemini does not reject `{"type":"null"}`.
- Image fetch falls back to the caller-provided MIME (guessed from
URL extension) when the server omits Content-Type instead of
dropping the image as `non-image content-type=<none>`.
- Per-request aggregate caps on remote image inlining (8 images,
20MB total) so a single chat request cannot force unbounded
backend downloads.
- Frontend exposes the reasoning ladder for `gemini-2.5-flash-lite`
(`none/minimal/low/medium/high/max`) so the UI can drive the
thinkingBudget the backend already supports.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 18 Gemini follow-ups (PR #5720)
- `tool_choice="none"` now opts out of hosted builtin tools on every
provider path, not just Gemini and OpenAI Responses. Anthropic
web_search / web_fetch / code_execution, Kimi `$web_search` early
return, and OpenRouter `plugins:[{id:"web"}]` are all gated on
`tool_choice_disabled`. Passing `enabled_tools=[...]` with
`tool_choice="none"` no longer triggers provider-side search /
code execution for any provider.
- `_stream_anthropic` accepts `tool_choice` and threads it through;
the dispatcher in `stream_chat_completion` forwards it.
- Frontend `isServerSideBuiltinToolPart` simplified to drop only on
(marker) OR (canonical name + native_part). The previous shape
heuristic on `args.kind`/`args.command`/`args.code`/`args.prompt`
dropped real user-declared `code_execution` / `image_generation`
functions. Pre-PR persisted hosted cards lacking the marker now
leak to non-native providers on switch -- preferred to silently
deleting legitimate function-call history.
- Backend `_is_marked_server_builtin_tool_call` and the OpenAI
Responses translator's matching filter accept BOTH `_server_tool`
marker AND `args.google.native_part` as durable provider-side
signals so Gemini code_execution / image_generation cards are
still dropped on a provider switch.
- Per-request remote image count cap now counts ATTEMPTS, not just
successful inlines, so 100 failing/slow URLs cannot each consume
the 15s fetch timeout. Data: URL images now share the same count
and byte caps as fetched remote URLs.
- OpenAI Responses translator tracks skipped server-builtin
`function_call` ids and drops their matching `role="tool"`
follow-ups, preventing orphan `function_call_output` items in the
outbound body.
- Gemini schema sanitizer preserves multi-type unions with null:
`{"type":["string","integer","null"]}` becomes
`anyOf:[{string},{integer}] + nullable:true` instead of being
flattened to the first non-null type.
- Gemini model id validation moved to the top of `_stream_gemini`
so an invalid model id rejects the request before any remote
image fetch / message translation side effect.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 19 Gemini follow-ups (PR #5720)
- `_build_external_messages` now skips an empty assistant turn when
`_filter_tool_calls` drops every synthetic builtin tool_call (was
guarded only on the `content is None` branch; the string-content
and list-content branches still forwarded
`{"role":"assistant","content":""}` which several providers
reject). Also tracks the dropped server-builtin tool_call ids and
skips the matching `role="tool"` follow-ups so the receiving
provider does not see an orphan tool_result.
- OpenRouter `web_search_active` (the synthetic tool_start /
tool_end emitter) is now also gated on `tool_choice_disabled` so
a request with `tool_choice="none"` does not surface a fake
web_search card in the chat UI even though the plugin was
correctly stripped from the outbound body.
- `_stream_anthropic` translates an OpenAI role="tool" with list
content (`content=[{"type":"text","text":"..."}]`) into a native
`tool_result` block on a user message; previously only the
string-content shape was translated, so list-content tool results
were forwarded as invalid `role:"tool"` messages.
- Gemini `data:` URL image_url parts now require an `image/*` MIME
type; a `data:text/html;base64,...` is dropped instead of being
forwarded as `inlineData.mimeType="text/html"` (Gemini rejects
the malformed image part). Symmetric with the fetched-remote
image fetch path that already rejects non-image Content-Type.
- YouTube `fileData.fileUri` now declares `video/mp4` as the
mimeType instead of `image/jpeg` guessed from the URL path. The
YouTube/fileData input is the documented Gemini video path; the
guessed image MIME made valid YouTube inputs malformed.
- OpenAI Responses translator preserves `response.output` ordering
on assistant turns that emitted both text and a function_call:
assistant text is now serialized BEFORE the function_call item
so the subsequent function_call_output (the matching role=tool
follow-up) lands in the right position. Previously the order
was function_call -> assistant text -> function_call_output,
which can confuse multi-turn function-calling flows.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 20 Gemini follow-ups (PR #5720)
Convergent reviewer findings from round 20:
- tool_choice="none" no longer flips responseModalities=[TEXT,IMAGE]
on image-tier Gemini models. Forced-function tool_choice (e.g.
{type:function, function:{name:lookup}}) also drops hosted Search /
code execution from the Gemini body so the caller's pinned user
function is not silently joined by hosted builtins.
- Gemini code-execution thoughtSignature replay now uses an ordered
parts list (native_part.parts[]) so per-part signatures stay
attached to the exact part Gemini emitted. The previous merged
shape fanned one top-level thoughtSignature across executableCode
+ codeExecutionResult + inlineData and tripped Gemini 3 strict
validators. Backward-compat fallback keeps pre-round-21 persisted
history working: a legacy native_part with a single subpart still
replays the signature on that subpart; merged legacy objects pin
the signature to executableCode only.
- Remote-image fetch threads the remaining per-request byte budget
into _safe_fetch_image_for_gemini, so over-budget URLs are
refused via Content-Length pre-check / short read instead of
fully downloaded then discarded after the aggregate cap check.
- Gemini role=tool with OpenAI list-form content
([{type:text,text:result}]) now flattens text parts before
building functionResponse.response.result; previously the parts
arrived as the result value instead of the actual tool output.
- Frontend chat-adapter merges native_part by concatenating parts
lists (preserving per-part thoughtSignature). Wire types expose
enable_prompt_caching as boolean|string (Gemini cached-content
name) and OpenAIChatDelta now carries tool_calls and extra_content.
- Test test_openrouter_no_synthetic_web_search_event_on_tool_choice_none
reads _toolEvent from the top-level SSE payload so a backend
regression cannot mask the assertion.
Adds 7 regression tests covering image_generation gate, forced-function
gate, native_part list replay, legacy fallback, list-content
functionResponse flattening, fetch byte-budget threading, and wire
types.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Apply forced-function tool_choice gate to Anthropic, OpenRouter, Kimi
Previously only the Gemini path treated `tool_choice={"type":"function",
"function":{"name":...}}` as a hosted-tool opt-out. Anthropic,
OpenRouter, and Kimi still attached hosted web_search / web_fetch /
code_execution when the caller explicitly pinned a user function plus
`enabled_tools=[...]`. That contradicts the explicit function pin and
bills the caller for unwanted server-side calls.
Mirror the Gemini gate symmetrically:
- Anthropic web_search / web_fetch / code_execution
- OpenRouter `plugins:[{id:"web"}]` + the synthetic web_search SSE
event the same path emits at stream close
- Kimi `_stream_kimi_web_search` dispatch
Adds 4 regression tests:
- test_anthropic_forced_function_tool_choice_drops_hosted_tools
- test_openrouter_forced_function_tool_choice_drops_web_plugin
- test_kimi_forced_function_tool_choice_skips_web_search_helper
- test_openrouter_no_synthetic_web_search_event_on_forced_function_tool_choice
All 146 existing backend tests still pass.
* Strip Gemini-only synthetic tool history on local-GGUF dispatch
After a Gemini chat that ran code_execution / image_generation, switching
the same thread to a local GGUF model used to forward the synthetic
provider-side tool_calls (tagged with `args._server_tool` or carrying a
Gemini `args.google.native_part` payload) and the message-level
`extra_content` to llama-server. The receiving backend has no tool
declaration for those names and no use for Gemini thoughtSignature
metadata; in the worst case it can produce an orphan tool_call_id and a
confused continuation.
Add `_strip_provider_synthetic_tool_history()` and wire it through the
two local message builders:
- `_openai_messages_for_passthrough` (OAI-compat passthrough)
- `_openai_messages_for_gguf_chat` (standard GGUF chat path)
Real user-function `tool_calls` and their matching `role="tool"` replies
survive unchanged; only synthetic provider-side cards and Gemini-only
`extra_content` are stripped. If the synthetic call was the assistant
turn's only payload, the now-empty turn is dropped too so llama-server
does not reject the request.
Adds 2 regression tests:
- test_strip_provider_synthetic_tool_history_drops_synthetic_only
- test_strip_provider_synthetic_tool_history_drops_empty_assistant
142 existing backend tests still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Disable Search/Code composer pills for Gemini image-tier models
For external Gemini image-tier models (gemini-2.5-flash-image,
gemini-3.x-image-preview, etc.), the backend unconditionally strips
code_execution and strips web_search on older image ids. Search is
still allowed on Gemini 3.x Pro/Flash image models, which
supportsBuiltinWebSearch already encodes per model.
Before this commit the composer pill gates were:
searchDisabled = !modelLoaded || !(supportsTools || supportsBuiltinWebSearch)
codeDisabled = !modelLoaded || !(supportsTools || supportsBuiltinCodeExecution) || imageModeDisablesCode
`supportsTools` here is a local-runtime fallback that becomes true when
any tool-capable local model has been loaded in the session. With a
local tool-capable runtime active, switching the chat to an external
Gemini image-tier model used to leave Search/Code clickable, even
though the backend will silently drop the tool on the wire.
Detect "external provider is Gemini AND the model is image-tier" (via
supportsBuiltinImageGeneration) and gate the two pills strictly on the
provider's own builtin support in that case. Non-Gemini paths and
non-image Gemini models keep the supportsTools fallback unchanged.
* Apply forced-function tool_choice gate to OpenAI Responses path
Round 22 added the gate for Gemini / Anthropic / OpenRouter / Kimi but
missed the OpenAI Responses translator. When a caller pinned a user
function via `tool_choice={"type":"function","function":{"name":...}}`
plus `enabled_tools=["web_search","code_execution","image_generation"]`,
the Responses body still attached `{"type":"web_search"}`,
`{"type":"shell"}`, and `{"type":"image_generation"}` server tools. The
function pin should suppress those for the same privacy + billing reason
the other provider paths now do.
Compute `_responses_tool_choice_forced_function` next to
`_responses_tool_choice_none` and gate each hosted-tool append on
`_responses_hosted_builtins_allowed = not none and not forced_function`.
The fix has to be applied in TWO places: the initial body builder and
`_build_body()` (called by the container-expiry retry path). User
function declarations still flow through so the pin has something to
target, and the Responses-shape `{type:"function", name:"..."}`
`tool_choice` is forwarded unchanged.
Adds regression test `test_openai_responses_forced_function_tool_choice_drops_hosted_tools`.
All 166 existing backend tests across Gemini + Responses + image-gen +
code-exec suites still pass.
* Round 24 P1s: SSRF shared-address gap + extra_content text-only leak + custom-Gemini model list
Three convergent P1s from round 24 review:
1. SSRF: the shared SSRF validator in `tools._validate_and_resolve_host`
used a denylist (is_private / loopback / link_local / multicast /
reserved / unspecified). Python classifies shared address space
(100.64.0.0/10 carrier-grade NAT, plus 240.0.0.0/4, benchmarking
ranges, etc.) with `is_private=False` AND `is_global=False`. The new
Gemini server-side image fetcher therefore accepts URLs whose
hostname resolves to 100.64.0.1 in cloud/VPC deployments. Add
`not ip.is_global` as the primary gate -- a single source of truth
that covers every current and future non-global range.
2. _strip_provider_synthetic_tool_history previously only stripped
message-level `extra_content` when the assistant turn had tool_calls.
A plain text Gemini reply carrying
`extra_content.google.thought_signature` flowed through to
llama-server when the thread was switched to a local GGUF backend.
Always strip message-level `extra_content` on assistant turns.
3. routes/providers.list_provider_models applied Gemini's native
`model_id_allowlist` regex to every Gemini provider, including
custom OAI-compatible bases (LiteLLM, deployment gateways). IDs like
`google/gemini-2.5-flash` and team-prefixed deployment aliases got
filtered out even though the chat-dispatch path now routes them via
the OpenAI-compatible client. Skip registry-level model-id filters
when the configured Gemini base_url host is not the canonical
`generativelanguage.googleapis.com`, mirroring the chat-dispatch
gate.
Three regression tests added:
- test_validate_and_resolve_host_blocks_shared_address_space
- test_strip_provider_synthetic_tool_history_drops_text_only_extra_content
- test_gemini_custom_oai_compat_base_skips_native_allowlist
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Round 25 P1s: skip synthetic server-tool replay + inline $ref/$defs into Gemini schema
Two convergent reviewer findings on the native Gemini path:
1. _stream_gemini's tool_calls replay loop falls through to a generic
functionCall emission whenever it sees an assistant tool_call. Marked
server-side builtin cards (web_search / web_fetch tagged with
_server_tool or args.google.native_part) hit that fallthrough with no
replayable native_part, which produces an outbound functionCall whose
name is not a declared user function. The Gemini turn 400s on the
undeclared name. Guard the loop to drop those entries instead, while
keeping the existing code_execution / image_generation native-part
replay branch intact.
2. _sanitize_gemini_schema uses a strict allowlist that drops local
$ref / $defs references. Pydantic-generated tool schemas hoist nested
object shapes into $defs and reference them via {"$ref": "#/$defs/X"},
so a property like address: {"$ref": "#/$defs/Address"} collapsed to
{} on the wire and the model lost the nested fields, types, and
required keys. Resolve local #/... pointers against the schema root
and inline the referenced subtree, with local siblings overriding
the reference (normal JSON Schema composition) and a seen-ref guard
for self-referential schemas.
Added regression coverage:
- test_gemini_native_skips_synthetic_server_builtin_replay
- test_function_declarations_inline_local_refs_into_gemini_schema
- test_function_declarations_inline_local_refs_in_anyof_and_items
- test_function_declarations_self_referential_schema_terminates
All 145 Gemini provider tests pass; touched provider regression set
(OpenAI Responses, code execution, image generation, Anthropic code
execution, Anthropic web_fetch) also 43/43 green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Round 26 P1s: drop orphan Gemini functionResponse + Anthropic /messages synthetic-history strip
Reviewer round 26 surfaced two convergent asymmetric-fix bugs.
1. _stream_gemini drops a synthetic server-tool tool_call (web_search /
web_fetch tagged _server_tool) and also replays code_execution /
image_generation tool_calls as Gemini-native executableCode /
codeExecutionResult / inlineData parts. The matching role="tool"
follow-up was still falling through to the generic functionResponse
branch, producing either an orphan functionResponse (synthetic case)
or a duplicate response pointing at a name with no
functionDeclarations entry (native-part case). Both forms 400 the
next Gemini turn. Track skipped + native-replayed tool_call_ids in
_gemini_skip_tool_result_ids and short-circuit the role="tool"
branch on a match.
2. The Anthropic-compatible local /v1/messages route only called
_drop_empty_assistant_sentinels on the OpenAI-translated history,
while the sibling /v1/chat/completions and GGUF passthrough builders
chain that with _strip_provider_synthetic_tool_history. An Anthropic
caller replaying a prior provider-side tool_use therefore forwarded
fake builtin tool history straight into local llama-server. Apply
the same strip on the Anthropic route after the
anthropic_messages_to_openai conversion.
Regression coverage added:
- test_gemini_native_skips_orphan_function_response_for_dropped_builtin
- test_gemini_native_skips_orphan_function_response_for_native_part_replay
Gemini suite 147/147; touched provider regression set 43/43.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Round 27 P1s: native_part location fallback + Gemini image request budget for base64
Two convergent reviewer findings on the native Gemini path.
1. _stream_gemini's synthetic-builtin detector at lines 3519-3524
recognizes args.google.native_part as a server-tool marker, but
_native_part was only loaded from tc.extra_content.google.native_part.
A direct OpenAI-compatible API caller or imported third-party thread
round-trips the payload through function.arguments because
tool_calls[].extra_content is not in the OpenAI spec. The round-25
guard then saw a synthetic builtin with no _native_part and dropped
the entire assistant turn, so the next native Gemini request lost
the prior executableCode / inlineData / codeExecutionResult context.
Fall back to args.google.native_part when extra_content path is
missing, mirroring what the synthetic detector already accepts.
2. _GEMINI_REMOTE_IMAGE_MAX_TOTAL_BYTES capped DECODED bytes at 20MB.
Gemini receives images base64-encoded inside JSON, and base64
inflates payload size by ~4/3. With 20MB decoded the actual JSON
body is ~26.7MB plus prompt overhead, well over Gemini's ~20MB
request limit. Drop the decoded cap to 14MB so realistic multi-
image turns stay safely under 20MB encoded.
Added regression test test_gemini_native_part_falls_back_to_args_google
covering an OpenAI-compat-shaped image_generation tool_call whose
native_part lives only in function.arguments.
Gemini suite 148/148.
* Fix TS build errors from main merge: restore imageParts + refusal return [] + cast image-edit ref
Three errors in chat-adapter.ts surfaced by the frontend tsc step after merging
main into feat/gemini-provider:
1. The Anthropic refusal early-return used main's but
toOpenAIMessages returns SerializedMessage[]; flip to .
2. Restore -- the line
was lost when removing main's conflict block from the function body.
3. selectedImageEditReference splice was inserting OpenAIChatMessage
into a SerializedMessage[] array; the shapes differ on tool_calls.id
nullability. Cast the reference message through unknown -- it carries
no tool_calls, so the runtime payload is structurally compatible.
Reproduced locally with `tsc -b --pretty false` (now passes). Build
also failing in the in-repo `npm run build` step on PR CI; this commit
unblocks all 12 failing UI/API workflows.
* Tighten verbose comments in external_provider.py + chat-adapter.ts
Compress multi-line explanatory comments in the Gemini translator
and the chat adapter without changing any behaviour. All 148 Gemini
provider tests still pass; tsc --noEmit clean.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com>
* Studio: add VLM image-size control for training
Studio vision fine-tuning had no explicit way to cap image resolution, so
users could not trade visual detail against context and memory use from the
training UI, YAML config, or API payload. :) Add a nullable `vision_image_size`
setting that keeps the current model default when unset and applies a
max-side resize when provided.
- Add `vision_image_size` to the training request model, route payload, backend
training config, and frontend API/types plumbing.
- Validate the value server-side as either null or an integer in the supported
256-2048 range.
- Surface an Image Size selector for vision LoRA training with Default plus
common preset sizes.
- Include the value in training start payloads only for image-dataset vision
models, and serialize it into vision-aware YAML configs.
- Map backend model defaults back into the training store and reset the value
when reapplying model defaults.
- Pass the resize through the Torch trainer via `UnslothVisionDataCollator`
using max-dimension semantics.
- Apply the same max-dimension resize in the MLX VLM path before mlx-vlm's
internal collation, preserving aspect ratio and avoiding upscaling.
- Add backend validation coverage and MLX resize-size tests for the new
behavior.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: thread vision_image_size into DeepSeek OCR + writable MLX ndarray
- trainer.py: DeepSeek OCR collator now honors the new vision_image_size
setting as image_size. Falls back to 640 when null. base_size stays at
1024 and crop_mode stays True so the Gundam preset's dynamic cropping
of large documents keeps working.
- worker.py: _resize_mlx_vlm_image returns np.array(image, copy=True)
instead of np.asarray(image). The PIL view from np.asarray is not
writable, which makes HF VLM processors emit "The given NumPy array
is not writable, and PyTorch does not support non-writable tensors..."
when they call torch.from_numpy. copy=True keeps the same shape and
dtype but produces a writable buffer.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: align YAML export gate with API mapper + extend Image Size dropdown
- training-section.tsx: handleSaveConfig now passes
isVisionModel && isDatasetImage === true to serializeConfigToYaml,
matching buildTrainingStartPayload. Stops vision_image_size from
leaking into exported YAML for text-only datasets where the API
would have sent null.
- params-section.tsx: add 256 to visionImageSizePresets so the
dropdown spans the validator's full [256, 2048] range. Also render
a synthetic SelectItem for the current value when it was loaded
from YAML or model defaults and is not in the preset list, so the
controlled Select always shows the active size.
* Studio: validate vision_image_size in YAML/model-default loader
mapBackendModelConfigToTrainingPatch now mirrors the backend validator
at studio/backend/models/training.py:169 by dropping any value that is
not an integer in [256, 2048]. Pre-fix, an imported YAML like
vision_image_size: 4096 or 640.5 would land in the store and the UI
would happily display it, only to fail when Start Training posted to
the backend. With this guard the store never holds a value the backend
would reject.
* Studio: precise error messages for invalid vision_image_size inputs
Switch the field_validator to mode="before" so True/False surface as
bool (not Pydantic's coerced 1/0) and give a precise
"must be an integer or null" message instead of the misleading
"must be in [256, 2048] (got 1)". Also explicitly accepts numpy
Integral and integral Real scalars so YAML or programmatic callers
using numpy ints keep working.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: test that bool inputs yield the precise 'integer or null' error
Regression guard for the validator switch to mode="before". Pre-fix,
vision_image_size: True was rejected with "must be in [256, 2048]
(got 1)" because Pydantic coerced before our check ran. New test
asserts the message now reads "integer or null".
* Studio: tighten vision_image_size loader + YAML save + MLX rounding
Round 2 of follow-up review surfaced three usability issues:
- model-defaults.ts: switching to a model whose backend YAML omits
vision_image_size now explicitly resets the store value to null.
Pre-fix, a stale 2048 from a previous model would silently apply
to the new run because every checked-in model-default file omits
the key.
- training-section.tsx: handleSaveConfig now includes vision fields
unless isDatasetImage is definitively false. isDatasetImage is null
during dataset checks, after dataset edits, and on import; treating
unknown as "drop" would silently lose the user's selection in those
windows. Confirmed-text-only datasets still drop the value.
- worker.py: _mlx_vlm_max_resized_size now mirrors the Torch collator's
integer formula (w * size + size_func // 2) // size_func instead of
Python round(), which uses banker's rounding and disagreed by 1px on
half-pixel inputs like 333x1000 with target 500 (was 166, now 167).
Test_mlx_training_worker_config gains parity assertions.
* Studio: reset vision_image_size in the model-config error fallback path
mapBackendModelConfigToTrainingPatch resets stale image size on the
success path, but if the /api/models/config endpoint throws,
training-config-store.ts falls through to checkVisionModel and only
updates capability flags. Pre-fix that left a stale 2048 (or any
prior selection) in the store, so once dataset detection marked the
new dataset as image, the next training start would silently apply
the previous model's size. The error branch now also resets to the
DEFAULT_HYPERPARAMS.visionImageSize sentinel.
* Studio: revert DeepSeek OCR Image Size knob + move missing-key reset
Round 3 of the parallel-reviewer pass surfaced two issues that I had
introduced earlier in this PR's follow-ups.
- trainer.py: my prior change threaded vision_image_size into the
DeepSeek OCR collator's image_size argument. The collator's
(image_size, base_size, crop_mode) is a single preset
(Tiny / Small / Base / Large / Gundam); changing image_size in
isolation desynchronizes the per-crop pixel grid from num_queries
downstream and produces wrong token grids on documents larger than
the per-crop tile. The fix pins the collator back at the Gundam
preset and logs a clear "ignored for DeepSeek OCR" notice when the
user has selected a non-default Image Size.
- model-defaults.ts + training-config-store.ts: the round 4 fix that
reset visionImageSize when a model YAML omitted the key also fired
on same-model reloads (ensureModelDefaultsLoaded re-fires on page
refresh), wiping a value the user had just selected. The reset is
now in setSelectedModel, gated on selectedModel != previousModel,
so true model switches still clear stale values while reloads keep
the user's selection.
* Studio: extend DeepSeek OCR Image Size exclusion to MLX + frontend
Round 4 of the parallel-reviewer pass flagged that the Torch trainer
exclusion I added did not have a matching MLX guard, and that the UI
still offered the dropdown for DeepSeek OCR even though the backend
ignores it.
- worker.py: _run_mlx_training now mirrors the Torch exclusion. When
the model name matches DeepSeek OCR, vision_image_size is forced
back to None before _adapt_for_mlx_vlm sees it, so dataset images
pass through unchanged just like the Torch path. Emits a clear
status line when this happens.
- params-section.tsx: the Image Size Row is now gated on
showVisionImageSize (showVisionLora && !isDeepseekOcr) instead of
showVisionLora alone, so DeepSeek OCR users no longer see a control
that silently has no effect.
- mappers.ts: buildTrainingStartPayload sends null for vision_image_size
whenever the selected model is DeepSeek OCR, so the backend log line
about ignoring the value never fires from a UI-driven start.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten YAML import/save for vision_image_size
Two YAML-path asymmetries that could leak a stale image size into
training:
- parseYamlConfig now treats a missing training.vision_image_size as
null. Without this, importing a YAML saved before this feature (or
any config that omits the key) preserved whatever value the user had
previously set on a different model. The model-defaults reload path
still uses Object.hasOwn so same-model defaults reloads do not wipe
a manual selection; only file import normalises the missing key.
- handleSaveConfig now passes a DeepSeek-OCR-specific guard to
serializeConfigToYaml so saved YAML matches what the API mapper
actually sends. Previously a state with visionImageSize set could
emit the key even though Studio ignored it at training time for
DeepSeek OCR, and a later import for a non-DeepSeek vision model
would activate the stale value.
serializeConfigToYaml gains an optional third parameter
includeVisionImageSize defaulting to includeVisionFields, preserving
the existing 2-arg call signature for backwards compatibility.
* Studio: also reset vision_image_size when YAML lacks a training section
Round 9's parseYamlConfig normalization only fired when the YAML had a
training mapping that omitted vision_image_size. A lora-only or
logging-only YAML (or one with `training: null`) still left trainingObj
unset, the mapper saw no vision_image_size key, and the previously
selected store value persisted into the next training run.
Now an absent or null training section is synthesised as
{ vision_image_size: null } so model-defaults.ts always patches
visionImageSize back to Default on file import. Same-model defaults
reloads still preserve manual choices via the existing Object.hasOwn
gate in mapBackendModelConfigToTrainingPatch.
* Studio: unify parseYamlConfig non-object training handling
A fresh static review (Opus subagent) flagged P3-1: parseYamlConfig
only synthesised vision_image_size: null when raw.training was either
absent or a plain object missing the key. If raw.training is a scalar
or an array (malformed but still parseable), the value was passed
through unchanged, the mapper's Object.hasOwn returned false, and any
previously selected visionImageSize persisted - the same stale-state
leak the lora-only fallback was added to close.
Treat any non-plain-object raw.training (null, array, scalar) as a
malformed/missing section and reset to { vision_image_size: null }.
* Studio: tighten code comments for vision_image_size path
* Studio: tighten vision_image_size validator + restore lost comment context
Two issues surfaced by a fresh adversarial review of the validator:
1. v.strip().lstrip("+-").isdigit() let "++512" / "--256" / "+-+512"
slip past the gate, then int("++512") raised an uncaught ValueError
and Pydantic surfaced "invalid literal for int() with base 10: '++512'"
instead of the contracted "vision_image_size must be an integer or null".
2. str.isdigit() returns True for Unicode digit families (full-width '５１２',
Arabic-Indic '٥١٢', Devanagari '१०२४'), and int() coerces them, so the
value reaching the backend wasn't the ASCII the user typed.
Replaced the lstrip+isdigit pair with re.fullmatch(r'[+-]?[0-9]+', stripped),
which rejects both shapes with the precise error and accepts the documented
ones ('256', '+512', ' 1024 '). Added 8 regression test cases covering
multi-sign strings, lone sign, and the three Unicode digit families.
Also restored comment context lost in f9c39331:
- model-defaults.ts: name studio/backend/models/training.py:_check_vision_image_size
as the spec the [256, 2048] range mirrors, so a maintainer changing the
cap in one file can find the other.
- training-section.tsx: enumerate the three windows in which isDatasetImage
is null (before a check, after dataset edits, on import) so a future
maintainer doesn't simplify the gate to `isCheckingDataset`.
- worker.py: qualify the writable-ndarray comment with "when a resize is
requested" so it doesn't misadvertise the resize=None early-return.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* feat(recipes): round-trip local model variants
* feat(recipes): add local model selector
* feat(recipes): wire selector into model editors
* fix(recipes): clear stale model state on relink
* feat(recipes): load selected local models for jobs
* chore(frontend): simplify biome scripts
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(recipes): handle local selector edge cases
* fix(recipes): polish local model selector behavior
* fix(recipes): delay local model restore until terminal runs
* fix(recipes): accept resolved default gguf variants
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: add Anthropic fast_mode toggle + surface streaming refusals
Fast mode (beta `fast-mode-2026-02-01`) lets Claude Opus 4.6 and 4.7
generate output tokens up to 2.5x faster at 6x standard Opus
pricing. The toggle lives in Configuration → Provider when the
selected Anthropic model is Opus 4.6 or 4.7 and is otherwise
hidden. Backend gates the same prefixes a second time so a stale
frontend cannot make Anthropic 400 the request, and the
`fast-mode-2026-02-01` beta header is merged onto whatever other
betas the request already needed (code-execution, compaction).
Streaming refusals (`message_delta.delta.stop_reason="refusal"` on
Claude 4 models) now surface a short user-facing notice in the
assistant message before the translated OpenAI chunk emits the
existing `finish_reason="content_filter"`. Previously the chat
bubble truncated silently because the SSE stopped mid-stream with
no visible explanation. Per the upstream docs the conversation
must be reset before continuing, so the notice tells the user
exactly that.
Reference:
- https://platform.claude.com/docs/en/build-with-claude/fast-mode
- https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/handle-streaming-refusals
Tests:
- studio/backend/tests/test_anthropic_fast_mode_and_refusal.py (8 cases
pinning fast_mode pass-through on 4.6/4.7, silent drop on Sonnet /
Haiku / older Opus / None / False, and the refusal notice + finish
reason on a synthetic refusal stream).
* Studio: drop refused Anthropic turns from the next request
Anthropic's streaming-refusal guidance says the refused assistant
turn must be removed or updated before the next call -- otherwise
the safety classifier keeps refusing. The PR only added a
user-visible notice; the partial assistant output (plus the notice
itself) still rode the next request via toOpenAIMessage.
Tag the refusal turn with an HTML-comment sentinel emitted alongside
the notice. The chat-adapter checks for that sentinel in
toOpenAIMessage and returns null, so the refused turn is excluded
from outboundMessages. The notice still renders in the transcript
(HTML comments don't display), so users keep the explanation.
* Studio: filter None finish_reason entries in test helper
test_refusal_maps_to_content_filter expects only ['content_filter']
in the finish_reasons list, but the post-PR refusal path emits a
user-visible content notice chunk first. Every _content_chunk
carries 'finish_reason: None' by construction; the helper was
appending those, so the assertion saw [None, 'content_filter']
instead of ['content_filter'].
None is not a finish reason -- it's just mid-stream delta noise.
Skip None values in _finish_reasons so the helper reflects what
the test names actually claim to check. Same fix applies cleanly
to the other helper usages (pause_turn test expects [] and the
sibling stop test expects ['stop'], both unaffected).
* Studio: cover Anthropic fast-mode edge cases
Adds 19 cases on top of the 9 in test_anthropic_fast_mode_and_refusal.
The base file pins the happy path; this file fills in the cliffs:
* Dated-snapshot prefix matching: claude-opus-4-7-2026-02-01 and
claude-opus-4-6-2026-02-01 still gate fast_mode through, while
claude-opus-4-5-2025-08-01 and claude-sonnet-4-6-2026-02-01 do not.
* Strict opt-in: a future claude-opus-4-8 or claude-opus-5 does NOT
auto-enable fast_mode -- the prefix tuple must be bumped explicitly
when a new family is whitelisted upstream.
* Beta-header merge: fast_mode coexists with code-execution-2025-08-25
and compact-2026-01-12 in one comma-separated anthropic-beta header
with no duplicates and no truncation. Pins the value to the exact
fast-mode-2026-02-01 docs token so a typo would fail CI.
* Non-destruction: fast_mode=None produces byte-identical outbound
body and headers to the version that omits the argument entirely.
Same for fast_mode=False. Guarantees the upgrade path is
non-breaking on existing Anthropic streams.
* Refusal stream ordering: the user-visible notice precedes the
finish_reason chunk so a streaming UI paints text before flipping
to content_filter. Refusal sentinel emitted exactly once. Notice
rides a normal content delta chunk with finish_reason still null.
Partial assistant deltas survive before the notice.
* Provider-side refusal coverage: a refusal on Sonnet (not just Opus)
still emits the notice + sentinel + content_filter mapping, since
refusal handling is not gated on fast-mode capability.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Persist fastMode, drop refused user message on retry
Two follow-ups on #5715:
1) sanitizeInferenceParams stripped fastMode. fastMode is in
PERSISTED_INFERENCE_PARAM_KEYS but the storage sanitizer only kept
numeric fields plus systemPrompt and trustRemoteCode, so the new
toggle was silently dropped on reload and on the
/api/chat/settings round-trip. Save it the same way trustRemoteCode
is saved.
2) Refusal recovery now also drops the triggering user turn.
Returning null from toOpenAIMessage on the assistant side left the
user prompt that caused the refusal in the outbound history, so
the very next request would re-trigger the same classifier.
Anthropic's refusal-handling guidance is explicit on this: remove
the refused turn AND the user message that triggered it before
the next call. Implemented via a pre-pass that pops the trailing
user message when an assistant carries the refusal sentinel.
Typecheck clean.
* Studio: out-of-band refusal signal + fast-mode prefix/usage/pricing fixes
The text sentinel for the Anthropic refusal drop signal was spoofable:
any assistant message containing the literal
<!--studio:anthropic-refusal--> would prune the prior user + assistant
pair on the next request. Move the signal onto a separate _toolEvent
chunk that the chat adapter latches into
assistant.metadata.custom.anthropicRefusal; assistant text can no
longer control the pruner.
Tighten the fast-mode model gate (backend + frontend) to require a "-"
family boundary so claude-opus-4-70 / claude-opus-4-7b style IDs do
not get speed: "fast" on a naive startswith match.
Use survivingMessages for the image / audio attachment scan so a
refused user turn does not gate or mis-attribute the next non-refused
turn.
Propagate Anthropic usage.speed onto the OpenAI-style usage chunk and
apply the documented 6x fast-mode multiplier in the cost calculator
(stacks with prompt-cache multipliers per the docs); expose the new
multiplier on the pricing snapshot for the UI tooltip.
Tests cover the tool-event chunk shape, the prefix-collision rejects,
usage.speed propagation, the 6x pricing math, and that the visible
refusal text carries no embedded sentinel.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shorten fast-mode and refusal comments for PR #5715
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: strip orphan tool_call XML from streamed visible content
The speculative-buffer state machine in
`studio/backend/core/inference/llama_cpp.py` can slice a tool_call XML
block between the silent DRAINING path and the user-visible
content_accum, depending on when in the model's emission the BUFFERING
-> STREAMING -> DRAINING transitions fire. Three leak shapes were
observed in a 2026-05-22 sweep of 900 Qwen3.5 / Qwen3.6 GGUF runs:
Pre-fix XML leak rate: 20/900 (2.22%), concentrated 6.7% on the
larger Q8 / MTP configs:
Qwen3.6-35B-A3B Q8_0 4/60 (6.7%)
Qwen3.6-35B-A3B-MTP Q4 4/60 (6.7%)
Qwen3.5-35B-A3B Q8_0 3/60 (5.0%)
Qwen3.6-27B Q8_0 3/60 (5.0%)
The existing `_TOOL_XML_RE` only matched well-formed
`<tool_call>...</tool_call>` and `<function=...></function>` pairs, so
unterminated openings (close was DRAINED) and orphan closes (opening
was DRAINED) survived the strip and reached the user.
Fix relaxes the regex to also strip:
1. Orphan opening up to end-of-string: `(?:</tool_call>|\Z)`
2. Orphan closing tag: bare `</tool_call>` / `</function>`
Verified on the full sweep: 20/900 -> 0/900 (100% of detected leaks
eliminated). 16 unit tests in `test_tool_xml_strip.py` pin all three
leak shapes plus the well-formed cases, plus parametrised checks on
the 5 actual real-world leak samples from the sweep data.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strip tail-only </parameter> orphan + tighten regex
The 2026-05-22 gdpval sweep surfaced a 4th XML-leak shape not caught
by the earlier regex: a bare `</parameter>\n\n` at end-of-buffer (7
of 192 trials, all Qwen3.5-27B + a few Qwen3.6-27B). The model emits
the full `<tool_call><function=...><parameter=...>...content...
</parameter></function></tool_call>` envelope, the speculative buffer
DRAINS the opening tags as intended, but EOS (max_tokens cutoff)
truncates the outer `</function></tool_call>` close, leaving just
`</parameter>` as the visible tail.
We strip this ONLY when end-anchored (`\s*\Z`) so legitimate
mid-text uses (user code samples, documentation discussing the
Qwen tool-call XML shape) survive. Verified on the 192-trial
gdpval corpus: before=7, after=0.
While at it, fold the five top-level alternations into three by
sharing tag-name and prefix subgroups:
<tool_call>... + <function=\w+>... + --> <(?:tool_call|function=\w+)>...
</tool_call> | </function> --> </(?:tool_call|function)>
Semantically identical (verified by replay over the 192-trial
corpus + adversarial inputs, 0 diffs) and 1.34x faster on real
workloads. Backtracking-safety pinned by two new perf guards
(256KB '<' spam, 1000x orphan opens).
Tests: 16 -> 28 (6 new functional + 4 well-formed-vs-orphan +
2 perf guards).
* Tighten comments in XML-strip regex and tests
Code says what it does; comments were repeating it. Strip the verbose
explanations down to the WHY-only bits (engine quirk, tail-anchor
rationale, real-world source of each test sample). No code changes.
inference.py: 21 -> 12 lines around _TOOL_XML_RE
test_tool_xml_strip.py: 343 -> 259 lines (-84)
Tests: 28/28 still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: PDF / document attachments for Anthropic + OpenAI
Studio's local-GGUF chat already supports image attachments via the
`image_url` content part shape. PDFs and other documents had no
plumbing for the external-provider path: there was no normalised
content type the frontend could send that translated to Anthropic's
native `document` block or OpenAI's `input_file`.
Add a Studio-side `input_document` content part on assistant /
user messages with three shapes:
{type: "input_document",
file_data: "data:application/pdf;base64,<DATA>",
filename?: "name.pdf",
media_type?: "application/pdf"}
{type: "input_document",
file_url: "https://example.com/doc.pdf",
filename?: "doc.pdf"}
Translation:
- Anthropic Messages API: emits a `document` block with
`{source: {type:"base64", media_type, data}}` or
`{source: {type:"url", url}}`, plus an optional `title` from
`filename`. PDFs are extracted server-side by Anthropic per their
vision/document docs and counted toward input tokens.
- OpenAI Responses API: emits `{type:"input_file", file_data |
file_url, filename?}`. PDFs are extracted server-side.
Empty / unparseable `input_document` parts are silently dropped so
a malformed frontend payload can't blow up the request.
Tests:
- New `test_multimodal_document.py` with 6 cases pinning the
outbound body shape for base64 + URL inputs on both providers,
and the empty-part drop behavior on both.
- The Anthropic assertions strip the prompt-cache wrapper
(`cache_control:{type:ephemeral}` that the tail-message caching
layer adds) before comparing the document core fields, so this
test stays focused on the translation, not the caching layer.
Live verified end-to-end against both providers: a 363-byte
single-page "HELLO" PDF, base64-encoded, attached as a `document`
block to Opus 4.7 and as an `input_file` to gpt-5.5. Both models
correctly extracted the word "HELLO" from the PDF.
Follow-up (out of scope):
- Pydantic schema entry on ChatMessage.content for `input_document`
(today it rides through because ChatCompletionRequest uses
extra=allow). Will tighten when the frontend attach button lands.
- Frontend file-picker UX for non-image attachments on the external
provider path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: gate empty-content msg + skip empty data-URI payload
Gemini High + Codex P2 on PR #5689:
1. Anthropic translation appended an empty `anthropic_parts` array
when every part was dropped (e.g. user sent only an unparseable
input_document). Anthropic 400s on "messages.N.content: at least
one block is required". Skip the whole-message append when no
parts survived. The OpenAI Responses path already had the
equivalent guard, so this brings the two providers into parity.
2. `data:application/pdf;base64,` with no payload (or whitespace-only)
parses to an empty `source.data` string. Anthropic rejects that
with 400 as well. Skip the document block before constructing it.
Plus 2 new test cases pinning both behaviors:
- `test_anthropic_empty_only_document_drops_whole_message`: confirms
a turn whose only content is an unparseable input_document does
NOT make it onto the outbound `messages` array.
- `test_anthropic_empty_data_uri_payload_is_dropped`: confirms an
empty-payload data-URI is filtered out at translation time.
(Note re: gemini's other High note about adding `input_document` to
the Pydantic ContentPart union -- ChatCompletionRequest is configured
with `extra=allow` so the part rides through today. Tightening the
union belongs with the frontend attach-button PR that surfaces the
field; called out as follow-up in the PR description.)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: register input_document in ContentPart + builder
Reviewer caught that the translation code on the external_provider
side was unreachable from a real ChatCompletionRequest:
- ContentPart is a discriminated Union of (text, image_url) only, so
any `{"type": "input_document", ...}` part was rejected by Pydantic
at request parsing with a discriminator error before the helper
could see it.
- _build_external_messages in routes/inference.py only walked text
and image_url parts, so even with a permissive schema the document
parts would have been silently dropped instead of forwarded to
the per-provider translator.
Fixes:
- Add InputDocumentContentPart with optional file_data / file_url /
filename / media_type and Tag("input_document") on the Union.
- Extend _build_external_messages to pass input_document through as
a plain dict for vision-capable providers (so external_provider's
existing Anthropic `document` and OpenAI Responses `input_file`
mappers actually run) and strip them on non-vision providers.
Tests added: schema accepts input_document, builder passes it to
vision providers, builder strips it on non-vision providers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: validate file_data before preferring over file_url
Codex P2 caught that the OpenAI input_document translator treats any
truthy file_data as valid and never falls back to file_url. That
means a malformed `data:application/pdf;base64,` (empty payload) or
a whitespace-only data URI gets forwarded as `file_data=""` and
400s the whole turn, AND silently discards a perfectly recoverable
file_url on the same part.
Mirror the Anthropic-side guard onto the OpenAI Responses path:
treat any "data:" URI with no actual base64 payload as missing and
fall through to file_url. Standalone-empty data URIs (no fallback)
are dropped entirely instead of being sent to the wire.
Tests added: empty data URI + valid file_url -> file_url wins,
whitespace-only data URI + valid file_url -> file_url wins,
empty data URI without fallback -> part is dropped.
* Address review: Anthropic side also falls back to file_url on empty data URI
Codex P2 follow-up to my earlier fix: I added the empty-data-URI ->
file_url fallback to the OpenAI Responses translator but missed
the Anthropic translator, which still `continue`d on empty payloads
and discarded an otherwise valid file_url on the same part. Result:
when the frontend supplied both file_data (placeholder / broken)
AND a working file_url, Anthropic silently lost the attachment;
when the message contained only that part, the whole message could
be dropped before reaching the wire.
Mirrored the OpenAI guard: any "data:" URI with no actual base64
payload (`data:application/pdf;base64,` or whitespace-only) is
treated as missing, and the file_url branch takes over. The
all-parts-dropped guard further down already handles the
no-fallback case.
Tests added: empty data URI + valid file_url -> URL source on the
wire with the filename preserved; whitespace-only data URI + valid
file_url -> URL source on the wire.
* Address review: gate input_document passthrough to anthropic + openai
Codex P1: only `_stream_anthropic` and `_stream_openai_responses`
have explicit translation logic for input_document parts (the former
maps to {type:"document", source:...}, the latter to
{type:"input_file", file_data|file_url}). Every other provider
(gemini / mistral / kimi / openrouter / deepseek / qwen / custom)
goes through the generic /chat/completions passthrough that forwards
`messages` verbatim, so any input_document part on a non-vision
route on those providers would 400 with an unknown content_part
type.
Added `_INPUT_DOCUMENT_PROVIDERS = frozenset({"anthropic", "openai"})`
constant and gated the pass-through branch on `provider_type in
_INPUT_DOCUMENT_PROVIDERS`. Every other provider strips the part
(text content survives). Threaded provider_type through from
_proxy_to_external_provider's call site.
Tests updated: vision + provider in {anthropic, openai} still
forwards; six unmapped providers (gemini/mistral/kimi/openrouter/
deepseek/qwen) strip the part; missing provider_type strips
defensively. The existing non-vision drop test still passes.
* Fix stale web_fetch tool-version assertion after merging main
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: wire Anthropic server-side context compaction
Anthropic ships server-side context compaction as a beta
(`compact-2026-01-12`). When the rendered prompt crosses the
configured input-token threshold, Anthropic runs an extra LLM pass
that summarises older turns and the request continues against the
compacted prefix. The response carries the original top-level fields
plus a new `context_management` block (with `applied_edits`) and
`usage.iterations[]` accounting per pass.
Per the docs the feature is currently supported on Opus 4.6, Opus 4.7,
Sonnet 4.6, and Mythos preview. The minimum threshold is 50k tokens;
under-50k requests 400.
Changes:
- Add prefix gate + helper `_anthropic_supports_compaction` plus
constants `_ANTHROPIC_COMPACTION_PREFIXES`, `_ANTHROPIC_COMPACTION_BETA`,
`_ANTHROPIC_COMPACTION_TYPE`, `_ANTHROPIC_COMPACTION_MIN`.
- Add `compaction_threshold: Optional[int]` to ChatCompletionRequest
(50k ge bound, 2M le bound). Thread through `routes/inference.py`
-> `stream_chat_completion` -> `_stream_anthropic`.
- In `_stream_anthropic`, when threshold is set AND the model
accepts compaction, attach `context_management.edits[{type:
"compact_20260112", trigger:{type:"input_tokens", value:N}}]` to
the outbound body. Sub-50k values are clamped up to 50k to keep
the request well-formed.
- Refactor the anthropic-beta header builder to merge any combination
of `code-execution-2025-08-25` + `compact-2026-01-12` flags into
one header value. Unrelated betas added at the registry level still
pass through.
- Add `test_anthropic_compaction.py` with 16 cases: gate matrix
(every doc-listed model), correct body shape, threshold clamping,
beta header merge with code execution, silent no-op on unsupported
models, omitted-threshold pass-through.
Live verified end-to-end against the real Anthropic API:
`compact_20260112` accepted on Opus 4.7, response carries
`context_management.applied_edits` + `usage.iterations[]` as
documented. (The first WebFetch-summarised version of these docs
suggested `compact_20260120`; the actual API only accepts
`compact_20260112`, matching the beta-header date. Worth pinning
behind a test so a future doc update can't drift back.)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: drop ge=50_000 clamp + parse usage.iterations[]
Two reviewer follow-ups on the compaction PR:
1. Pydantic ge=50_000 on compaction_threshold was dead code.
FastAPI rejected sub-50k threshold values with a 422 before the
`max(int(...), _ANTHROPIC_COMPACTION_MIN)` clamp in
_stream_anthropic could ever fire. Relaxed the floor to ge=1 so
the in-helper clamp actually does its job; the schema comment
now explains why this is intentional. Added a regression test
that posts a value of 1 and 49_999 through the real request
schema.
2. Anthropic publishes per-iteration token counts in
`usage.iterations[]` whenever a fresh compaction has run, and
the top-level input_tokens / output_tokens cover only the
`message` iteration -- billing must add the compaction
iterations on top. Aggregate compaction iteration tokens into
`last_usage["compaction_input_tokens" / "compaction_output_tokens"]`
so the cost surface (PR 5690) can read them without re-walking
the array, and surface both figures in the closing stream
summary log. Added two tests: one that pins the aggregation on a
compacted turn and one that pins `None` when no fresh
iterations land (so re-applied compaction blocks don't double-bill).
Sourcing: https://platform.claude.com/docs/en/build-with-claude/compaction
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: round-trip Anthropic compaction blocks across turns
Codex P1: once context_management is enabled and Anthropic runs
server-side compaction mid-stream, the response carries a
`{type:"compaction", content:"<summary>"}` content block on the
assistant message. The translator only handled text_delta and
input_json_delta on content_block_delta, so the compaction block
was silently dropped. Worse, the request schema's ContentPart
discriminated Union didn't accept `type:"compaction"`, and
_build_external_messages didn't pass it through, so even a
hand-crafted assistant message carrying the block would 422 at
parse time. Net result: Anthropic re-compacted from scratch on
every subsequent turn, wasting input tokens and reasoning budget.
End-to-end backend wiring of the round-trip:
1. SSE translator. _stream_anthropic now tracks a `current_compaction`
state slot. content_block_start with type=="compaction" seeds it
(Anthropic may include the summary on the start event AND/OR
stream it via text_delta events on the same block index --
handle both). text_delta inside a compaction block routes into
the compaction buffer instead of the user-visible content
stream, since the summary is opaque internal state, not
assistant prose. content_block_stop emits a `compaction_block`
tool_event carrying the full summary so the chat-adapter can
persist it. compaction_blocks_seen is surfaced in the closing
summary log.
2. Pydantic schema. Added CompactionContentPart with Tag("compaction")
on the ContentPart Union so requests carrying the block parse
cleanly. Required `content` field with a docstring pointing at
the Anthropic docs.
3. Message builder. _build_external_messages forwards compaction
parts on both vision and non-vision paths; the per-provider
stream helper decides whether to forward to the wire (Anthropic
does; other providers ignore the part). When a non-vision route
ends up with a single text part, collapse back to a string
so providers that don't accept content arrays still get the
expected shape.
4. _stream_anthropic outbound translator. {type:"compaction"} parts
on an assistant message land on the wire verbatim. Empty/missing
`content` is skipped so a malformed stored block can't 400
Anthropic.
Tests added (5): stream emits compaction_block tool event with the
summary intact; user-visible content stream does NOT carry the
summary text; outbound body forwards compaction parts verbatim on
the next turn; Pydantic schema accepts the part; builder passes
it through on both vision and non-vision provider routes.
Frontend follow-up: the chat-adapter needs to persist the
compaction_block tool_event onto the stored assistant message so
turn N+1 includes it in payload.messages. Pinned in the PR
description.
Sourcing: https://platform.claude.com/docs/en/build-with-claude/compaction
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: gate compaction-part passthrough to Anthropic only
Codex P1: my previous round-trip change preserved {type:"compaction"}
parts on every provider route in _build_external_messages. That
meant a chat history with prior compaction state silently leaked
the Anthropic-specific block to OpenAI/DeepSeek/Mistral/Gemini/
Kimi/OpenRouter on a provider switch, where generic
/chat/completions passthrough hands the unknown content type to
the upstream API and 400s the whole turn.
Added a `provider_type` kwarg to _build_external_messages and
gated the compaction forwarder on `provider_type == "anthropic"`.
Every other value (including the legacy None for callers that
don't pass it yet) strips the part. The Anthropic stream helper
still maps it to a native `compaction` block on the wire.
Threaded provider_type through from _proxy_to_external_provider's
call site.
Tests updated: vision + provider="anthropic" still forwards; six
non-anthropic providers strip the part; missing provider_type
strips defensively; non-vision + anthropic still forwards; non-vision
+ non-anthropic collapses back to a text string.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: Persist chat history in backend storage
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address chat tombstone batching review
* fix: update desktop auth routes stub
* chat db settings storage
* chat db settings routes
* chat db settings client
* chat db settings store
* chat db settings wiring
* chat db history storage
* chat db settings migration
* chat db settings fallback
* chat db container metadata
* chat db legacy migration fixes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* chat ci auth background reads
* chat auth storage fixes
* chat migration final fixes
* chat export batch message lookup
* chat history review fixes
* chat prune sync fix
* chat settings hydration retry
* gate settings persistence
* Scope chat-history rows by subject; fix hijack, clear-confirm, hydrate race
Backend storage and routes:
- chat_threads / chat_messages / chat_settings carry a NOT NULL subject
column with composite PRIMARY KEY (id, subject). Two authenticated
identities can no longer see or wipe each other's data.
- Pre-existing rows on an existing studio.db migrate under sentinel
subject __legacy_unscoped__ via rename + rebuild + copy; single-user
installs see no behavior change.
- ON CONFLICT(id, subject) DO UPDATE ... WHERE chat_messages.thread_id =
excluded.thread_id refuses cross-thread re-parenting via upsert.
upsert_chat_message + sync_chat_messages now raise
ChatMessageThreadMismatch which the routes map to HTTP 409.
- replace_thread_messages rejects body messages whose threadId does not
match the URL thread (HTTP 400) instead of silently rewriting them.
- DELETE /api/chat requires ?confirm=true, returns row count, logs the
subject and count.
- upsert_chat_settings_merge does read + deep-merge + write inside a
single BEGIN IMMEDIATE so concurrent writers no longer drop each
other's updates. The route delegates to this helper.
- New POST /api/chat/messages:batch returns {thread_id -> messages[]}
for many threads in one HTTP call. Subject-scoped. Unknown ids return
empty lists instead of 404 so the sidebar/search caller can rebuild
atomically.
Frontend:
- chat-runtime-store: hydrate-failure catch sets settingsHydrated:true
so a transient backend blip no longer permanently disables
persistence. setParams bumps inferenceParamMutationVersions
unconditionally so a slow hydration response cannot clobber a
pre-hydrate user edit. saveSettingsPatch replaces the serial chain
with a debounced pendingPatch + deep merge; flush on beforeunload.
- chat-history-storage: clearStoredChats returns ClearStoredChatsResult
distinguishing backend / legacy / both outcomes.
listStoredChatThreadsWithMessages uses the batched fetch (one HTTP
call) instead of Promise.all per-thread; legacy Dexie fallback only
fires when the batch result is empty.
- chat-api: batchListChatMessages with graceful 404 / 405 fallback to
per-thread listChatMessages for older servers.
- chat-thread-tombstones: store {id, deletedAt} tuples with 90-day GC
and a 5000-entry cap so localStorage stays bounded. Back-compat reads
pre-fix plain strings. Adds removeChatThreadTombstones (rollback) and
clearAllChatThreadTombstones (post-legacy-purge clean-up).
- use-chat-sidebar-items: deleteChatItem tombstones synchronously
BEFORE the backend round-trip and rolls back on failure (restores
pre-PR optimistic UX). 300 ms trailing debounce on
CHAT_HISTORY_UPDATED_EVENT plus requestSeq guard so stream-time event
bursts produce at most one fetch per quiet window.
Tests:
- studio/backend/tests/pr5272_sim/ adds 64 regression tests covering
schema migration from pre-fix shape, subject scoping, cross-thread
hijack, bulk-replace mismatch, clear-confirm, concurrent settings,
unicode + 2MB content + SQL-injection-safe binding, chunking
boundary at 900 and 901 ids, batched endpoint (multi-subject + 1200
ids + per-thread order), and grep contracts for the frontend patches.
test_chat_history_storage.py updated to pass subject.
Verified locally on Linux + macOS + Windows GitHub Actions runners
(staging fork): 64 pass + 2 from the PR's own backend test on all
three OSes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop subject scoping and clear-confirm gate (Studio is single-user)
Per maintainer feedback: subject scoping, cross-thread message hijack
guard, and DELETE /api/chat ?confirm=true gate are unnecessary because
Studio is intentionally single-user (the client already shows a confirm
dialog before clear-all).
This commit reverts those backend changes and keeps only the
non-multi-user pieces from the earlier fix commit:
- studio_db.py: restored to pre-fix shape; adds upsert_chat_settings_merge
which does atomic read + deep-merge + write under BEGIN IMMEDIATE so
two concurrent slider drags cannot drop one another's updates.
- routes/chat_history.py: restored; put_settings now calls the atomic
merge instead of doing the read-merge-write across three separate
connections. Adds POST /api/chat/messages:batch to collapse the
sidebar/search rebuild from N round-trips to 1.
- frontend/api/chat-api.ts: align batchListChatMessages request and
response keys with the backend (threadIds / messagesByThreadId).
- tests/test_chat_history_storage.py: add atomic-merge concurrency test,
deep-merge nested-key test, and 901-id chunking-boundary test.
- Drop the pr5272_sim test directory (those tests covered the reverted
subject-scoping/hijack/confirm behavior).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix sidebar delete crash, keepalive on settings beforeunload flush, search rebuild race
Two correctness bugs and one perf race surfaced by a fresh code review of
the prior fix commit:
- chat-api.ts: notifyChatHistoryUpdated was declared as a non-exported
function, but use-chat-sidebar-items.ts imports it. The import would
fail tsc with TS2305 and at runtime the optimistic-delete and
delete-failure rollback paths would both throw.
- chat-runtime-store.ts + chat-settings-api.ts + chat-settings-storage.ts:
the beforeunload settings flush is now actually keepalive. Without it
the browser cancels the in-flight PUT on tab close, so the last slider
drag is silently dropped (which is exactly the case the
debounce+beforeunload combination was meant to protect against).
- use-chat-search-index.ts: rebuilds now coalesce with a 300ms trailing
debounce and discard out-of-order responses via a requestSeq guard.
Matches the sibling pattern in use-chat-sidebar-items.ts so two rapid
CHAT_HISTORY_UPDATED_EVENTs (run-start + run-end save during a turn)
cannot land with stale data winning.
- chat-thread-tombstones.ts: drop dead clearAllChatThreadTombstones with
no call sites; Dexie is never wiped so the function has no use.
* fix(studio): protect chat persistence writes
* fix(studio): align chat history clear semantics
* fix(studio): show partial chat clear feedback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): preserve chat persistence fallbacks
* fix(studio): harden chat thread persistence checks
* Preserve chat message timestamps
* Gate chat stream on history save
* Make chat thread backfill best effort
* Avoid chat message 404 probe
* Tighten chat legacy fallbacks
* chat: server-side ledger so legacy Dexie import is recoverable
The boolean localStorage sentinel
(unsloth_chat_legacy_imported_to_studio_db) made importLegacyChatsIfNeeded
non-recoverable: deleting studio.db while the browser keeps the flag
silently hides every legacy Dexie thread from the sidebar (verified by
the 3-GPU validation probe; matches the third review comment on PR
#5272). Same trap fires for browser-profile sync to a fresh machine
and any other path that wipes studio.db while keeping IndexedDB.
Source of truth moves into studio.db itself via a new
chat_legacy_import_log table keyed by legacy thread id. The ledger
disappears together with studio.db, so the next launch re-runs the
import from whatever Dexie still holds. localStorage stays as a
per-session perf hint only.
Performance, all bounded by the three new fast-paths before any
backend work:
A) localStorage hint says "imported earlier in this session" -- 0
network, ~0 ms. Covers the warm sidebar mount.
B) indexedDB.databases() reports no "unsloth-chat" DB -- 0 network,
~1 ms. Covers every new user who never had the old browser-only
Studio (the common case after launch).
C) db.threads.count() + db.messages.count() are both 0 -- 0 network,
~5 ms. Covers returning users who migrated long ago and Dexie was
never repopulated.
Only when all three miss does the code talk to the backend
(GET /api/chat/import-ledger -> diff vs Dexie -> existing import path
-> POST /api/chat/import-ledger to record what was just imported).
Per-thread tracking is enough because Dexie is read-only after this
PR; a thread's message set does not grow.
Backend deployments that predate the import-ledger routes are
handled transparently: the client treats 404/405 as an empty ledger
and re-runs the (idempotent via UPSERT) import on next launch.
Changes:
- storage/studio_db.py: new chat_legacy_import_log table (WITHOUT
ROWID, PK on legacy_thread_id) + list_chat_legacy_import_log() +
record_chat_legacy_import_log() (idempotent batch UPSERT).
- routes/chat_history.py: GET + POST /api/chat/import-ledger with the
obvious request/response models.
- frontend api/chat-api.ts: listChatImportLedger() (returns a Set for
O(1) diff) + recordChatImportLedger(), both with 404/405 fallback.
- frontend utils/chat-history-storage.ts: importLegacyChatsIfNeeded
gains three fast-paths, ledger fetch on the slow path, and writes
the ledger after a successful import. The localStorage helper is
unchanged on the surface; it just stops being authoritative.
- tests: 5 new test_legacy_import_log_* cases (empty default, record
+ list round-trip, idempotency, input dedup, empty/null ignore).
All 9 pre-existing tests still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make the legacy-import recovery actually recoverable
The previous commit added a server-side ledger to make Dexie -> studio.db
import recoverable after a studio.db wipe, but the localStorage perf hint
still short-circuited the import gate before the ledger was ever consulted.
After a wipe, the hint stayed "true" and the bulk re-import never ran -- the
ledger sat empty and only the per-thread lazy materialize-on-continue path
restored data.
Changes:
- Remove the localStorage short-circuit from importLegacyChatsIfNeeded so
the ledger is checked on every fresh tab. legacyChatImportPromise keeps
the per-session cache; the hint now only matters for the listing paths.
- Batch the slow path: one db.messages.where().anyOf().toArray() and one
batchListChatMessages() instead of 2N round-trips. At 1k threads this
drops a multi-second blocking import to a single request pair.
- recordChatImportLedger returns {accepted, inserted, supported}. The
localStorage hint is only flipped when supported is true, so old
backends (404 / 405 / 501) no longer permanently poison recovery.
- Ledger backfill: threads already present in chat_threads but missing
from the ledger now get added too, so old-FE-then-new-FE deployments
don't redo the diff every launch.
- Backend response field renamed recorded -> {accepted, inserted}.
accepted is the deduped non-empty input count; inserted is the rows
actually new (via INSERT ... RETURNING). Bounded by Field(max_length=
10_000) on the request payload.
- Storage helpers renamed: chat_legacy_import_log -> chat_legacy_imports,
record_* -> upsert_* to match the existing noun/verb conventions.
- DEXIE_DB_NAME exported from db.ts; duplicate constant in
chat-history-storage.ts removed.
- 3 new route-level tests for /api/chat/import-ledger covering the
round-trip, the (accepted, inserted) split, and the 10k payload cap.
All 18 chat-history tests pass.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: shine1i <wasimysdev@gmail.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: per-session cost calculator + /api/providers/pricing endpoint
Neither the Anthropic Messages API nor the OpenAI Responses API
reports a `cost` field on the response. Both expose detailed token
counts (input, output, cache hits, server-tool invocations); pricing
multipliers live in the provider docs. The frontend's "cost so far"
display was impossible without scraping the server log.
Land the math + a snapshot endpoint so the cost calculator can run
client-side from the existing usage chunk plumbing. The actual UI
hookup belongs in a frontend follow-up (and is gated on PR #5670's
usage-chunk emission landing so the frontend sees the usage block
in the first place).
Changes:
- New `core/inference/pricing.py` with:
- Per-MTok base pricing tables for every active Anthropic and
gpt-5.x family member. Dated snapshots inherit the canonical-id
price via prefix match so future snapshots cost the same as the
canonical id until pricing changes.
- Shared multipliers for Anthropic cache writes (5m: 1.25x, 1h: 2x)
and reads (0.1x); OpenAI cache reads (0.1x); Anthropic server
tool surcharges ($10 / 1k web_search, $0.05 / hour code_exec
beyond the 50-hour daily free tier).
- `calculate_cost(provider, model, usage)` returns a per-turn USD
breakdown plus billable token counts, with priced=False for
unknown models so the UI can still render token counts.
- `pricing_snapshot()` returns the whole table for the frontend
so it doesn't re-implement the multipliers.
- New `GET /api/providers/pricing` returning the snapshot, scoped
behind the existing auth dependency.
- New `backend/tests/test_pricing.py` with 12 cases pinning the
math against documented values: base input/output multiplication,
5m / 1h / read multipliers, default-to-5m fallback when the
breakdown is absent, web_search per-1k pricing, code_execution
per-hour pricing, dated-snapshot fallback, OpenAI cache-read
discount accounting (cached tokens subtracted from full-price
bucket and re-billed at 0.1x), unknown model graceful-degrade,
and the snapshot endpoint shape.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: verified OpenAI pricing + fix billable input double-count
Address the cost-calculator review:
- OpenAI prices were 2-6x under the actual published rates.
Cross-checked the live developers.openai.com/api/docs/pricing page
and replaced every entry. gpt-5.5 is 5/30, gpt-5.5-pro is 30/180,
gpt-5.4 is 2.5/15, gpt-5.4-mini 0.75/4.5, gpt-5.4-nano 0.20/1.25,
gpt-5.3-codex 1.75/14. Added chat-latest alias to the canonical
chat-snapshot rate. Dropped o3 / o4 / gpt-4.5 rows that are no
longer listed on the page; calculator returns priced=False instead
of silently billing at zero.
- billable_input_tokens was double-counting cached tokens for
OpenAI. Anthropic excludes cache_* buckets from input_tokens so
we add them; OpenAI folds cache_read_input_tokens into
input_tokens already, so the tooltip read 1.8M for a 1.0M bill.
Branched the math by provider and added a regression test.
Sourcing notes in the module docstring updated.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: canonical 4.5 ids, long-context tier, OpenAI tool fees
Three Codex P1 follow-ups on the cost calculator:
1. Canonical Anthropic 4.5 ids missing from ANTHROPIC_PRICING.
claude-opus-4-5 / claude-sonnet-4-5 / claude-haiku-4-5 (no date
suffix) are the ids used by backend defaults
(PROVIDER_REGISTRY['anthropic'].default_models), but the table
only had the dated forms. _lookup's prefix fallback doesn't help
because the canonical id is SHORTER than the dated key, so
str.startswith goes the wrong way and the calculator returned
priced=False + zero cost. Added the canonical aliases for
opus-4-5, sonnet-4-5, haiku-4-5, and opus-4-1.
2. OpenAI long-context tier. gpt-5.5 and gpt-5.4 cross over at
272k input tokens to a 2x input / 1.5x output rate (gpt-5.5:
$5/$30 -> $10/$45; gpt-5.4: $2.50/$15 -> $5/$22.50). Turns past
the threshold were systematically undercounted at headline
rates. Added long_context_threshold / long_context_input_per_mtok /
long_context_output_per_mtok columns and a tier-selection step
in calculate_cost; model_priced gains a "(long-context >272000)"
suffix when the higher tier applies so the tooltip can show
which rate was used. gpt-5.5-pro / gpt-5.4-pro / mini / nano /
codex have no published long-context tier today, so they keep a
single rate.
3. OpenAI server-tool surcharges. web_search is $10/1000 calls and
the hosted shell container is $0.03 per 20-minute session on the
default 1g tier (~$0.09/hr). server_tools_usd was previously
stuck at 0.0 for OpenAI even when web_search and shell tools
fired, so sessions with tool use understated cost. Added
OPENAI_WEB_SEARCH_USD_PER_1K and OPENAI_CONTAINER_USD_PER_HOUR
constants plus a parallel of the Anthropic surcharge block that
reads counts from usage["openai_tool_use"]. The SSE translator
wires the counts in a follow-up commit; the calculator is now
ready for them. pricing_snapshot also exposes both constants so
the frontend tooltip can render the per-call rate.
Existing tests updated to stay in the short-context tier where they
were testing base rates; new tests pin canonical 4.5 lookups,
long-context crossover on gpt-5.5/gpt-5.4, the absence of crossover
on mini/nano/codex, and OpenAI tool surcharges (web_search,
container hours, combined total).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: support Anthropic 1h cache TTL via prompt_cache_ttl field
Anthropic exposes two ephemeral cache pools per request: the default
5-minute pool, and a 1-hour pool selected by attaching `ttl:"1h"` to
the `cache_control` marker. 1h writes are billed at 2x base input vs
1.25x for 5m, but reads stay at 0.1x for both, so a single extra read
landing more than 5 minutes after the write pays off the premium.
Studio hardcoded the 5m pool via `cache_control: {type:"ephemeral"}`
on both breakpoints. For chats with multi-minute idle gaps (people
juggling tabs, long-running tool calls between turns), the cache
expires before the next turn and every read becomes a cache_creation,
not a cache_read -- exactly the case where the 1h pool wins.
Changes:
- Add `prompt_cache_ttl: Optional[Literal["5m", "1h"]]` to
ChatCompletionRequest. Default (None) preserves today's 5m behavior.
- Thread through `routes/inference.py` ->
`stream_chat_completion` -> `_stream_anthropic`.
- Build a shared `cache_marker` dict in `_stream_anthropic`; attach
`ttl` only when the request asks for one of the two valid values.
Unknown TTL strings are silently dropped to avoid sending malformed
markers (the upstream API would 400).
- Apply the same marker to both existing breakpoints (system block at
line 1175 and the latest-message tail at line 1198 / 1213) so the
pool selection is consistent across the whole prefix.
- Add `test_anthropic_cache_ttl.py` with 11 parametrized cases
pinning the outbound body shape: omitted -> default marker;
explicit `5m`/`1h` -> ttl field set; unknown values dropped;
caching off -> no markers at all.
Verified upstream that `cache_control: {type:"ephemeral", ttl:"1h"}`
is accepted by the Anthropic API today; no beta header required.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Relax prompt_cache_ttl to Optional[str] (Codex P1)
Declaring `prompt_cache_ttl` as `Optional[Literal["5m", "1h"]]` made
FastAPI/Pydantic 422 the request before _stream_anthropic could even
see the field. The whole point of the downstream drop-unknown-values
behaviour was to keep a stale frontend from crashing the request;
the strict Literal at the request layer defeated that.
Loosen the schema to Optional[str]; the existing in-helper guard
already restricts forwarded values to {"5m", "1h"} (everything else
is silently dropped). Test suite stays unchanged -- the bogus-value
cases in test_anthropic_cache_ttl.py already pass arbitrary strings
through and assert they are dropped before the wire.
* Address review: confirm extended-cache-ttl beta header is GA
Reviewer asked whether the 1h cache TTL still requires the
`extended-cache-ttl-2025-04-11` anthropic-beta header. Investigated:
- Live-tested api.anthropic.com on claude-opus-4-7 (2026-05-22)
with cache_control={type:"ephemeral", ttl:"1h"} and NO beta
header. Got status 200 and ephemeral_1h_input_tokens populated
on the create turn, plus cache_read_input_tokens populated on
the reuse turn.
- Cross-checked the current prompt-caching docs: no mention of
any beta header on the 1h TTL path.
Conclusion: the gate has been promoted to GA. The code already
does not send the beta header (the cache_marker dict only carries
`type`/`ttl`), so no wire change is needed. Pinned the contract
with two regression tests that assert the header is NOT on the
outbound request, and added a docstring note explaining the
investigation outcome so a future reader does not re-add it.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Daniel Han
