* feat(studio): add S3 dataset configuration foundation (#4539)
Add foundational types and configuration for S3 bucket dataset loading:
- Add S3Config type to frontend training types
- Add S3Config Pydantic model to backend training models
- Add "s3" as a DatasetSource option
- Add s3Config state and setS3Config action to training config store
- Add i18n translations for S3 configuration (English and Chinese)
This provides the type definitions and UI text for S3 integration.
Full implementation requires boto3 dependency and data loading logic.
Refs: #4539
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Wire S3 config into training pipeline and prevent secrets persistence
- Pass s3_config from request into training_kwargs so it flows to training subprocess
- Add s3Config to NON_PERSISTED_STATE_KEYS to prevent AWS secrets from being
saved to localStorage
Addresses code review feedback on PR #5951.
* Exclude S3 config from database persistence to protect secrets
Filter out s3_config (which contains secret_access_key) from the
config_json stored in training_runs table, preventing AWS credentials
from being persisted to disk.
Addresses P1 security feedback on PR #5951.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Re-raise HTTPException in start_training and defer s3 DatasetSource widening for PR #5951
* Redact s3_config from W&B run config and accept camelCase S3 credential aliases for PR #5951
* feat(studio): implement S3 dataset loading end-to-end
Builds the actual S3 loader on top of the hardened #5951 foundation,
turning the 501-gated scaffold into a working dataset source.
Backend:
- Add core/training/s3_dataset.py: lists and downloads supported dataset
files (parquet/json/jsonl/csv) from an S3 bucket to a temp dir, using
IAM-role or access-key credentials. boto3 is imported lazily (optional dep).
- Wire s3_config into UnslothTrainer.load_and_format_dataset (downloads then
reuses the existing local-file path) and thread it through worker.py.
- Replace the 501 "not implemented" gate with a boto3-availability guard so
S3 works when boto3 is present and fails clearly when it is not.
- Add boto3 to studio.txt requirements.
- Add tests/test_s3_dataset.py (8 tests) covering download/filtering,
collisions, missing-boto3, and S3Config camelCase/IAM validation.
Frontend:
- Widen DatasetSource to include "s3"; add s3_config to the training payload
type and mapper; add an S3 validation branch and selectS3Source store action.
- Add s3-config-form.tsx (bucket/region/prefix/keys/IAM toggle) reusing the
existing studio.dataset.s3.* i18n strings.
- Add a Hugging Face / Local / Amazon S3 source toggle in dataset-section;
the S3 config card replaces the dataset combobox when S3 is selected.
- Fix DatasetPreviewDialog to accept the widened DatasetSource type.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix S3 dataset loader for PR #6222
* Fix S3 dataset edge cases for PR #6222
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix S3 IAM payload handling for PR #6222
* Block multimodal S3 datasets for PR #6222
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Ash <ash@MacBook-Pro.local>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: serve DiffusionGemma GGUFs with the on-device visual decoder
* Studio: render the DiffusionGemma denoising canvas live in chat with honest stats
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden DiffusionGemma runner resolution (Windows .exe, build/bin lookup, clear stale audio flag, safe PYTHONPATH, Linux-only pdeathsig)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(studio/responses): forward chat_template_kwargs enable_thinking to chat request
The /v1/responses translation in _build_chat_request dropped
chat_template_kwargs (e.g. {"enable_thinking": true}) sent via the
Responses extra-body, so reasoning control was silently ignored.
Lift enable_thinking onto the typed ChatCompletionRequest field,
mirroring openai_chat_completions, so both the non-streaming and
streaming Responses pass-through paths honor it.
Fixes#6198
Signed-off-by: Tai An <antai12232931@outlook.com>
* Fix/adjust Responses reasoning for PR #6202
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust reasoning none for PR #6202
* Fix/adjust structured reasoning for PR #6202
* Fix/adjust responses reasoning review findings for PR #6202
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust responses reasoning follow-ups for PR #6202
* Fix/adjust think parsing gate for PR #6202
---------
Signed-off-by: Tai An <antai12232931@outlook.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: Add Tensor-Parallel llama.cpp support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: harden Tensor-Parallel fallback and GPU selection
* Studio: reconcile split-mode extras and harden tensor-split planning
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: reconcile split-mode extras in backend duplicate-load guard
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: preserve inherited non-tensor split modes on reload
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: honor cancellation in tensor fallback, preserve tensor mode on rollback, and don't raise an explicit small context
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: reconcile split-mode in reload check and strip it on tensor downgrade
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Strip --tensor-split alongside --split-mode so inherited ratios don't override the tensor planner
An inherited or stale --tensor-split in llama_extra_args was appended after
Studio's computed --tensor-split and won last in llama.cpp, re-introducing the
asymmetric-GPU OOM tensor mode is meant to prevent. Group -ts/--tensor-split
into the split-mode shadow set so it is stripped on inherit and on the layer
fallback; parse_split_mode_override still keys on the mode value only.
* Drop quantized KV for the tensor attempt and report native max context
Tensor mode aborts on a quantized KV cache, so a user with q8_0/q4_1 etc. who
enabled Tensor Parallelism silently fell back to layer split. Clear the cache
type (and strip inherited/explicit --cache-type) for the tensor attempt only;
the layer fallback re-runs with tensor off and keeps the user's choice.
Also report max_available_ctx from the native context, not an explicit small
-c, so the context slider no longer warns too early in tensor mode.
* Reconcile inherited split-mode extras in the already-loaded check
When a same-model load omitted llama_extra_args, the tensor comparison resolved
the raw (None) request and treated an inherited --split-mode tensor server as a
mismatch, forcing a needless reload. Compare using the stored extras stripped
the same way the reload strips them.
* Pass tensor_parallel through compare-mode loads
The generalized compare path loaded each GGUF without tensor_parallel, so
compare ran layer split even with the toggle on and left the settings sheet
stale. Send the toggle and hydrate the loaded state from the response, matching
the main chat and recipe load paths.
* Add --tensor-parallel flag to unsloth studio run
The headless one-liner could only reach tensor mode by passing --split-mode
tensor as a raw llama.cpp extra. Add a first-class --tensor-parallel/
--no-tensor-parallel option that sets the tensor_parallel field on the
/api/inference/load payload, forwarded through the studio-venv re-exec like the
other polarity flags. Matches the web UI toggle and the API field.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* fix: allow absolute save_directory in export paths to prevent cross-drive copy failures
The GGUF export pipeline (and all other export flows) forced every
save_directory through resolve_export_dir(), which always resolved
the path under exports_root() — typically ~/.unsloth/studio/exports/
on the system drive (C: on Windows).
When a user selected an output directory on a different drive (E:):
1. The absolute path was rejected at the Pydantic validator level.
2. Even if it got through, resolve_export_dir would re-resolve it
under C:\Users\.unsloth\studio\exports\.
3. After GGUF conversion completed on E:, the relocation step would
try to move/copy the finished files to C:, causing:
- WinError 17 (cross-drive move failure when shutil.move falls
through to a cross-filesystem copy)
- WinError 112 (disk full on C:)
Fix both layers:
- _validate_save_directory: accept absolute paths (they represent an
explicit user choice of output location).
- resolve_export_dir, resolve_output_dir, resolve_tensorboard_dir:
return absolute paths as-is instead of forcing them under the
default root. Keep the existing safety checks (null bytes, '..'
segments) and fall through to resolve_under_root for relative paths.
Fixes: https://github.com/unslothai/unsloth/issues/6082
* refactor: centralize user path validation into _resolve_user_path helper
Addresses code review feedback: the null-byte, '..', and absolute-path
checks were duplicated across resolve_output_dir, resolve_export_dir,
and resolve_tensorboard_dir. Extract a single _resolve_user_path helper
that all three delegate to.
No behavioral change — pure consolidation.
* fix: address code review — contain destructive cleanup and scope absolute paths
Address all review feedback from gemini-code-assist:
1. P1: destructive subdirectory cleanup (export_gguf)
The flattening loop in export_gguf previously rmtree'd every
subdirectory under abs_save_dir. When targeting an existing user
directory on a different drive (#6082), this could nuke unrelated
subdirectories. Now snapshot existing subdirectories before the
export and only clean up dirs created during this run.
2. P2: keep scan/read endpoints contained
Only resolve_export_dir accepts absolute paths (export is a write
path where user picks location). Reverted resolve_output_dir and
resolve_tensorboard_dir to use resolve_under_root directly — these
are used by scan/read/training endpoints that must stay contained
under their respective roots.
3. Centralization feedback
Removed the _resolve_user_path helper since it's no longer needed
with the narrowed scope. resolve_export_dir has the absolute path
logic inline with a clear docstring.
* fix: skip pre-existing subdirs in GGUF flatten loop and clean stale export intermediates
Two issues caught in code review (chatgpt-codex-connector):
1. The flattening loop moved ALL .gguf files from ALL subdirectories
into abs_save_dir, including pre-existing unrelated user subdirs.
Now skip pre-existing subdirs entirely unless they are known
export-owned intermediates (model/, model_gguf/).
2. After a failed export, known export-owned subdirectories (model/,
model_gguf/) were snapshotted as pre-existing on retry and never
cleaned up. These are now always cleaned up regardless, since they
are known intermediates created by the export pipeline.
* fix: separate write vs read export paths, guard same-dir rmtree
Three issues caught in code review (chatgpt-codex-connector):
1. P1: scan endpoint containment
resolve_export_dir was changed to accept absolute paths, but it's
also used by scan/read endpoints (routes/models.py) that must stay
contained under exports_root(). Split into:
- resolve_export_dir: contained, used by scans
- resolve_export_write_dir: accepts absolute paths, used by export
backend only
2. P1: same-directory rmtree
When a non-PEFT checkpoint's gguf_dir resolves to the same path as
abs_save_dir (user selected the checkpoint's gguf output as their
export directory), shutil.rmtree(gguf_dir) would delete the user's
chosen output directory. Now skip relocation when both paths resolve
to the same location.
3. P1: pre-existing subdir flatten loop
Reverted _EXPORT_OWNED_SUBDIRS logic — 'model/' and 'model_gguf/'
are common directory names in shared model folders and don't prove
export ownership. Now only clean up subdirs that didn't exist before
the export started.
* fix: remove dead _EXPORT_OWNED_SUBDIRS and fix _export_details for absolute paths
Two fixes from review comments:
1. Remove unused _EXPORT_OWNED_SUBDIRS declaration (leftover from
previous iteration that was intentionally removed).
2. _export_details now returns the full absolute path when the export
target is outside exports_root(), instead of truncating to basename.
Users who export to E:\ can now see the full destination path in
the success dialog.
* fix: use unique tmp dir for GGUF intermediates to avoid overwriting user dirs
When exporting to an absolute destination that already contains a
model/ subdirectory (e.g. a shared models folder), the hard-coded
model_save_path would overwrite files in that unrelated directory.
Use _tmp_model_<uuid> as the intermediate path instead, so user
directories are never touched. The tmp dir is created as a new subdir
of abs_save_dir and cleaned up by the flatten loop after GGUF files
are relocated.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix GGUF local export paths for PR #6088
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address GGUF export follow-ups for PR #6088
* Clean GGUF temp dirs on export failure for PR #6088
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust export path tests for PR #6088
* Fix/adjust export path review findings for PR #6088
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust home export path handling for PR #6088
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: Add inline confirmation (Allow/Always allow/Deny) for tool calls
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix race in tool-call confirmation gate
* Studio: gate built-in tool calls and harden the confirmation handshake
The Allow / Always allow / Deny controls only lived in the fallback tool
card, but the built-in tools (web search, python, terminal, code
execution, image generation) render with their own components and so
never showed the buttons. Those calls paused after tool_start with no way
to approve them, hanging until the 1 hour timeout. Only MCP tools, which
use the fallback renderer, actually worked.
Render the controls for every tool card by wrapping each registered tool
component (and the fallback) in thread.tsx with a shared
ToolConfirmationControls, so the gate applies uniformly.
Also make the handshake robust:
- The gate keys on a per-call approval_id minted by the backend and
echoed in tool_start, instead of session_id alone, so a stale or
concurrent confirmation can no longer resolve the wrong call.
- The approval slot is registered before tool_start is yielded, closing
the race where a fast click or an auto "Always allow" could reach the
backend before the waiter existed.
- The frontend resolves with the same session id the request was sent
with (plus the approval_id), fixing the new-thread mismatch where the
confirmation targeted a different session than the blocked stream.
- The confirm endpoint returns {resolved}; the UI keeps the buttons and
shows a retry hint until the backend confirms a match, instead of
hiding them on a failed or mistargeted post.
- The gate runs after the disabled-tool and duplicate-call checks, so a
call that will not execute is not put up for approval. A denied call is
still excluded from duplicate detection, so re-issuing and approving it
works.
- "Always allow" is scoped per session to match the backend gate.
Add backend tests for the approval registry, the SSE no-deadlock
handshake, and the loop integration (allow, deny, disabled, duplicate,
re-issue after deny).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Move "Confirm tool calls" to the Tools section
* Studio: Keep tool group open while a tool call awaits confirmation
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix tool confirmation session scope for PR #5869
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix confirmation follow-ups for PR #5869
* Apply pre-commit formatting for PR #5869
* Fix confirmation cleanup for PR #5869
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden confirmation lookups for PR #5869
* Studio: make the tool-call confirmation decision immutable
resolve_tool_decision accepted a second confirmation for the same approval_id
and overwrote slot["decision"] in the window before the waiter reads it and
pops the slot, so a duplicate or out-of-order POST could flip an Allow to Deny
(and returned a misleading resolved:true). Reject once the slot's event is
already set so the first decision wins. Adds a regression test.
* Fix/adjust tool confirmations for PR #5869
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* studio: import MCP servers from a config file
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* import config' on the add-server form
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: defensively handle MCP config imports
* fix: address MCP import review follow-ups
* fix: preserve apostrophes in Windows MCP commands
* fix: preserve apostrophe-wrapped Windows MCP args
* fix: align Windows MCP parsing with list2cmdline
* fix: preserve explicit MCP remote transport intent
* fix: trim MCP remote URLs before transport checks
---------
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Studio: report the real llama-server context window and add an opt-in overflow policy for OpenAI-compatible serving
A community report showed OpenCode failing tool calls every few minutes
against Studio's OpenAI-compatible API while the same GGUF was stable on
LM Studio. Root cause: Studio advertises the requested context length, but
llama-server can allocate less (memory-fit step on small GPUs, --parallel
slot split), so clients budget against a window that does not exist. Their
generations truncate mid tool call at the real wall (finish_reason=length
with cut JSON arguments) and eventually the prompt itself exceeds the real
window, returning a 400 that agentic clients treat as non-retryable.
Changes:
- After llama-server health, read default_generation_settings.n_ctx from
/props and adopt it whenever it is below Studio's computed context, with
a warning. The load response, status route, UI value, and the passthrough
max_tokens ceiling all become honest automatically.
- Expose context_length and max_context_length on /v1/models so clients can
budget against the enforced window.
- Accept empty role=tool content (commands with no output are routine in
agentic loops; OpenAI and llama-server both accept it) instead of a 400.
- Add context_overflow=truncate_middle (per request, or server-wide via
UNSLOTH_CONTEXT_OVERFLOW=truncate_middle): on exceed_context_size_error
the passthrough drops whole middle turn-groups (system prompt, first turn,
and recent turns kept; tool calls stay paired with their results), clips
oversized contents middle-out when group-dropping is not enough, clamps
max_tokens to the generation headroom, and retries. Default stays 'error'
with code=context_length_exceeded so clients running their own compaction
keep full control.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: allocate the requested context for real (kv-unified, fit-ctx floor)
Two launch-flag gaps caused the advertised vs allocated divergence at the
source:
- llama-server enables --kv-unified only when the slot count is auto; Studio
always passes --parallel N, which silently splits -c into per-slot windows
of -c/N. Pass --kv-unified when N > 1 so a single request can use the full
advertised window (same total KV memory, shared pool).
- with --fit on the fit step may set ctx as low as 4096; pass
--fit-ctx <requested> for explicit requests so fit offloads or fails into
the existing --fit off retry instead of silently shrinking the window.
Both flags are gated on --help capability probing so older builds keep the
current behavior, where the /props readback remains the backstop. Verified
live: -c 98304 --parallel 4 now serves per-slot n_ctx 98304 (was 24576),
48k-token requests pass through the passthrough, and the readback warning no
longer fires.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: surface the llama.cpp update affordance when MTP is disabled
When a model asks for MTP (auto on an MTP model, or forced mtp / mtp+ngram)
but it gets disabled, the load already degrades gracefully and serves without
speculative decoding. Until now the UI gave no hint why, or that an update
would fix it.
Record why MTP was dropped on the backend (spec_fallback_reason): the probe
found no mtp token (binary_no_mtp), the spawn aborted with an outdated-arch /
context-build error such as a prebuilt that predates the Gemma drafter
(binary_outdated), or the current build could not run it, e.g. a CUDA kernel
limit (runtime_error). Expose it in the inference status. In the chat
Speculative Decoding section, show a short note and, for the two update-fixable
reasons, an inline Update llama.cpp button that reuses the existing update flow.
A runtime_error gets the note without an update push, since a newer build may
not fix it.
Backend tests cover the reason being set / cleared. Frontend typechecks.
* Address review: tighten the update hint to genuinely outdated binaries
Reserve binary_outdated (which surfaces the Update llama.cpp affordance) for an
unknown-architecture abort, which proves the prebuilt predates the model;
classify the generic memory/context build failures as runtime_error, where an
update may not help. Frontend: only append the "Update llama.cpp to enable it"
sentence when an update is actually available, so the text never points at an
action the UI is not offering.
* Studio: sync detected model capabilities into models[] after load
The chat composer gates audio upload on activeModel.hasAudioInput, but
/api/models/list omits audio fields for default and active-GGUF entries
and the single chat load path never wrote the load response's
capability flags back into the store. Audio-capable models such as the
Gemma 4 GGUFs therefore never unlocked audio input in the main chat,
while the compare composer (which does sync) worked.
Add syncModelCapabilities and call it after a successful load and after
the status fetch in refresh, so the flags also survive F5 and are not
clobbered by stale catalog data.
* Studio: merge audio upload into the Add photos & files picker
Remove the separate Upload audio row from the composer plus menu and
register an AudioAttachmentAdapter in the shared attachment pipeline,
so the standard picker and drag-drop accept wav, mp3, m4a, ogg, flac
and webm directly. Gating matches images: the picker always lists
audio and models without audio input get a toast at add() time. The
50MB limit is kept and the file shows as a normal attachment chip.
On send the adapter emits an audio content part on the attachment and
findLatestUserAudioBase64 now also scans attachment content, so the
request still carries audio_base64 exactly as before.
* Studio: extract AudioAttachmentAdapter into its own module
Move the adapter out of runtime-provider.tsx so it is importable in
isolation, export the audio send-path and capability-sync helpers for
tests, and guard attachment id generation for non-secure contexts
(crypto.randomUUID is undefined over plain HTTP on a LAN, matching the
existing guard in startCompare).
* Studio: do not claim .webm by extension in the audio adapter
A video/webm file would match the .webm extension entry and route to
the audio adapter. Real audio webm (MediaRecorder output) always
reports the audio/webm MIME, so matching webm by MIME only keeps video
files out while keeping recorded audio working.
* Studio: only send audio from the newest user message
audio_base64 switches the backend onto the audio generation path
(generate_whisper_response ignores chat messages entirely and
generate_audio_input_response bypasses the normal streaming path), so
replaying audio from an older turn hijacked text-only follow-ups:
Whisper would retranscribe the stale clip instead of erroring cleanly,
and audio VLMs lost tools and streaming. Stop the scan at the newest
user message, matching the consumed-on-send semantics of the legacy
pendingAudio path. Regenerating the audio turn itself still resends
its audio since it is the newest user message in that run.
Also guard extractAudioPartBase64 against null parts in deserialized
history content.
* Studio: forward audio input to llama-server for GGUF models (#6096)
* Studio: forward audio input to llama-server for GGUF models
* Studio: harden GGUF audio input handling (multi-format decode, size cap)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: carry GGUF audio in the message list so it works with tools
* Studio: bound decoded audio length and make the soundfile decoder optional
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Handle audio attachment edge cases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate audio file picker by loaded model capability (#6142)
* Gate audio attachments by loaded model
* Use conditional spread for audio attachment adapter
* Preserve audio fallback while filtering picker
---------
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
Co-authored-by: oobabooga <oobabooga4@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: support separate-file MTP GGUF drafters (Gemma 4)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix review findings for separate-file MTP drafters
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: pair local MTP drafters by name and include them in reload dedup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: manage --model-draft in extras and reject MTP/ copies as models
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Adds a self-contained RAG stack to Studio: knowledge bases with chunked indexing, hybrid (dense + lexical) retrieval, and an automatic first-pass context inject into chat. Embeddings run through a local llama-server GGUF backend (default unsloth/bge-small-en-v1.5-GGUF) with a sentence-transformers fallback. The chat tool loop gains a search_knowledge_base tool, a per-turn re-search cap, and source citation, layered on top of the shared ToolLoopController.
* Studio: fix OpenAI- and Anthropic-compatible API spec compliance
* Studio: fix API spec-compliance gaps on passthrough and streaming paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: carry context_length_exceeded through the OpenAI passthrough error path
* Studio: count tool-schema tokens in the Anthropic server-tool stream, and small stream-handling guards
* Studio: guard message_delta usage against None and normalize developer role before proxying
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: honor max_completion_tokens on the external-provider proxy path
* Studio: forward llama-server cached_tokens into OpenAI prompt_tokens_details
* Studio: sanitize messages in count_tokens to match the /v1/messages prompt
* Studio: report max_tokens for truncated tool calls and guard null usage in metadata events
* Studio: drop the request-id middleware (headers aren't declared in either spec)
* Studio: include the required request_id field in Anthropic error bodies
* Studio: honor max_completion_tokens on the audio (TTS / audio-input) paths
* Studio: add the _effective_max_tokens helper and route all max-token sites through it
* Studio: align API compatibility edge cases
* Studio: clarify multi-choice chat support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: clarify logprobs chat support
* Studio: opt the local chat UI into the streaming usage chunk so the context bar and tok/s repopulate
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: forward seed to llama-server, and fix Anthropic server-tool stop_reason, tool_result id correlation, and parallel-tool execution cap
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: align OpenAI chat completion spec edge cases
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: align backend API compatibility tests
* Studio: honor tool caps and internal stream usage
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: coerce nullable stream usage counts
* Studio: preserve system prompts with developer messages
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: fix recipe dataset preview
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Trim and tighten code comments and docstrings across studio/ Python. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.
Normalize misplaced system-role messages in /v1/messages by hoisting their content into the top-level Anthropic system field, fixing the 422 that newer Claude Code clients trigger. Null and non-text system content is ignored rather than stringified.
Fixes#6001
* added remote MCP server support
* trim
* added tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* increased timeout
* disabling MCP chat toggle
* Fix MCP OpenAI function-name validation + cancel propagation for PR #5750
OpenAI requires function.name to match ^[a-zA-Z0-9_-]{1,64}$ before
streaming starts. The existing 64-char length check is necessary but
not sufficient: MCP servers can return tool names containing '.', '/',
spaces, etc. that would 400 the whole chat request. Validate the
composed mcp__<server_id>__<tool> name against the regex, skip + warn
on miss, and drop duplicate tool names from the same server (which
would also 400 the request as "duplicates").
Also propagate the agentic-loop cancel_event into MCP tool execution
so a /cancel POST during a long-running MCP call (e.g. GitHub MCP
search across a large repo) actually interrupts the in-flight HTTP
call instead of waiting out the 300 s timeout. The watcher polls the
threading.Event at 50 ms cadence inside the asyncio loop (matches
routes/inference.py's existing cancel-watcher cadence) and races
against the call task with asyncio.wait FIRST_COMPLETED.
Tests added:
- test_mcp_specs_skip_invalid_openai_function_names: drops bad chars
- test_mcp_specs_skip_empty_tool_name
- test_mcp_specs_drops_duplicate_names
- test_call_tool_sync_respects_pre_set_cancel_event
Also fix test_desktop_auth.py's router stub that listed every existing
router but missed mcp_servers_router, so importing main.py fails after
this PR adds it to routes/__init__.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* PR #5750 round 2: OAuth cleanup on delete/url-change + mcp_enabled standalone
Round 2 of cross-platform validation surfaced two more P1 findings:
1. OAuth tokens never get cleared. fastmcp keys tokens by MCP URL, not by
server row, and delete / URL change / use_oauth toggle only updated
the SQLite row. Re-registering the same URL would silently reuse the
old account's credentials. Adds clear_oauth_tokens_async() in
mcp_client.py and calls it from the delete + put route handlers when
the row had use_oauth=True and either the URL changes or OAuth is
turned off.
2. mcp_enabled=true was ignored unless the caller also sent
enable_tools=true. The frontend always sends both together so the UI
path was fine, but a direct API caller sending only mcp_enabled would
silently get no MCP tools, which contradicts the field's documented
"append tools from every enabled MCP server" behavior. Loosens the
use_tools gate in both the GGUF and safetensors paths so mcp_enabled
opens the tool loop on its own; when the caller did not also opt
into built-ins, the built-in list starts empty.
Tests added:
- test_clear_oauth_tokens_async_no_op_safe
- test_delete_server_calls_oauth_cleanup_when_oauth_was_on
- test_delete_server_skips_oauth_cleanup_when_oauth_off
- test_update_server_clears_oauth_on_url_change
- test_update_server_clears_oauth_when_oauth_disabled
26 backend MCP tests pass; full studio/backend suite 1710 passed locally.
Cross-platform CI (Linux, macOS, Windows) green on staging fork.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* PR #5750 round 3: reject null bool updates + /test surfaces 400
Round 3 of cross-platform validation:
1. PUT /api/mcp/servers/<id> would 500 with TypeError when the body
explicitly set is_enabled or use_oauth to null. Pydantic accepts
None for an Optional[bool] and _changes_from_payload then passed
None into mcp_servers_db.update_server, which int(None)d. Reject
explicit null at the validation layer with 400 instead.
2. POST /api/mcp/servers/test caught HTTPException under
"except Exception", so an invalid URL came back as HTTP 200 with
{"ok": false, "error": "400: ..."} instead of a real 400. The
create + update paths return 400 for the same input. Move
validation outside the transport try/except so it surfaces 400.
Tests added:
- test_changes_from_payload_rejects_null_is_enabled
- test_changes_from_payload_rejects_null_use_oauth
- test_test_endpoint_surfaces_url_validation_as_400
* PR #5750 round 4: hyphenated MCP tool names + empty-tool-list gate
Round 4 surfaces two more interaction bugs between the new MCP path
and existing safetensors tool plumbing:
1. OpenAI accepts ^[a-zA-Z0-9_-]{1,64}$ for function.name, and round 1
widened the MCP regex to that set, so MCP tools can now be advertised
as `mcp__srv__list-issues`. But the XML tool-call parser in
tool_call_parser.py used `\w+` (no hyphen), so the model could call
the tool but Studio could not parse the call. Same in
routes/inference.py's `_TOOL_XML_RE` stripper, which would leave
hyphenated tool-call XML in the visible content. Both regexes now
use `[\w-]+`.
2. safetensors_agentic treats `tools=[]` as "allow all" (documented
contract, exercised by test_empty_tools_list_does_not_enforce_allowlist).
When a caller sends `enable_tools=true` + `enabled_tools=[]` +
`mcp_enabled=true` and MCP discovery returns 0, the resolved tool
list is genuinely empty and built-in tools (web_search / python /
terminal) could execute via the model's emitted call. Fix at the
route gate instead of breaking the documented contract: set
`use_tools=False` when the resolved list is empty, in both GGUF and
safetensors paths. Existing callers who omit `enabled_tools` still
get ALL_TOOLS and are unaffected.
Tests added (32 total):
- test_tool_xml_parser_handles_hyphenated_function_names
- test_tool_xml_strip_handles_hyphenated_function_names
- test_safetensors_agentic_empty_allowlist_still_means_allow_all
(documents the contract round 4 preserved)
1716 passed locally; cross-platform CI on staging fork still green.
* PR #5750 round 5: GGUF allow-list + CLI policy + hyphenated params + cancel race
Round 5 of parallel-reviewer aggregation surfaced six additional
findings; five are real and fixed here:
1. Hyphenated MCP parameter names (`<parameter=issue-number>`) were
dropped by the XML parser's `\w+` regex. Extended to `[\w-]+` in
both core/inference/tool_call_parser.py and core/tool_healing.py.
The latter is GGUF's own copy of the parser/strip patterns and was
missed by round 4.
2. core/tool_healing.py's `strip_tool_call_markup` still used
`<function=\w+>` so hyphenated MCP tool-call XML leaked into the
GGUF visible content even after round 4 fixed the shared parser.
3+4. `mcp_enabled` re-opened the tool loop even when the operator
passed `unsloth run --disable-tools` (CLI policy False). Round 2's
`(_tools_on or payload.mcp_enabled)` gate ignored the raw process
policy. Now reads `state.tool_policy.get_tool_policy()` and gates
mcp_enabled on `_cli_policy is not False`. Applied to both GGUF
and safetensors paths.
5. GGUF's agentic loop called `execute_tool(tool_name, ...)` without
checking the model-emitted name against the per-request tool list,
while the safetensors loop already enforces this. Added the same
allow-list check so a model that hallucinates a filtered MCP name
or a built-in the caller opted out of returns "not enabled" instead
of executing.
Bonus P2 fixes:
- `call_tool_sync` now checks `cancel_event.is_set()` BEFORE
creating the call task, so a pre-set cancellation does not open
the HTTP transport.
- `clear_oauth_tokens_async` moved the OAuth import + construction
inside the protected try block; a fastmcp.client.auth load error
used to escape and 500 the delete / update route.
NOT fixed (verified false or out of scope):
- finding #10 "structured_content vs structuredContent": fastmcp's
CallToolResult dataclass uses snake_case (verified live against
structured-only tool result; fields are
`dict_keys(['content', 'structured_content', 'meta', 'data', 'is_error'])`).
- finding #11 "asyncio.run from running loop": call_tool_sync is
invoked from `asyncio.to_thread` worker threads which have no
event loop; asyncio.run() is safe there.
Tests added (37 total): hyphenated param names, tool_healing strip,
GGUF allow-list gate, cancel pre-set short-circuit, OAuth cleanup
constructor-error swallowing. 1721 passed locally, no regressions.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Studio: add Gemini provider with web_search, code_execution, prompt caching, and Nano Banana image generation
Wires Google's native Gemini API into Studio's external-provider stack
so users can pick gemini-2.5-pro / gemini-2.5-flash / gemini-2.5-flash-image
(Nano Banana) alongside the existing OpenAI / Anthropic / OpenRouter
providers. Gemini does not speak OpenAI Chat Completions on its primary
endpoint; the new `_stream_gemini` async generator translates between
the two shapes the same way `_stream_anthropic` handles the Messages API.
Backend:
- New `_stream_gemini` translator in external_provider.py. Converts
OpenAI messages -> Gemini `contents` + `systemInstruction`; maps
generationConfig (temperature / topP / topK / maxOutputTokens);
forwards `tools: [{googleSearch: {}}]` for web_search and
`{codeExecution: {}}` for code_execution; passes `cachedContent`
through for prompt caching; sets `responseModalities=[TEXT, IMAGE]`
for Nano Banana image generation.
- Translates streamed `GenerateContentResponse` SSE frames back into
OpenAI chat.completion.chunk frames (text deltas, function_call ->
tool_calls deltas, inlineData -> image_b64 tool_end envelope, usage
chunk before [DONE]).
- Registry entry switched to native base URL
`https://generativelanguage.googleapis.com/v1beta` with
`openai_compatible: False` and the `x-goog-api-key` auth header.
Model lineup curated to current 2.5 / 2.0 family + Nano Banana.
Frontend:
- Provider-capability matrix: Gemini supports temperature, top_p, top_k,
presence_penalty (matches generationConfig); min_p / repetition_penalty
hidden because the API does not accept them.
- `providerSupportsBuiltinWebSearch` / `providerSupportsBuiltinCodeExecution`
/ `providerSupportsBuiltinImageGeneration` extended for Gemini.
- Prompt caching toggle now also lit on Gemini.
Tests:
- 21 new tests in `test_gemini_provider.py` using httpx.MockTransport.
Cover request body shape conversion, URL/header wiring, web_search
forwarded as googleSearch, function-call translation both directions,
prompt caching passthrough, image generation emitting image_b64,
grounded-search citations -> tool_end, finish_reason mapping, and
vision data URL -> inlineData translation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: forward presence_penalty to Gemini and recover function name from tool_call_id
Two follow-up fixes for the Gemini provider:
* Thread presence_penalty into _stream_gemini and set
generationConfig.presencePenalty when non-zero. The OpenAI-side
capability matrix already exposes the slider for Gemini, so the
value was being collected and silently dropped on the way out.
* When an OpenAI role=tool message omits 'name' and only carries
'tool_call_id', recover the function name from the matching
functionCall on the prior assistant turn. Gemini 400s on an empty
functionResponse name.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: surface Gemini code execution parts as code_execution tool events
The Gemini stream parser only handled text/functionCall/inlineData
parts, so when the user toggled the Code pill on a Gemini model the
sandbox output (executableCode + codeExecutionResult parts) was
dropped on the floor while adjacent text reached the UI. Reviewers
flagged this as the headline feature being silently broken.
Translate both parts into the existing code_execution tool envelope
that CodeExecutionToolUI already consumes for OpenAI / Anthropic:
* executableCode -> tool_start with kind=code_execution and the
source code under arguments.code. We mint a tool_call_id and
stash it so the matching result block can pair to it.
* codeExecutionResult -> tool_end on that id with the stdout under
result. Non-OK outcomes (OUTCOME_FAILED / OUTCOME_DEADLINE_EXCEEDED)
are prefixed onto the text so the failure is visible.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: native Gemini model catalog, function-call ids, and honest cache claim
Three follow-ups to the Gemini provider PR after the codex pass:
* list_models() now translates Gemini's native /v1beta/models
payload ({models[{name, baseModelId, displayName,
supportedGenerationMethods}]}) into the OpenAI-compatible shape
Studio expects. Without this the picker stayed empty for Gemini
and fell back to hardcoded defaults. Embedding-only models are
filtered out.
* Forward the OpenAI tool_call id into Gemini's functionCall.id
and mirror it onto functionResponse.id. Two parallel calls to
the same function name can now be paired unambiguously on the
follow-up turn.
* Drop Gemini from the prompt-caching capability set. The wire
flow requires a separate cachedContents POST first and the
boolean Studio emits today is a no-op; the toggle should not
advertise a feature it cannot apply. Leaves a pointer to the
docs for the eventual two-step orchestration.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: distinct tool_calls index per emitted Gemini function call
Codex flagged that the Gemini stream parser hardcoded
tool_calls[0].index to 0 on every emitted functionCall. OpenAI
reassemblers key tool_calls by index when joining deltas, so two
parallel function calls in one assistant turn collapsed onto a
single slot and the second call's arguments overwrote the first.
Track the running count via len(emitted_function_call_ids) - 1
and emit it as the per-call index. The dedupe guard above (skip
when fc_id already in the set) means the index is monotonic and
stable for the lifetime of the stream. Regression test asserts
[0, 1] across two parallel calls in one candidate parts list.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: surface Gemini 3.5/3.1/3 + Nano Banana 2/Pro and plumb thinking budget
`gemini-2.0-flash` / `gemini-2.0-flash-exp` were retired by Google in 2026
(`/v1beta/models/gemini-2.0-flash:streamGenerateContent` returns HTTP 404
"no longer available to new users"), and the picker had nothing past the
2.x family. Verified against the live ListModels catalog: drop the retired
ids from `default_models` + allowlist and surface the chat-capable
3.5 / 3.1 / 3 families plus the Nano Banana image trio.
Also plumb `enable_thinking` / `reasoning_effort` into Gemini's
`generationConfig.thinkingConfig`. Without this, Gemini 3.5 Flash,
gemini-pro-latest, and the 3.x previews silently spend the caller's
`max_tokens` budget on hidden "thoughts" before emitting any visible
answer -- the chat shows a truncated stub like "The capital of" and
streams stop. Mapping:
- enable_thinking=False / reasoning_effort=none -> thinkingBudget=0
(Flash tier; Pro tier coerces to a small positive budget because
the API 400s on 0 with "This model only works in thinking mode")
- minimal/low/medium/high -> 512/2048/8192/24576 budget tokens
- max/xhigh -> -1 (dynamic)
- default (neither knob set) -> thinkingConfig omitted, model decides
Frontend `getExternalReasoningCapabilities` now surfaces a
`reasoning_effort` picker for every Gemini chat id (Pro tier hides the
"none" option; image-tier ids stay knob-less). Adds 6 unit tests
covering Flash/Pro effort mapping, the off-toggle coercion on Pro,
default omission, and the nano-banana-pro-preview alias routing
through the image modalities path. 28 -> 34 tests in
`test_gemini_provider.py`, all green; full backend suite still passes
(1459/1460; the unrelated test_help_output flake is pre-existing and
not in any file this PR touches).
Live verification against generativelanguage.googleapis.com on
2026-05-24 with `_stream_gemini` directly:
text gemini-3.5-flash single PASS multi PASS
text gemini-3.1-pro-preview single PASS multi PASS
text gemini-3.1-flash-lite single PASS multi PASS
text gemini-3-pro-preview single PASS multi PASS
text gemini-3-flash-preview single PASS multi PASS
text gemini-2.5-pro single PASS multi PASS
text gemini-2.5-flash single PASS multi PASS
text gemini-2.5-flash-lite single PASS multi PASS
text gemini-flash-latest single PASS multi PASS
text gemini-flash-lite-latest single PASS multi PASS
text gemini-pro-latest single PASS multi PASS
image gemini-2.5-flash-image PASS (1082 KB png returned)
image gemini-3.1-flash-image-preview PASS (Nano Banana 2)
image gemini-3-pro-image-preview PASS (Nano Banana Pro)
tool web_search PASS
tool code_execution PASS
-> 16/16 e2e through the actual ExternalProviderClient code path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten Gemini provider after review (PR #5720)
Fixes a batch of bugs surfaced by a second-pass review on top of the
3.5/3.1/3 + Nano Banana 2/Pro additions in c6724dbd.
Backend (external_provider.py):
- Constructor normalises legacy /v1beta/openai base URLs to /v1beta so
Gemini providers saved before the native switch keep working without
a manual re-config.
- Skip thinkingConfig, googleSearch, and codeExecution on image-tier
models (-image / nano-banana). The image responseModalities path is
mutually exclusive with text-tool wiring and stale UI state would
otherwise 400 the turn.
- _PRO_THINKING_PREFIXES now includes gemini-3.5-pro and uses anchored
prefix matching (exact id or "<prefix>-...") so the image-tier
gemini-3-pro-image-preview cannot accidentally match the pro guard.
- Gemini 3 functionCall thoughtSignature is round-tripped through the
tool_calls envelope via extra_content.google.thought_signature on
emit, and replayed as a sibling of functionCall on the next request.
- finishReason swaps STOP -> tool_calls when any functionCall was
emitted on the same turn so OAI clients trigger tool execution
(matches the OpenAI Chat Completions contract).
- usageMetadata.thoughtsTokenCount is rolled into output_tokens and
surfaced on output_tokens_details.reasoning_tokens so total_tokens
reflects the full billable spend instead of dropping the hidden
reasoning slice.
Registry (providers.py):
- Drop gemini-3-pro-preview from default_models. Google shut it down
on 2026-03-09 and auto-redirects to gemini-3.1-pro-preview; we
surface the canonical id only.
- Add model_id_deny_exact = ("gemini-3-pro-preview",) so the live
ListModels fetch does not re-surface the redirect alias.
Route schema (models/inference.py):
- enable_prompt_caching widened to Optional[Union[bool, str]] so the
/v1/chat/completions caller can pass a Gemini cachedContent resource
name (e.g. cachedContents/abc123). Without this widening _stream_gemini
s string cachedContent passthrough was unreachable from the public
route (bool_parsing 422). stream_chat_completion signature mirrors.
Frontend (provider-capabilities.ts, chat-page.tsx, chat-adapter.ts):
- providerSupportsBuiltinImageGeneration now also recognises
nano-banana ids (nano-banana-pro-preview was hidden from the image
pill before).
- providerSupportsBuiltinWebSearch takes the model id so Gemini image
models hide the Search pill (mirrors the backend skip).
- providerSupportsBuiltinCodeExecution uses the same isGeminiImageModel
guard for nano-banana ids.
- GEMINI_THINKING_PRO_PREFIXES gains gemini-3.5-pro; gemini-3-pro
tightened to gemini-3-pro-preview to avoid the image-id overlap.
- Updated 3 callers of providerSupportsBuiltinWebSearch to thread the
selected model id through.
Tests (test_gemini_provider.py): 34 -> 42, all green
- test_image_models_skip_thinking_config
- test_image_models_drop_text_only_tools
- test_gemini_35_pro_recognized_as_pro_thinking
- test_legacy_openai_base_url_normalized
- test_finish_reason_swaps_to_tool_calls_when_function_call_emitted
- test_thought_signature_round_trips_into_gemini_function_call
- test_thought_signature_emitted_in_tool_call_delta
- test_usage_chunk_includes_thoughts_tokens
Verification:
- Backend pytest 1518/1519 passing (one unrelated Qwen3.5 flash-attn
test fails on main as well; nothing in this PR touches that path).
- Frontend npx tsc -b clean.
- Live e2e 16/16 against generativelanguage.googleapis.com through the
patched _stream_gemini code path (all 11 chat models single + multi
turn, all 3 image models returned image bytes, web_search and
code_execution tools both emit the expected envelope).
- Live /api/providers/models against the patched backend surfaces 16
ids (gemini-3-pro-preview correctly filtered via deny_exact).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: address second-pass review findings on Gemini (PR #5720)
Round-2 reviewer.py flagged a phantom web_search card on image
turns (12/12 reviewers), route-layer stripping of tool_calls /
tool_call_id / name, an over-narrow image-mode tool guard, and
silent safety blocks. This patch fixes all four.
Backend (external_provider.py):
- web_search_active is now derived from the outbound tools_array
(whether googleSearch was actually forwarded), not the raw
enabled_tools intent. Image-mode turns dropped the tool above so
the inbound stream no longer emits a phantom "search complete"
tool_start / tool_end on those turns.
- text_tools_allowed now uses is_image_model (covers both `-image`
/ `nano-banana` picker models AND text models that requested
`image_generation` via enabled_tools). Verified against the live
Gemini API which rejects both googleSearch and codeExecution
alongside responseModalities=["TEXT","IMAGE"] with explicit 400s
("Search as tool is not enabled for this model", "Code execution
is not enabled for this model").
- promptFeedback.blockReason is surfaced as a 400 content-filter
error chunk instead of returning an empty successful assistant
response. The streaming loop closes the response before exiting.
Route (routes/inference.py):
- _build_external_messages now propagates tool_calls (assistant),
tool_call_id, and name (tool result) through every code path
(string content, multimodal content, non-vision fallback). Without
this Gemini 3 function-call round trips lost their thoughtSignature
+ tool_call_id at the route boundary, and functionResponse.name
arrived empty on the second turn.
- Assistant messages with content=None and tool_calls populated are
preserved as a synthetic empty-string content turn so the
Gemini translator can rebuild the functionCall part.
Tests (test_gemini_provider.py): 42 -> 45, all green
- test_image_models_suppress_phantom_web_search_card
- test_image_generation_tool_drops_text_tools
- test_prompt_feedback_block_reason_surfaces_as_error
Verification:
- Backend pytest 1736 / 1736 (the two pre-existing unrelated fails
on main, test_help_output and Qwen3.5 flash-attn pin, are skipped).
- Frontend npx tsc -b clean.
- Live e2e 16/16 against generativelanguage.googleapis.com:
11 chat models single + multi turn, 3 image models returning
image bytes, web_search and code_execution both PASS.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix third-pass Gemini findings (PR #5720)
Round 3 review follow-ups:
Backend (studio/backend/core/inference/external_provider.py):
- Close response AND aiter_lines iterator in a finally so normal,
prompt-block, and cancellation exits all clean up (eliminates the
RuntimeWarning about aclose never being awaited).
- Pair the synthetic web_search tool_start with a tool_end on the
promptFeedback.blockReason path so the UI does not leave a stuck
"searching..." spinner after the error toast.
- Preserve native id and thoughtSignature on executableCode and
codeExecutionResult tool events under google.native_part, and pair
the tool_end on the code-exec id so multi-turn code-execution
replays do not lose Gemini-required history.
- Carry part-level thoughtSignature on text deltas via
delta.extra_content.google.thought_signature and on inline image
tool_end via google.thought_signature so Gemini 3 image editing
and tool turns round-trip the signature on the next request.
- Guess remote image_url MIME from the URL path so PNG / WebP / GIF
inputs are not silently relabeled as JPEG.
- Roll usageMetadata.toolUsePromptTokenCount into translated input
tokens and surface thoughtsTokenCount as
completion_tokens_details.reasoning_tokens in _build_usage_chunk.
- Only normalize the Google-hosted /v1beta/openai legacy base URL;
custom proxies whose paths happen to end in /openai are left
untouched.
- Forward ChatCompletionRequest.tools and tool_choice through
stream_chat_completion into _stream_gemini, translating to
tools[].functionDeclarations and toolConfig.functionCallingConfig.
Frontend:
- chat-adapter: when Gemini image-generation is enabled for the turn,
also disable Search and Code so the request, builder, and active
pills agree with what the backend actually sends (the backend
already strips text tools when image_generation is in enabled_tools).
- chat-adapter: consume OpenAI-shape delta.tool_calls chunks so
Gemini function-call deltas without text surface as tool-call parts.
- shared-composer: disable Search and Code pills while Gemini image
mode is active so the UI matches the request.
Tests (studio/backend/tests/test_gemini_provider.py): adds coverage
for proxy base-url gating, remote image MIME inference,
toolUsePromptTokenCount, reasoning_tokens propagation, prompt-block
web_search tool_end pairing, native code-exec id/thoughtSignature
metadata, inline image thoughtSignature, text-chunk extra_content,
OpenAI tools/tool_choice translation, and image-model tool drop.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: Gemini 3 thinkingLevel + image-model Search grounding (PR #5720)
Gemini 3.x migrated to a string `thinkingConfig.thinkingLevel`
(MINIMAL/LOW/MEDIUM/HIGH) and rejects `thinkingBudget`+`thinkingLevel`
in the same request. Gemini 3 also cannot turn thinking fully off, so
the lowest position is "minimal" (Flash) or "low" (Pro rejects
"minimal").
- external_provider._stream_gemini: split thinking translation by
family. Gemini 3.x (3 / 3.1 / 3.5 + gemini-pro-latest /
gemini-flash-latest / gemini-flash-lite-latest) emits
thinkingConfig.thinkingLevel; effort none/off coerces to "low" on
Pro and "minimal" on Flash. Gemini 2.5 stays on thinkingBudget.
- external_provider._stream_gemini: allow `tools: [{googleSearch: {}}]`
on the Gemini 3 image family (gemini-3-pro-image-preview,
gemini-3.1-flash-image-preview, nano-banana-pro). Google's docs
document Search grounding on these. codeExecution stays blocked
on image mode (still mutually exclusive with responseModalities).
- provider-capabilities.ts: mirror the Gemini 3 effort ladders in
resolveGeminiReasoningCapabilities (Pro: low/medium/high; Flash:
minimal/low/medium/high; 2.5 Flash keeps the off-position).
- provider-capabilities.ts: providerSupportsBuiltinWebSearch now
returns true on the documented Gemini 3 image models so the pill
is reachable; older image ids (gemini-2.5-flash-image) still hide.
Tests: splits the existing thinkingBudget cases by family (Gemini 3
checks thinkingLevel; Gemini 2.5 keeps thinkingBudget), adds positive
googleSearch coverage for Gemini 3 image models and negative
googleSearch coverage for legacy image models.
References:
- https://ai.google.dev/gemini-api/docs/thinking
- https://ai.google.dev/gemini-api/docs/gemini-3
- https://ai.google.dev/gemini-api/docs/models/gemini-3-pro-image-preview
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: attach Gemini code_execution inline images to the code card (PR #5720)
When a text Gemini turn wires codeExecution and the sandbox produces a
matplotlib plot, the inline image part ships right after the
codeExecutionResult. Previously this surfaced as a separate empty
image_generation card. Track the most recent code_execution
tool_call_id + result text and, when an inline image follows with
code_execution active, emit a second tool_end on the same id that
appends the image as a data: URI under the `__IMAGES__:` marker the
chat-adapter already understands.
Image-picker turns (`-image` / `nano-banana`) keep the standalone
image_generation envelope so Nano Banana outputs render the same way.
Tests: covers the merged code-execution card emission with no
standalone image_generation event when code_execution is the active
tool.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix fourth-pass Gemini findings (PR #5720)
Round 4 review follow-ups:
Backend:
- `_is_openai_compatible` + `_auth_headers` detect Gemini connections
pointed at a custom OpenAI-compatible proxy (non-Google host whose
path ends in `/openai`) and route them through the OpenAI-compat
surface with `Authorization: Bearer ...` instead of the native
`_stream_gemini` translator + `x-goog-api-key`. Google-hosted Gemini
keeps the native dispatch path it migrated to in this PR.
- `_stream_gemini` thinkingLevel handling for Gemini 3 Pro now coerces
both "minimal" and "medium" effort to "low" / "high" respectively
(Pro tier only accepts low/high per
https://ai.google.dev/gemini-api/docs/thinking).
- `providers.py` `default_models` restores the advertised
`gemini-3.5-pro` and the rolling `gemini-pro-latest` /
`gemini-flash-latest` / `gemini-flash-lite-latest` aliases that the
allowlist already admits.
Frontend:
- chat-adapter: lean on `providerSupportsBuiltinWebSearch` (which
already encodes the Gemini 3 image-model Search allowance) instead
of blanket-disabling Search whenever Gemini image mode is active.
Code execution stays blocked because Gemini image mode rejects it.
- shared-composer: mirror the same gate -- only the Code pill is
unconditionally disabled in Gemini image mode; the Search pill is
driven by `supportsBuiltinWebSearch`.
- provider-capabilities: Gemini 3 Pro reasoning levels now expose only
"low" and "high" (no Medium pill) to match the API.
Tests: covers the Gemini 3 Pro medium / minimal coercion, the custom
proxy OAI-compat dispatch + Authorization Bearer auth, and the
native-vs-proxy detection. Also closes the mocked httpx.AsyncClient
inside the test event loop so the Python 3.13 `aclose was never
awaited` warning no longer fires.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix fifth-pass Gemini findings (PR #5720)
Round 5 review follow-ups:
Backend:
- `_is_openai_compatible` + `_auth_headers` now treat ANY non-Google
Gemini base URL as OpenAI-compat (LiteLLM / custom OAI gateways /
OpenAI-compat vLLM routers), not just paths ending in `/openai`.
Pre-existing saved Gemini proxies on `/v1` keep working.
- Gemini 3 thinkingLevel coercion narrowed to the documented
inconsistencies: only "minimal" is coerced to "low" on Pro tier.
"medium" passes through (Gemini 3.1 Pro accepts it per
https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/3-1-pro).
- `_stream_gemini` only flips `responseModalities=[TEXT,IMAGE]` when
the selected model is image-capable. A stale
`enabled_tools=["image_generation"]` on a text model is silently
dropped instead of producing an invalid Gemini request.
- `_stream_gemini` validates the model id against
`[A-Za-z0-9._-]+` before URL interpolation so a model like
`../cachedContents/x` cannot redirect the request to an unintended
endpoint with the configured API key attached.
- Empty-text Gemini parts that still carry `thoughtSignature` emit a
content-free delta with `extra_content.google.thought_signature` so
Gemini 3 turns that end with a signature-only fragment do not lose
the replay state.
- ConnectError / ReadTimeout / generic HTTPError paths in
`_stream_gemini` now close the synthetic web_search tool_start
with a matching tool_end before the error chunk so the UI does not
leave a stuck "searching..." card on transport failure.
- `providers.py` default_models drop the non-existent
`gemini-3.5-pro` (Google launched only `gemini-3.5-flash` at
I/O 2026; Pro tier remains `gemini-3.1-pro-preview`).
- `routes/inference.py` only forwards `payload.top_k` when the caller
explicitly set it on the request (Pydantic `model_fields_set`).
Omitted top_k stays omitted, restoring the pre-PR behavior where
Gemini uses its server default.
- `ChatCompletionRequest.enable_prompt_caching` adds a `mode="before"`
validator that coerces the canonical string literals "true"/"false"
back to bool so historical opt-out callers keep working after the
field widened to `Union[bool, str]` for Gemini cache resource names.
Frontend:
- `providerSupportsBuiltinWebSearch` / Code / Image now accept the
saved connection `baseUrl` and return false for custom OAI-compat
Gemini proxies. Backend skips `_stream_gemini` for those bases, so
native tool envelopes never reach them; hiding the pills keeps the
request, builder, and UI consistent.
- `provider-capabilities.ts` Gemini 3 Pro effort ladder restores
`["low", "medium", "high"]` to match Google's documented levels.
- Call sites in `chat-page.tsx` and `chat-adapter.ts` pass through
`provider.baseUrl` so the proxy gate fires.
Tests: covers Gemini 3 Pro medium pass-through, custom proxy dispatch
on `/v1` and `/openai` bases, path-traversal model id rejection,
top_k omission when not explicit, text-model image_generation drop,
empty-text + thoughtSignature surfacing, and
enable_prompt_caching string coercion.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: fix sixth-pass Gemini findings (PR #5720)
Round 6 review follow-ups:
Frontend:
- chat-adapter `delta.tool_calls` accumulates fragments by `id` /
`index` instead of pushing a new tool-call card per chunk. The
standard OpenAI Chat Completions stream contract sends `id`/`name`
on the first chunk and partial `function.arguments` on subsequent
chunks; our previous handler parsed each fragment as a standalone
tool call. Local llama.cpp and OAI-compat providers that stream
fragments now reassemble into a single function-call part.
- chat-adapter also preserves `extra_content` on streamed tool-call
deltas so Gemini 3 `thoughtSignature` survives to the next turn.
- provider-capabilities Gemini 3 Pro restores "medium" in the
reasoning-effort ladder (Google's official Gemini API thinking
doc lists low/medium/high for Gemini 3.1 Pro; my earlier round 4
coercion was wrong).
- provider-capabilities orders `gemini-2.5-flash-lite` ahead of the
broader `gemini-2.5-flash` prefix so Flash-Lite falls into the
"no native thinking knob" branch as documented.
* Studio: round-trip Gemini tool_calls and tool results (PR #5720)
Recurring round 3-6 P1: the chat-adapter renders Gemini function-call
parts and code-execution events but `toOpenAIMessage` only serialized
text + image content, so the next turn lost the assistant
`tool_calls[]` (including Gemini 3's required
`extra_content.google.thought_signature`) and the matching
`role="tool"` result. Gemini 3 multi-turn function calling and code
execution failed validation on the second turn.
Frontend:
- types/api.ts widens OpenAIChatMessage to permit `role="tool"`,
`tool_calls`, `tool_call_id`, `name`, and `content: null`. Adds
OpenAIToolCallPart with `extra_content` for the Gemini round-trip.
- chat-adapter: new `toOpenAIMessages` expands an assistant turn with
tool-call parts into [assistant w/ tool_calls + extra_content,
role=tool result, ...]. tool result content is JSON-serialized so
the backend translator can rebuild Gemini's `functionResponse`
shape.
- chat-adapter outbound history now uses `flatMap(toOpenAIMessages)`
so each assistant tool-call round-trips through the standard OAI
shape the backend's `_stream_gemini` already understands.
* Studio: replay Gemini code_execution and image native parts on history (PR #5720)
Multi-turn Gemini history previously lost the native executableCode,
codeExecutionResult, and inlineData parts because the outbound
translator regenerated a generic functionCall for every assistant
tool_call. Stow the native dict on tool_end (frontend) and replay it
verbatim with thoughtSignature (backend) so follow-up turns preserve
the prior execution and image generation state. Skip role="tool"
fan-out for server-side builtin tools so Gemini does not 400 on a
functionResponse with no matching user-declared function.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: complete Gemini built-in tool replay round-trip (PR #5720)
Round 7 follow-up to the multi-turn native-part work. Three asymmetric
storage/consume gaps remained between the backend translator and the
chat adapter, so realistic Gemini follow-up turns degraded to generic
functionCalls instead of native history.
- Frontend collectAssistantToolCalls now drops web_search outright,
drops code_execution / image_generation when the native part is
missing, and promotes args.google to extra_content.google so the
backend native_part replay branch actually fires.
- Backend image_generation tool_end now emits google.native_part
with the inlineData (mimeType + base64) and thoughtSignature so the
follow-up image-edit turn can replay the prior image as a native
Gemini model part.
- Backend code-execution plot tool_end now stows google.native_part
with the inlineData so the merged code-exec card can round-trip
executableCode + codeExecutionResult + inlineData on the same id.
- Added regression tests for image-gen native-part replay and the
code-exec plot native_part stow.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 8 Gemini follow-ups (PR #5720)
- Text-part thoughtSignature: stow on the assistant message during
streaming and replay onto the last text part on the next turn so
Gemini 3 strict function-calling does not reject history.
- Function declarations: recursively strip Gemini-unsupported OpenAPI
keys (additionalProperties, $schema, $defs, strict, etc.) so OpenAI
strict tools stop 400ing as INVALID_ARGUMENT on Gemini.
- OpenAI-compat fallback: forward tools/tool_choice so custom Gemini
proxies (LiteLLM, gateways) keep function-calling.
- enable_prompt_caching: cover the Pydantic v1 legacy off/on/f/n/t/y
string set so explicit opt-outs stay opt-out (Gemini was sending
cachedContent: "off" otherwise).
- Frontend collectAssistantToolCalls / collectToolResultMessages: use
google.native_part + result presence to disambiguate provider
builtins from same-named user-declared functions.
- Added regression tests for text-signature replay and schema
sanitization.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 9 Gemini follow-ups (PR #5720)
Two round-9 convergent finds across the 12 reviewers:
- Server-side web_search was leaking onto the next turn as a fake
user functionCall/functionResponse. The previous heuristic (skip
builtin only when no native_part AND no result) let it through
because the synthetic tool card has a non-empty result string.
Always skip web_search by name on both serializers, accept that a
user-declared function literally named "web_search" must use a
different name.
- Assistant `extra_content` was dropped by ChatMessage validation
before _stream_gemini could replay text-part thought signatures.
Add the field to ChatMessage and forward it through
_build_external_messages so the multi-turn signature path actually
carries data.
Includes a regression test for the ChatMessage round-trip.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 10 Gemini follow-ups (PR #5720)
Three convergent round-10 reviewer findings closed:
- Tag synthetic provider-side builtins with `args._server_tool=True`
via a central helper that runs in every `_emit_tool_event` /
`_emit_synthetic_tool_event` path. The frontend filter now skips
on that marker instead of on the public tool name, so local
llama.cpp `web_search` and OpenAI function tools literally named
`web_search` / `code_execution` / `image_generation` round-trip
cleanly while Gemini grounding / hosted code-exec / hosted image
cards stay skipped.
- Gate Gemini image-mode (responseModalities=[TEXT,IMAGE]) on the
Images pill (enabled_tools containing `image_generation`).
Selecting an image-capable model with the pill off no longer forces
image output the UI says is disabled.
- Frontend missing-key guard now exempts custom Gemini OAI-compat
proxies (LiteLLM, gateways) the same way the backend already
does, so a saved Gemini connection on `http://localhost:4000/v1`
with no API key stops being blocked.
Existing tests updated to pass `enabled_tools=["image_generation"]`
on image-mode capture paths.
* Studio: round 11 Gemini follow-ups (PR #5720)
Four round-11 findings closed:
- Kimi _stream_kimi_web_search's local _synthetic_chunk helper now
runs through _stamp_server_tool_marker so Kimi search history is
not replayed as a fake user functionCall on the next turn (was an
asymmetric miss after the round-10 tagging work).
- OpenAI Responses path (/v1/responses for gpt-5.x) forwards
caller-supplied tools / tool_choice, translating the Chat
Completions function-tool shape into the Responses native shape.
Without this, standard OpenAI tools silently dropped on
Responses-routed traffic.
- Decoupled the Gemini image-tier model-id guards (text-tool /
thinking strip) from the Images pill flip
(responseModalities=[TEXT,IMAGE]). gemini-2.5-flash-image with
Search/Code on and the Images pill OFF no longer forwards
googleSearch + thinkingConfig (Gemini 400s on those for legacy
image ids).
- Gemini-only extra_content is now forwarded by
_build_external_messages only when provider_type=="gemini" so
Google's thought_signature does not leak into OpenAI / Mistral /
Kimi / OpenRouter request bodies as an unknown field.
Added a regression test for the image-tier strict-guard split and
extended the extra_content test to cover the non-Gemini suppression.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 12 Gemini follow-ups (PR #5720)
Three round-12 convergent findings closed:
- extra_content leak to custom Gemini OAI-compat proxies (8/12
reviewers). _build_external_messages now gates extra_content on
the native generativelanguage.googleapis.com host, not just
provider_type=="gemini", so LiteLLM / custom gateways routed
through /chat/completions do not get an unknown top-level field.
- OpenAI Responses function-tool round-trip (5/12 reviewers). I
added user `tools` forwarding in round 11 but did not parse the
matching response.output_item.done items of type=function_call.
The parser now translates them into Chat Completions
delta.tool_calls and the terminal chunk reports
finish_reason="tool_calls" when the model invoked a user
function.
- Image-tier model with Images pill OFF (2/12). Google's image
models default to text+image when responseModalities is omitted,
so the previous fix silently still billed image output. Force
responseModalities=["TEXT"] when the Images pill is off and the
selected model is image-capable.
Updated the two pre-existing tests that pinned the synthetic-tool
arguments shape to include the new `_server_tool: True` marker, and
added a regression test for the Responses function-call output
translation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 13 Gemini/Responses follow-ups (PR #5720)
Three round-13 convergent findings closed:
- OpenAI Responses function_call indices: my round-12 translator
hardcoded every emitted tool_calls[*].index to 0, so parallel
function calls collapsed for index-keyed clients. Track and
increment function_call_index per emit (mirrors the Gemini
branch's distinct-index pattern). 10/12 reviewers flagged.
- _SERVER_SIDE_BUILTIN_TOOL_NAMES now includes web_fetch so
Anthropic-hosted web_fetch cards carry the _server_tool marker
and the frontend history serializer doesn't replay them as fake
user functions. 4 reviewers flagged.
- OpenAI Responses follow-up tool results now serialize as
Responses-shape function_call / function_call_output items keyed
by call_id, instead of Chat Completions role="tool" content.
Skips assistant tool_calls tagged with _server_tool so hosted
builtins don't round-trip as user functions. 2 reviewers flagged.
Updated the Anthropic code_execution and web_fetch test argument
pins to include the new _server_tool marker, and added two
regression tests (distinct indices on parallel function_call,
function_call_output round-trip).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 14 Gemini follow-ups (PR #5720)
Three round-14 findings closed:
- Remote `image_url` translation (5 reviewers convergent). Public
HTTPS image URLs can't be sent as `fileData.fileUri` -- Gemini
reserves that path for Files API URIs and YouTube. Fetch the
bytes server-side and inline them as base64 `inlineData`,
mirroring the pre-PR OpenAI-compat behaviour. YouTube URLs and
generativelanguage.googleapis.com/v1beta/files/* stay as
`fileData`.
- Nullable JSON Schema type arrays. OpenAI strict tools commonly
use `"type": ["string", "null"]`; the Gemini sanitizer now
flattens that to `"type": "string", "nullable": true` so strict
function tools stop 400ing.
- Parallel functionResponses now ride on one user content block
with multiple `functionResponse` parts, matching Google's
parallel tool docs. Consecutive `role="tool"` messages merge
into the previous user turn instead of splitting into separate
Gemini user turns.
Three regression tests added (remote URL fetch + inline, Files
API / YouTube fileData preservation, schema nullable flattening,
parallel-tool grouping).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: SSRF harden Gemini remote image fetch (PR #5720)
Round 15 convergent finding (12/12 reviewers). My round-14 fix to
download user-controlled image URLs for inlineData inlining was an
SSRF / data-exfiltration path: no scheme check, no private-host
guard, no size cap, no Content-Type validation, redirects could
bounce to internal services, and the full URL was logged.
Replace the inline fetch with `_safe_fetch_image_for_gemini`:
- Require https:// (reject http, file, data, ftp, etc).
- Resolve the hostname via socket.getaddrinfo and reject if ANY
resolved address is private / loopback / link-local / multicast /
reserved / unspecified (covers 127.0.0.0/8, 10/8, 172.16/12,
192.168/16, ::1, 169.254/16 metadata, RFC 6890).
- Block IP-literal URLs that resolve into those same ranges.
- Cap response body at 10 MB (Content-Length pre-check + streamed
byte counter).
- Require Content-Type to start with `image/`.
- Disable redirect following so a 302 to a private host can't slip
past the address check.
- Use a short 15s timeout and a tiny connection pool dedicated to
these fetches.
- Log only the host name + error class -- no full URL, no signed
querystring leak.
If the guard rejects, the image part is silently dropped (instead
of forwarding raw bytes or a fileData fallback). Files API URIs
and YouTube URLs still ride as `fileData.fileUri` unchanged.
Tests: replaced the live-fetch test with a `_safe_fetch_image_for_gemini`
monkeypatch, added four new SSRF-guard tests (non-https rejected,
loopback / private IP literals rejected, hostnames that resolve to
private IPs rejected).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 16 Gemini follow-ups (PR #5720)
- IP-pinned image fetch (`_safe_fetch_image_for_gemini`): reuse the
validated-once-then-pin pattern from `tools._fetch_page_text` via
`asyncio.to_thread`, so DNS rebinding between validation and the
HTTP connect cannot redirect us at a private/metadata address.
Catch malformed-bracketed IPv6 urlparse errors. Follow up to 4
redirect hops with per-hop SSRF re-validation.
- Replace contains-substring detection of Gemini Files API + YouTube
URLs with parsed scheme/host/path checks, so attacker URLs like
`https://evil.example/path/youtube.com/x.png` no longer skip the
safe-fetch path and serialize as `fileData.fileUri`.
- `_build_external_messages`: strip per-tool-call `extra_content`
for non-native-Gemini providers; the Gemini-only
`thought_signature` payload was leaking through `tool_calls[]`
into /chat/completions on OpenAI, Anthropic, and custom Gemini
OAI-compat gateways.
- `_server_tool` marker now gated on the function name being one of
the canonical builtin names (`web_search`, `web_fetch`,
`code_execution`, `image_generation`) AND the marker being set,
so a user function whose schema happens to define an
`_server_tool` field is no longer dropped. Frontend filter mirrors
the same gate, plus a backward-compat fallback for pre-PR
persisted server-tool cards (no marker) routed via name +
native_part / web-tool heuristic.
- Gemini schema sanitizer collapses `anyOf: [{X}, {"type":"null"}]`
to `{X, "nullable": true}` so Optional[X] tool args from
OpenAI/Pydantic schemas no longer 400 the Gemini request.
- Frontend tool-result serializer emits `{"result":""}` for empty
string outputs so the ChatMessage validator does not reject
`role="tool"` with empty content.
- Coerce `medium` thinkingLevel to `high` for legacy
`gemini-3-pro*` / `gemini-3-pro-preview*` (only low/high
documented; shut down 2026-03-09); 3.1+ Pro still passes through.
- Hide Gemini native thinking ladder on custom OAI-compat Gemini
gateways by routing `getExternalReasoningCapabilities` through
`isGeminiCustomOpenAICompatBase(baseUrl)`; thread baseUrl through
all four call sites.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 17 Gemini follow-ups (PR #5720)
- Frontend `collectAssistantToolCalls` and `collectToolResultMessages`
no longer drop unmarked `web_search` / `web_fetch` cards by name
alone: a user-defined function with one of those names must
round-trip. Pre-PR persisted `code_execution` / `image_generation`
cards still get filtered via a shape heuristic (kind/command/code/
prompt fields) instead of bare name.
- `_build_external_messages._filter_tool_calls` now drops marked
server-side builtin `tool_calls` entirely for non-native-Gemini
providers, not just their `extra_content`. An assistant turn whose
only payload was a marked builtin is dropped completely so the
receiving provider does not see an orphan tool_call.
- `_stream_anthropic` translates OpenAI top-level `tool_calls` into
Anthropic native `{type:"tool_use", id, name, input}` content
blocks, and translates `role="tool"` follow-ups into `role:"user"`
messages carrying a `tool_result` block. Anthropic's native
Messages API rejects the OpenAI shapes.
- `_safe_fetch_image_for_gemini_sync` factors URL validation through
`_safe_parse_https`, so malformed `port` access (e.g.
`https://host:bad/x.png`) and malformed redirect targets (e.g. a
302 to `https://[bad/x.png`) drop the image instead of raising mid-
request.
- `tool_choice="none"` now disables hosted builtins (Gemini
googleSearch / codeExecution and OpenAI Responses web_search /
shell / image_generation), not just user function declarations.
- Schema sanitizer handles multi-type `anyOf` with null
(`Union[str, int, None]`): keep the slim non-null anyOf and add
`nullable: true` so Gemini does not reject `{"type":"null"}`.
- Image fetch falls back to the caller-provided MIME (guessed from
URL extension) when the server omits Content-Type instead of
dropping the image as `non-image content-type=<none>`.
- Per-request aggregate caps on remote image inlining (8 images,
20MB total) so a single chat request cannot force unbounded
backend downloads.
- Frontend exposes the reasoning ladder for `gemini-2.5-flash-lite`
(`none/minimal/low/medium/high/max`) so the UI can drive the
thinkingBudget the backend already supports.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 18 Gemini follow-ups (PR #5720)
- `tool_choice="none"` now opts out of hosted builtin tools on every
provider path, not just Gemini and OpenAI Responses. Anthropic
web_search / web_fetch / code_execution, Kimi `$web_search` early
return, and OpenRouter `plugins:[{id:"web"}]` are all gated on
`tool_choice_disabled`. Passing `enabled_tools=[...]` with
`tool_choice="none"` no longer triggers provider-side search /
code execution for any provider.
- `_stream_anthropic` accepts `tool_choice` and threads it through;
the dispatcher in `stream_chat_completion` forwards it.
- Frontend `isServerSideBuiltinToolPart` simplified to drop only on
(marker) OR (canonical name + native_part). The previous shape
heuristic on `args.kind`/`args.command`/`args.code`/`args.prompt`
dropped real user-declared `code_execution` / `image_generation`
functions. Pre-PR persisted hosted cards lacking the marker now
leak to non-native providers on switch -- preferred to silently
deleting legitimate function-call history.
- Backend `_is_marked_server_builtin_tool_call` and the OpenAI
Responses translator's matching filter accept BOTH `_server_tool`
marker AND `args.google.native_part` as durable provider-side
signals so Gemini code_execution / image_generation cards are
still dropped on a provider switch.
- Per-request remote image count cap now counts ATTEMPTS, not just
successful inlines, so 100 failing/slow URLs cannot each consume
the 15s fetch timeout. Data: URL images now share the same count
and byte caps as fetched remote URLs.
- OpenAI Responses translator tracks skipped server-builtin
`function_call` ids and drops their matching `role="tool"`
follow-ups, preventing orphan `function_call_output` items in the
outbound body.
- Gemini schema sanitizer preserves multi-type unions with null:
`{"type":["string","integer","null"]}` becomes
`anyOf:[{string},{integer}] + nullable:true` instead of being
flattened to the first non-null type.
- Gemini model id validation moved to the top of `_stream_gemini`
so an invalid model id rejects the request before any remote
image fetch / message translation side effect.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 19 Gemini follow-ups (PR #5720)
- `_build_external_messages` now skips an empty assistant turn when
`_filter_tool_calls` drops every synthetic builtin tool_call (was
guarded only on the `content is None` branch; the string-content
and list-content branches still forwarded
`{"role":"assistant","content":""}` which several providers
reject). Also tracks the dropped server-builtin tool_call ids and
skips the matching `role="tool"` follow-ups so the receiving
provider does not see an orphan tool_result.
- OpenRouter `web_search_active` (the synthetic tool_start /
tool_end emitter) is now also gated on `tool_choice_disabled` so
a request with `tool_choice="none"` does not surface a fake
web_search card in the chat UI even though the plugin was
correctly stripped from the outbound body.
- `_stream_anthropic` translates an OpenAI role="tool" with list
content (`content=[{"type":"text","text":"..."}]`) into a native
`tool_result` block on a user message; previously only the
string-content shape was translated, so list-content tool results
were forwarded as invalid `role:"tool"` messages.
- Gemini `data:` URL image_url parts now require an `image/*` MIME
type; a `data:text/html;base64,...` is dropped instead of being
forwarded as `inlineData.mimeType="text/html"` (Gemini rejects
the malformed image part). Symmetric with the fetched-remote
image fetch path that already rejects non-image Content-Type.
- YouTube `fileData.fileUri` now declares `video/mp4` as the
mimeType instead of `image/jpeg` guessed from the URL path. The
YouTube/fileData input is the documented Gemini video path; the
guessed image MIME made valid YouTube inputs malformed.
- OpenAI Responses translator preserves `response.output` ordering
on assistant turns that emitted both text and a function_call:
assistant text is now serialized BEFORE the function_call item
so the subsequent function_call_output (the matching role=tool
follow-up) lands in the right position. Previously the order
was function_call -> assistant text -> function_call_output,
which can confuse multi-turn function-calling flows.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: round 20 Gemini follow-ups (PR #5720)
Convergent reviewer findings from round 20:
- tool_choice="none" no longer flips responseModalities=[TEXT,IMAGE]
on image-tier Gemini models. Forced-function tool_choice (e.g.
{type:function, function:{name:lookup}}) also drops hosted Search /
code execution from the Gemini body so the caller's pinned user
function is not silently joined by hosted builtins.
- Gemini code-execution thoughtSignature replay now uses an ordered
parts list (native_part.parts[]) so per-part signatures stay
attached to the exact part Gemini emitted. The previous merged
shape fanned one top-level thoughtSignature across executableCode
+ codeExecutionResult + inlineData and tripped Gemini 3 strict
validators. Backward-compat fallback keeps pre-round-21 persisted
history working: a legacy native_part with a single subpart still
replays the signature on that subpart; merged legacy objects pin
the signature to executableCode only.
- Remote-image fetch threads the remaining per-request byte budget
into _safe_fetch_image_for_gemini, so over-budget URLs are
refused via Content-Length pre-check / short read instead of
fully downloaded then discarded after the aggregate cap check.
- Gemini role=tool with OpenAI list-form content
([{type:text,text:result}]) now flattens text parts before
building functionResponse.response.result; previously the parts
arrived as the result value instead of the actual tool output.
- Frontend chat-adapter merges native_part by concatenating parts
lists (preserving per-part thoughtSignature). Wire types expose
enable_prompt_caching as boolean|string (Gemini cached-content
name) and OpenAIChatDelta now carries tool_calls and extra_content.
- Test test_openrouter_no_synthetic_web_search_event_on_tool_choice_none
reads _toolEvent from the top-level SSE payload so a backend
regression cannot mask the assertion.
Adds 7 regression tests covering image_generation gate, forced-function
gate, native_part list replay, legacy fallback, list-content
functionResponse flattening, fetch byte-budget threading, and wire
types.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Apply forced-function tool_choice gate to Anthropic, OpenRouter, Kimi
Previously only the Gemini path treated `tool_choice={"type":"function",
"function":{"name":...}}` as a hosted-tool opt-out. Anthropic,
OpenRouter, and Kimi still attached hosted web_search / web_fetch /
code_execution when the caller explicitly pinned a user function plus
`enabled_tools=[...]`. That contradicts the explicit function pin and
bills the caller for unwanted server-side calls.
Mirror the Gemini gate symmetrically:
- Anthropic web_search / web_fetch / code_execution
- OpenRouter `plugins:[{id:"web"}]` + the synthetic web_search SSE
event the same path emits at stream close
- Kimi `_stream_kimi_web_search` dispatch
Adds 4 regression tests:
- test_anthropic_forced_function_tool_choice_drops_hosted_tools
- test_openrouter_forced_function_tool_choice_drops_web_plugin
- test_kimi_forced_function_tool_choice_skips_web_search_helper
- test_openrouter_no_synthetic_web_search_event_on_forced_function_tool_choice
All 146 existing backend tests still pass.
* Strip Gemini-only synthetic tool history on local-GGUF dispatch
After a Gemini chat that ran code_execution / image_generation, switching
the same thread to a local GGUF model used to forward the synthetic
provider-side tool_calls (tagged with `args._server_tool` or carrying a
Gemini `args.google.native_part` payload) and the message-level
`extra_content` to llama-server. The receiving backend has no tool
declaration for those names and no use for Gemini thoughtSignature
metadata; in the worst case it can produce an orphan tool_call_id and a
confused continuation.
Add `_strip_provider_synthetic_tool_history()` and wire it through the
two local message builders:
- `_openai_messages_for_passthrough` (OAI-compat passthrough)
- `_openai_messages_for_gguf_chat` (standard GGUF chat path)
Real user-function `tool_calls` and their matching `role="tool"` replies
survive unchanged; only synthetic provider-side cards and Gemini-only
`extra_content` are stripped. If the synthetic call was the assistant
turn's only payload, the now-empty turn is dropped too so llama-server
does not reject the request.
Adds 2 regression tests:
- test_strip_provider_synthetic_tool_history_drops_synthetic_only
- test_strip_provider_synthetic_tool_history_drops_empty_assistant
142 existing backend tests still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Disable Search/Code composer pills for Gemini image-tier models
For external Gemini image-tier models (gemini-2.5-flash-image,
gemini-3.x-image-preview, etc.), the backend unconditionally strips
code_execution and strips web_search on older image ids. Search is
still allowed on Gemini 3.x Pro/Flash image models, which
supportsBuiltinWebSearch already encodes per model.
Before this commit the composer pill gates were:
searchDisabled = !modelLoaded || !(supportsTools || supportsBuiltinWebSearch)
codeDisabled = !modelLoaded || !(supportsTools || supportsBuiltinCodeExecution) || imageModeDisablesCode
`supportsTools` here is a local-runtime fallback that becomes true when
any tool-capable local model has been loaded in the session. With a
local tool-capable runtime active, switching the chat to an external
Gemini image-tier model used to leave Search/Code clickable, even
though the backend will silently drop the tool on the wire.
Detect "external provider is Gemini AND the model is image-tier" (via
supportsBuiltinImageGeneration) and gate the two pills strictly on the
provider's own builtin support in that case. Non-Gemini paths and
non-image Gemini models keep the supportsTools fallback unchanged.
* Apply forced-function tool_choice gate to OpenAI Responses path
Round 22 added the gate for Gemini / Anthropic / OpenRouter / Kimi but
missed the OpenAI Responses translator. When a caller pinned a user
function via `tool_choice={"type":"function","function":{"name":...}}`
plus `enabled_tools=["web_search","code_execution","image_generation"]`,
the Responses body still attached `{"type":"web_search"}`,
`{"type":"shell"}`, and `{"type":"image_generation"}` server tools. The
function pin should suppress those for the same privacy + billing reason
the other provider paths now do.
Compute `_responses_tool_choice_forced_function` next to
`_responses_tool_choice_none` and gate each hosted-tool append on
`_responses_hosted_builtins_allowed = not none and not forced_function`.
The fix has to be applied in TWO places: the initial body builder and
`_build_body()` (called by the container-expiry retry path). User
function declarations still flow through so the pin has something to
target, and the Responses-shape `{type:"function", name:"..."}`
`tool_choice` is forwarded unchanged.
Adds regression test `test_openai_responses_forced_function_tool_choice_drops_hosted_tools`.
All 166 existing backend tests across Gemini + Responses + image-gen +
code-exec suites still pass.
* Round 24 P1s: SSRF shared-address gap + extra_content text-only leak + custom-Gemini model list
Three convergent P1s from round 24 review:
1. SSRF: the shared SSRF validator in `tools._validate_and_resolve_host`
used a denylist (is_private / loopback / link_local / multicast /
reserved / unspecified). Python classifies shared address space
(100.64.0.0/10 carrier-grade NAT, plus 240.0.0.0/4, benchmarking
ranges, etc.) with `is_private=False` AND `is_global=False`. The new
Gemini server-side image fetcher therefore accepts URLs whose
hostname resolves to 100.64.0.1 in cloud/VPC deployments. Add
`not ip.is_global` as the primary gate -- a single source of truth
that covers every current and future non-global range.
2. _strip_provider_synthetic_tool_history previously only stripped
message-level `extra_content` when the assistant turn had tool_calls.
A plain text Gemini reply carrying
`extra_content.google.thought_signature` flowed through to
llama-server when the thread was switched to a local GGUF backend.
Always strip message-level `extra_content` on assistant turns.
3. routes/providers.list_provider_models applied Gemini's native
`model_id_allowlist` regex to every Gemini provider, including
custom OAI-compatible bases (LiteLLM, deployment gateways). IDs like
`google/gemini-2.5-flash` and team-prefixed deployment aliases got
filtered out even though the chat-dispatch path now routes them via
the OpenAI-compatible client. Skip registry-level model-id filters
when the configured Gemini base_url host is not the canonical
`generativelanguage.googleapis.com`, mirroring the chat-dispatch
gate.
Three regression tests added:
- test_validate_and_resolve_host_blocks_shared_address_space
- test_strip_provider_synthetic_tool_history_drops_text_only_extra_content
- test_gemini_custom_oai_compat_base_skips_native_allowlist
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Round 25 P1s: skip synthetic server-tool replay + inline $ref/$defs into Gemini schema
Two convergent reviewer findings on the native Gemini path:
1. _stream_gemini's tool_calls replay loop falls through to a generic
functionCall emission whenever it sees an assistant tool_call. Marked
server-side builtin cards (web_search / web_fetch tagged with
_server_tool or args.google.native_part) hit that fallthrough with no
replayable native_part, which produces an outbound functionCall whose
name is not a declared user function. The Gemini turn 400s on the
undeclared name. Guard the loop to drop those entries instead, while
keeping the existing code_execution / image_generation native-part
replay branch intact.
2. _sanitize_gemini_schema uses a strict allowlist that drops local
$ref / $defs references. Pydantic-generated tool schemas hoist nested
object shapes into $defs and reference them via {"$ref": "#/$defs/X"},
so a property like address: {"$ref": "#/$defs/Address"} collapsed to
{} on the wire and the model lost the nested fields, types, and
required keys. Resolve local #/... pointers against the schema root
and inline the referenced subtree, with local siblings overriding
the reference (normal JSON Schema composition) and a seen-ref guard
for self-referential schemas.
Added regression coverage:
- test_gemini_native_skips_synthetic_server_builtin_replay
- test_function_declarations_inline_local_refs_into_gemini_schema
- test_function_declarations_inline_local_refs_in_anyof_and_items
- test_function_declarations_self_referential_schema_terminates
All 145 Gemini provider tests pass; touched provider regression set
(OpenAI Responses, code execution, image generation, Anthropic code
execution, Anthropic web_fetch) also 43/43 green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Round 26 P1s: drop orphan Gemini functionResponse + Anthropic /messages synthetic-history strip
Reviewer round 26 surfaced two convergent asymmetric-fix bugs.
1. _stream_gemini drops a synthetic server-tool tool_call (web_search /
web_fetch tagged _server_tool) and also replays code_execution /
image_generation tool_calls as Gemini-native executableCode /
codeExecutionResult / inlineData parts. The matching role="tool"
follow-up was still falling through to the generic functionResponse
branch, producing either an orphan functionResponse (synthetic case)
or a duplicate response pointing at a name with no
functionDeclarations entry (native-part case). Both forms 400 the
next Gemini turn. Track skipped + native-replayed tool_call_ids in
_gemini_skip_tool_result_ids and short-circuit the role="tool"
branch on a match.
2. The Anthropic-compatible local /v1/messages route only called
_drop_empty_assistant_sentinels on the OpenAI-translated history,
while the sibling /v1/chat/completions and GGUF passthrough builders
chain that with _strip_provider_synthetic_tool_history. An Anthropic
caller replaying a prior provider-side tool_use therefore forwarded
fake builtin tool history straight into local llama-server. Apply
the same strip on the Anthropic route after the
anthropic_messages_to_openai conversion.
Regression coverage added:
- test_gemini_native_skips_orphan_function_response_for_dropped_builtin
- test_gemini_native_skips_orphan_function_response_for_native_part_replay
Gemini suite 147/147; touched provider regression set 43/43.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Round 27 P1s: native_part location fallback + Gemini image request budget for base64
Two convergent reviewer findings on the native Gemini path.
1. _stream_gemini's synthetic-builtin detector at lines 3519-3524
recognizes args.google.native_part as a server-tool marker, but
_native_part was only loaded from tc.extra_content.google.native_part.
A direct OpenAI-compatible API caller or imported third-party thread
round-trips the payload through function.arguments because
tool_calls[].extra_content is not in the OpenAI spec. The round-25
guard then saw a synthetic builtin with no _native_part and dropped
the entire assistant turn, so the next native Gemini request lost
the prior executableCode / inlineData / codeExecutionResult context.
Fall back to args.google.native_part when extra_content path is
missing, mirroring what the synthetic detector already accepts.
2. _GEMINI_REMOTE_IMAGE_MAX_TOTAL_BYTES capped DECODED bytes at 20MB.
Gemini receives images base64-encoded inside JSON, and base64
inflates payload size by ~4/3. With 20MB decoded the actual JSON
body is ~26.7MB plus prompt overhead, well over Gemini's ~20MB
request limit. Drop the decoded cap to 14MB so realistic multi-
image turns stay safely under 20MB encoded.
Added regression test test_gemini_native_part_falls_back_to_args_google
covering an OpenAI-compat-shaped image_generation tool_call whose
native_part lives only in function.arguments.
Gemini suite 148/148.
* Fix TS build errors from main merge: restore imageParts + refusal return [] + cast image-edit ref
Three errors in chat-adapter.ts surfaced by the frontend tsc step after merging
main into feat/gemini-provider:
1. The Anthropic refusal early-return used main's but
toOpenAIMessages returns SerializedMessage[]; flip to .
2. Restore -- the line
was lost when removing main's conflict block from the function body.
3. selectedImageEditReference splice was inserting OpenAIChatMessage
into a SerializedMessage[] array; the shapes differ on tool_calls.id
nullability. Cast the reference message through unknown -- it carries
no tool_calls, so the runtime payload is structurally compatible.
Reproduced locally with `tsc -b --pretty false` (now passes). Build
also failing in the in-repo `npm run build` step on PR CI; this commit
unblocks all 12 failing UI/API workflows.
* Tighten verbose comments in external_provider.py + chat-adapter.ts
Compress multi-line explanatory comments in the Gemini translator
and the chat adapter without changing any behaviour. All 148 Gemini
provider tests still pass; tsc --noEmit clean.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com>
* Studio: add VLM image-size control for training
Studio vision fine-tuning had no explicit way to cap image resolution, so
users could not trade visual detail against context and memory use from the
training UI, YAML config, or API payload. :) Add a nullable `vision_image_size`
setting that keeps the current model default when unset and applies a
max-side resize when provided.
- Add `vision_image_size` to the training request model, route payload, backend
training config, and frontend API/types plumbing.
- Validate the value server-side as either null or an integer in the supported
256-2048 range.
- Surface an Image Size selector for vision LoRA training with Default plus
common preset sizes.
- Include the value in training start payloads only for image-dataset vision
models, and serialize it into vision-aware YAML configs.
- Map backend model defaults back into the training store and reset the value
when reapplying model defaults.
- Pass the resize through the Torch trainer via `UnslothVisionDataCollator`
using max-dimension semantics.
- Apply the same max-dimension resize in the MLX VLM path before mlx-vlm's
internal collation, preserving aspect ratio and avoiding upscaling.
- Add backend validation coverage and MLX resize-size tests for the new
behavior.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: thread vision_image_size into DeepSeek OCR + writable MLX ndarray
- trainer.py: DeepSeek OCR collator now honors the new vision_image_size
setting as image_size. Falls back to 640 when null. base_size stays at
1024 and crop_mode stays True so the Gundam preset's dynamic cropping
of large documents keeps working.
- worker.py: _resize_mlx_vlm_image returns np.array(image, copy=True)
instead of np.asarray(image). The PIL view from np.asarray is not
writable, which makes HF VLM processors emit "The given NumPy array
is not writable, and PyTorch does not support non-writable tensors..."
when they call torch.from_numpy. copy=True keeps the same shape and
dtype but produces a writable buffer.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: align YAML export gate with API mapper + extend Image Size dropdown
- training-section.tsx: handleSaveConfig now passes
isVisionModel && isDatasetImage === true to serializeConfigToYaml,
matching buildTrainingStartPayload. Stops vision_image_size from
leaking into exported YAML for text-only datasets where the API
would have sent null.
- params-section.tsx: add 256 to visionImageSizePresets so the
dropdown spans the validator's full [256, 2048] range. Also render
a synthetic SelectItem for the current value when it was loaded
from YAML or model defaults and is not in the preset list, so the
controlled Select always shows the active size.
* Studio: validate vision_image_size in YAML/model-default loader
mapBackendModelConfigToTrainingPatch now mirrors the backend validator
at studio/backend/models/training.py:169 by dropping any value that is
not an integer in [256, 2048]. Pre-fix, an imported YAML like
vision_image_size: 4096 or 640.5 would land in the store and the UI
would happily display it, only to fail when Start Training posted to
the backend. With this guard the store never holds a value the backend
would reject.
* Studio: precise error messages for invalid vision_image_size inputs
Switch the field_validator to mode="before" so True/False surface as
bool (not Pydantic's coerced 1/0) and give a precise
"must be an integer or null" message instead of the misleading
"must be in [256, 2048] (got 1)". Also explicitly accepts numpy
Integral and integral Real scalars so YAML or programmatic callers
using numpy ints keep working.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: test that bool inputs yield the precise 'integer or null' error
Regression guard for the validator switch to mode="before". Pre-fix,
vision_image_size: True was rejected with "must be in [256, 2048]
(got 1)" because Pydantic coerced before our check ran. New test
asserts the message now reads "integer or null".
* Studio: tighten vision_image_size loader + YAML save + MLX rounding
Round 2 of follow-up review surfaced three usability issues:
- model-defaults.ts: switching to a model whose backend YAML omits
vision_image_size now explicitly resets the store value to null.
Pre-fix, a stale 2048 from a previous model would silently apply
to the new run because every checked-in model-default file omits
the key.
- training-section.tsx: handleSaveConfig now includes vision fields
unless isDatasetImage is definitively false. isDatasetImage is null
during dataset checks, after dataset edits, and on import; treating
unknown as "drop" would silently lose the user's selection in those
windows. Confirmed-text-only datasets still drop the value.
- worker.py: _mlx_vlm_max_resized_size now mirrors the Torch collator's
integer formula (w * size + size_func // 2) // size_func instead of
Python round(), which uses banker's rounding and disagreed by 1px on
half-pixel inputs like 333x1000 with target 500 (was 166, now 167).
Test_mlx_training_worker_config gains parity assertions.
* Studio: reset vision_image_size in the model-config error fallback path
mapBackendModelConfigToTrainingPatch resets stale image size on the
success path, but if the /api/models/config endpoint throws,
training-config-store.ts falls through to checkVisionModel and only
updates capability flags. Pre-fix that left a stale 2048 (or any
prior selection) in the store, so once dataset detection marked the
new dataset as image, the next training start would silently apply
the previous model's size. The error branch now also resets to the
DEFAULT_HYPERPARAMS.visionImageSize sentinel.
* Studio: revert DeepSeek OCR Image Size knob + move missing-key reset
Round 3 of the parallel-reviewer pass surfaced two issues that I had
introduced earlier in this PR's follow-ups.
- trainer.py: my prior change threaded vision_image_size into the
DeepSeek OCR collator's image_size argument. The collator's
(image_size, base_size, crop_mode) is a single preset
(Tiny / Small / Base / Large / Gundam); changing image_size in
isolation desynchronizes the per-crop pixel grid from num_queries
downstream and produces wrong token grids on documents larger than
the per-crop tile. The fix pins the collator back at the Gundam
preset and logs a clear "ignored for DeepSeek OCR" notice when the
user has selected a non-default Image Size.
- model-defaults.ts + training-config-store.ts: the round 4 fix that
reset visionImageSize when a model YAML omitted the key also fired
on same-model reloads (ensureModelDefaultsLoaded re-fires on page
refresh), wiping a value the user had just selected. The reset is
now in setSelectedModel, gated on selectedModel != previousModel,
so true model switches still clear stale values while reloads keep
the user's selection.
* Studio: extend DeepSeek OCR Image Size exclusion to MLX + frontend
Round 4 of the parallel-reviewer pass flagged that the Torch trainer
exclusion I added did not have a matching MLX guard, and that the UI
still offered the dropdown for DeepSeek OCR even though the backend
ignores it.
- worker.py: _run_mlx_training now mirrors the Torch exclusion. When
the model name matches DeepSeek OCR, vision_image_size is forced
back to None before _adapt_for_mlx_vlm sees it, so dataset images
pass through unchanged just like the Torch path. Emits a clear
status line when this happens.
- params-section.tsx: the Image Size Row is now gated on
showVisionImageSize (showVisionLora && !isDeepseekOcr) instead of
showVisionLora alone, so DeepSeek OCR users no longer see a control
that silently has no effect.
- mappers.ts: buildTrainingStartPayload sends null for vision_image_size
whenever the selected model is DeepSeek OCR, so the backend log line
about ignoring the value never fires from a UI-driven start.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: tighten YAML import/save for vision_image_size
Two YAML-path asymmetries that could leak a stale image size into
training:
- parseYamlConfig now treats a missing training.vision_image_size as
null. Without this, importing a YAML saved before this feature (or
any config that omits the key) preserved whatever value the user had
previously set on a different model. The model-defaults reload path
still uses Object.hasOwn so same-model defaults reloads do not wipe
a manual selection; only file import normalises the missing key.
- handleSaveConfig now passes a DeepSeek-OCR-specific guard to
serializeConfigToYaml so saved YAML matches what the API mapper
actually sends. Previously a state with visionImageSize set could
emit the key even though Studio ignored it at training time for
DeepSeek OCR, and a later import for a non-DeepSeek vision model
would activate the stale value.
serializeConfigToYaml gains an optional third parameter
includeVisionImageSize defaulting to includeVisionFields, preserving
the existing 2-arg call signature for backwards compatibility.
* Studio: also reset vision_image_size when YAML lacks a training section
Round 9's parseYamlConfig normalization only fired when the YAML had a
training mapping that omitted vision_image_size. A lora-only or
logging-only YAML (or one with `training: null`) still left trainingObj
unset, the mapper saw no vision_image_size key, and the previously
selected store value persisted into the next training run.
Now an absent or null training section is synthesised as
{ vision_image_size: null } so model-defaults.ts always patches
visionImageSize back to Default on file import. Same-model defaults
reloads still preserve manual choices via the existing Object.hasOwn
gate in mapBackendModelConfigToTrainingPatch.
* Studio: unify parseYamlConfig non-object training handling
A fresh static review (Opus subagent) flagged P3-1: parseYamlConfig
only synthesised vision_image_size: null when raw.training was either
absent or a plain object missing the key. If raw.training is a scalar
or an array (malformed but still parseable), the value was passed
through unchanged, the mapper's Object.hasOwn returned false, and any
previously selected visionImageSize persisted - the same stale-state
leak the lora-only fallback was added to close.
Treat any non-plain-object raw.training (null, array, scalar) as a
malformed/missing section and reset to { vision_image_size: null }.
* Studio: tighten code comments for vision_image_size path
* Studio: tighten vision_image_size validator + restore lost comment context
Two issues surfaced by a fresh adversarial review of the validator:
1. v.strip().lstrip("+-").isdigit() let "++512" / "--256" / "+-+512"
slip past the gate, then int("++512") raised an uncaught ValueError
and Pydantic surfaced "invalid literal for int() with base 10: '++512'"
instead of the contracted "vision_image_size must be an integer or null".
2. str.isdigit() returns True for Unicode digit families (full-width '５１２',
Arabic-Indic '٥١٢', Devanagari '१०२४'), and int() coerces them, so the
value reaching the backend wasn't the ASCII the user typed.
Replaced the lstrip+isdigit pair with re.fullmatch(r'[+-]?[0-9]+', stripped),
which rejects both shapes with the precise error and accepts the documented
ones ('256', '+512', ' 1024 '). Added 8 regression test cases covering
multi-sign strings, lone sign, and the three Unicode digit families.
Also restored comment context lost in f9c39331:
- model-defaults.ts: name studio/backend/models/training.py:_check_vision_image_size
as the spec the [256, 2048] range mirrors, so a maintainer changing the
cap in one file can find the other.
- training-section.tsx: enumerate the three windows in which isDatasetImage
is null (before a check, after dataset edits, on import) so a future
maintainer doesn't simplify the gate to `isCheckingDataset`.
- worker.py: qualify the writable-ndarray comment with "when a resize is
requested" so it doesn't misadvertise the resize=None early-return.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* feat(recipes): round-trip local model variants
* feat(recipes): add local model selector
* feat(recipes): wire selector into model editors
* fix(recipes): clear stale model state on relink
* feat(recipes): load selected local models for jobs
* chore(frontend): simplify biome scripts
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(recipes): handle local selector edge cases
* fix(recipes): polish local model selector behavior
* fix(recipes): delay local model restore until terminal runs
* fix(recipes): accept resolved default gguf variants
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: add Anthropic fast_mode toggle + surface streaming refusals
Fast mode (beta `fast-mode-2026-02-01`) lets Claude Opus 4.6 and 4.7
generate output tokens up to 2.5x faster at 6x standard Opus
pricing. The toggle lives in Configuration → Provider when the
selected Anthropic model is Opus 4.6 or 4.7 and is otherwise
hidden. Backend gates the same prefixes a second time so a stale
frontend cannot make Anthropic 400 the request, and the
`fast-mode-2026-02-01` beta header is merged onto whatever other
betas the request already needed (code-execution, compaction).
Streaming refusals (`message_delta.delta.stop_reason="refusal"` on
Claude 4 models) now surface a short user-facing notice in the
assistant message before the translated OpenAI chunk emits the
existing `finish_reason="content_filter"`. Previously the chat
bubble truncated silently because the SSE stopped mid-stream with
no visible explanation. Per the upstream docs the conversation
must be reset before continuing, so the notice tells the user
exactly that.
Reference:
- https://platform.claude.com/docs/en/build-with-claude/fast-mode
- https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/handle-streaming-refusals
Tests:
- studio/backend/tests/test_anthropic_fast_mode_and_refusal.py (8 cases
pinning fast_mode pass-through on 4.6/4.7, silent drop on Sonnet /
Haiku / older Opus / None / False, and the refusal notice + finish
reason on a synthetic refusal stream).
* Studio: drop refused Anthropic turns from the next request
Anthropic's streaming-refusal guidance says the refused assistant
turn must be removed or updated before the next call -- otherwise
the safety classifier keeps refusing. The PR only added a
user-visible notice; the partial assistant output (plus the notice
itself) still rode the next request via toOpenAIMessage.
Tag the refusal turn with an HTML-comment sentinel emitted alongside
the notice. The chat-adapter checks for that sentinel in
toOpenAIMessage and returns null, so the refused turn is excluded
from outboundMessages. The notice still renders in the transcript
(HTML comments don't display), so users keep the explanation.
* Studio: filter None finish_reason entries in test helper
test_refusal_maps_to_content_filter expects only ['content_filter']
in the finish_reasons list, but the post-PR refusal path emits a
user-visible content notice chunk first. Every _content_chunk
carries 'finish_reason: None' by construction; the helper was
appending those, so the assertion saw [None, 'content_filter']
instead of ['content_filter'].
None is not a finish reason -- it's just mid-stream delta noise.
Skip None values in _finish_reasons so the helper reflects what
the test names actually claim to check. Same fix applies cleanly
to the other helper usages (pause_turn test expects [] and the
sibling stop test expects ['stop'], both unaffected).
* Studio: cover Anthropic fast-mode edge cases
Adds 19 cases on top of the 9 in test_anthropic_fast_mode_and_refusal.
The base file pins the happy path; this file fills in the cliffs:
* Dated-snapshot prefix matching: claude-opus-4-7-2026-02-01 and
claude-opus-4-6-2026-02-01 still gate fast_mode through, while
claude-opus-4-5-2025-08-01 and claude-sonnet-4-6-2026-02-01 do not.
* Strict opt-in: a future claude-opus-4-8 or claude-opus-5 does NOT
auto-enable fast_mode -- the prefix tuple must be bumped explicitly
when a new family is whitelisted upstream.
* Beta-header merge: fast_mode coexists with code-execution-2025-08-25
and compact-2026-01-12 in one comma-separated anthropic-beta header
with no duplicates and no truncation. Pins the value to the exact
fast-mode-2026-02-01 docs token so a typo would fail CI.
* Non-destruction: fast_mode=None produces byte-identical outbound
body and headers to the version that omits the argument entirely.
Same for fast_mode=False. Guarantees the upgrade path is
non-breaking on existing Anthropic streams.
* Refusal stream ordering: the user-visible notice precedes the
finish_reason chunk so a streaming UI paints text before flipping
to content_filter. Refusal sentinel emitted exactly once. Notice
rides a normal content delta chunk with finish_reason still null.
Partial assistant deltas survive before the notice.
* Provider-side refusal coverage: a refusal on Sonnet (not just Opus)
still emits the notice + sentinel + content_filter mapping, since
refusal handling is not gated on fast-mode capability.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Persist fastMode, drop refused user message on retry
Two follow-ups on #5715:
1) sanitizeInferenceParams stripped fastMode. fastMode is in
PERSISTED_INFERENCE_PARAM_KEYS but the storage sanitizer only kept
numeric fields plus systemPrompt and trustRemoteCode, so the new
toggle was silently dropped on reload and on the
/api/chat/settings round-trip. Save it the same way trustRemoteCode
is saved.
2) Refusal recovery now also drops the triggering user turn.
Returning null from toOpenAIMessage on the assistant side left the
user prompt that caused the refusal in the outbound history, so
the very next request would re-trigger the same classifier.
Anthropic's refusal-handling guidance is explicit on this: remove
the refused turn AND the user message that triggered it before
the next call. Implemented via a pre-pass that pops the trailing
user message when an assistant carries the refusal sentinel.
Typecheck clean.
* Studio: out-of-band refusal signal + fast-mode prefix/usage/pricing fixes
The text sentinel for the Anthropic refusal drop signal was spoofable:
any assistant message containing the literal
<!--studio:anthropic-refusal--> would prune the prior user + assistant
pair on the next request. Move the signal onto a separate _toolEvent
chunk that the chat adapter latches into
assistant.metadata.custom.anthropicRefusal; assistant text can no
longer control the pruner.
Tighten the fast-mode model gate (backend + frontend) to require a "-"
family boundary so claude-opus-4-70 / claude-opus-4-7b style IDs do
not get speed: "fast" on a naive startswith match.
Use survivingMessages for the image / audio attachment scan so a
refused user turn does not gate or mis-attribute the next non-refused
turn.
Propagate Anthropic usage.speed onto the OpenAI-style usage chunk and
apply the documented 6x fast-mode multiplier in the cost calculator
(stacks with prompt-cache multipliers per the docs); expose the new
multiplier on the pricing snapshot for the UI tooltip.
Tests cover the tool-event chunk shape, the prefix-collision rejects,
usage.speed propagation, the 6x pricing math, and that the visible
refusal text carries no embedded sentinel.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shorten fast-mode and refusal comments for PR #5715
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: PDF / document attachments for Anthropic + OpenAI
Studio's local-GGUF chat already supports image attachments via the
`image_url` content part shape. PDFs and other documents had no
plumbing for the external-provider path: there was no normalised
content type the frontend could send that translated to Anthropic's
native `document` block or OpenAI's `input_file`.
Add a Studio-side `input_document` content part on assistant /
user messages with three shapes:
{type: "input_document",
file_data: "data:application/pdf;base64,<DATA>",
filename?: "name.pdf",
media_type?: "application/pdf"}
{type: "input_document",
file_url: "https://example.com/doc.pdf",
filename?: "doc.pdf"}
Translation:
- Anthropic Messages API: emits a `document` block with
`{source: {type:"base64", media_type, data}}` or
`{source: {type:"url", url}}`, plus an optional `title` from
`filename`. PDFs are extracted server-side by Anthropic per their
vision/document docs and counted toward input tokens.
- OpenAI Responses API: emits `{type:"input_file", file_data |
file_url, filename?}`. PDFs are extracted server-side.
Empty / unparseable `input_document` parts are silently dropped so
a malformed frontend payload can't blow up the request.
Tests:
- New `test_multimodal_document.py` with 6 cases pinning the
outbound body shape for base64 + URL inputs on both providers,
and the empty-part drop behavior on both.
- The Anthropic assertions strip the prompt-cache wrapper
(`cache_control:{type:ephemeral}` that the tail-message caching
layer adds) before comparing the document core fields, so this
test stays focused on the translation, not the caching layer.
Live verified end-to-end against both providers: a 363-byte
single-page "HELLO" PDF, base64-encoded, attached as a `document`
block to Opus 4.7 and as an `input_file` to gpt-5.5. Both models
correctly extracted the word "HELLO" from the PDF.
Follow-up (out of scope):
- Pydantic schema entry on ChatMessage.content for `input_document`
(today it rides through because ChatCompletionRequest uses
extra=allow). Will tighten when the frontend attach button lands.
- Frontend file-picker UX for non-image attachments on the external
provider path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: gate empty-content msg + skip empty data-URI payload
Gemini High + Codex P2 on PR #5689:
1. Anthropic translation appended an empty `anthropic_parts` array
when every part was dropped (e.g. user sent only an unparseable
input_document). Anthropic 400s on "messages.N.content: at least
one block is required". Skip the whole-message append when no
parts survived. The OpenAI Responses path already had the
equivalent guard, so this brings the two providers into parity.
2. `data:application/pdf;base64,` with no payload (or whitespace-only)
parses to an empty `source.data` string. Anthropic rejects that
with 400 as well. Skip the document block before constructing it.
Plus 2 new test cases pinning both behaviors:
- `test_anthropic_empty_only_document_drops_whole_message`: confirms
a turn whose only content is an unparseable input_document does
NOT make it onto the outbound `messages` array.
- `test_anthropic_empty_data_uri_payload_is_dropped`: confirms an
empty-payload data-URI is filtered out at translation time.
(Note re: gemini's other High note about adding `input_document` to
the Pydantic ContentPart union -- ChatCompletionRequest is configured
with `extra=allow` so the part rides through today. Tightening the
union belongs with the frontend attach-button PR that surfaces the
field; called out as follow-up in the PR description.)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: register input_document in ContentPart + builder
Reviewer caught that the translation code on the external_provider
side was unreachable from a real ChatCompletionRequest:
- ContentPart is a discriminated Union of (text, image_url) only, so
any `{"type": "input_document", ...}` part was rejected by Pydantic
at request parsing with a discriminator error before the helper
could see it.
- _build_external_messages in routes/inference.py only walked text
and image_url parts, so even with a permissive schema the document
parts would have been silently dropped instead of forwarded to
the per-provider translator.
Fixes:
- Add InputDocumentContentPart with optional file_data / file_url /
filename / media_type and Tag("input_document") on the Union.
- Extend _build_external_messages to pass input_document through as
a plain dict for vision-capable providers (so external_provider's
existing Anthropic `document` and OpenAI Responses `input_file`
mappers actually run) and strip them on non-vision providers.
Tests added: schema accepts input_document, builder passes it to
vision providers, builder strips it on non-vision providers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: validate file_data before preferring over file_url
Codex P2 caught that the OpenAI input_document translator treats any
truthy file_data as valid and never falls back to file_url. That
means a malformed `data:application/pdf;base64,` (empty payload) or
a whitespace-only data URI gets forwarded as `file_data=""` and
400s the whole turn, AND silently discards a perfectly recoverable
file_url on the same part.
Mirror the Anthropic-side guard onto the OpenAI Responses path:
treat any "data:" URI with no actual base64 payload as missing and
fall through to file_url. Standalone-empty data URIs (no fallback)
are dropped entirely instead of being sent to the wire.
Tests added: empty data URI + valid file_url -> file_url wins,
whitespace-only data URI + valid file_url -> file_url wins,
empty data URI without fallback -> part is dropped.
* Address review: Anthropic side also falls back to file_url on empty data URI
Codex P2 follow-up to my earlier fix: I added the empty-data-URI ->
file_url fallback to the OpenAI Responses translator but missed
the Anthropic translator, which still `continue`d on empty payloads
and discarded an otherwise valid file_url on the same part. Result:
when the frontend supplied both file_data (placeholder / broken)
AND a working file_url, Anthropic silently lost the attachment;
when the message contained only that part, the whole message could
be dropped before reaching the wire.
Mirrored the OpenAI guard: any "data:" URI with no actual base64
payload (`data:application/pdf;base64,` or whitespace-only) is
treated as missing, and the file_url branch takes over. The
all-parts-dropped guard further down already handles the
no-fallback case.
Tests added: empty data URI + valid file_url -> URL source on the
wire with the filename preserved; whitespace-only data URI + valid
file_url -> URL source on the wire.
* Address review: gate input_document passthrough to anthropic + openai
Codex P1: only `_stream_anthropic` and `_stream_openai_responses`
have explicit translation logic for input_document parts (the former
maps to {type:"document", source:...}, the latter to
{type:"input_file", file_data|file_url}). Every other provider
(gemini / mistral / kimi / openrouter / deepseek / qwen / custom)
goes through the generic /chat/completions passthrough that forwards
`messages` verbatim, so any input_document part on a non-vision
route on those providers would 400 with an unknown content_part
type.
Added `_INPUT_DOCUMENT_PROVIDERS = frozenset({"anthropic", "openai"})`
constant and gated the pass-through branch on `provider_type in
_INPUT_DOCUMENT_PROVIDERS`. Every other provider strips the part
(text content survives). Threaded provider_type through from
_proxy_to_external_provider's call site.
Tests updated: vision + provider in {anthropic, openai} still
forwards; six unmapped providers (gemini/mistral/kimi/openrouter/
deepseek/qwen) strip the part; missing provider_type strips
defensively. The existing non-vision drop test still passes.
* Fix stale web_fetch tool-version assertion after merging main
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: wire OpenAI Responses server-side context compaction
The OpenAI Responses API accepts a `context_management` field that
enables server-side compaction. When the rendered prompt crosses the
configured threshold, the API runs a server-side compaction step and
the request continues against the compacted prefix. No beta header
and no dated version pin are required, per the docs.
Changes:
- Add `compaction_threshold: Optional[int]` (ge=1_000, le=2_000_000)
to ChatCompletionRequest. Thread through `routes/inference.py` ->
`stream_chat_completion` -> `_stream_openai_responses`.
- In `_stream_openai_responses`, when threshold is set AND the base
URL points at cloud OpenAI (api.openai.com), attach
`context_management: [{type:"compaction", compact_threshold:N}]`
to the outbound body. Non-cloud bases (ollama, llama.cpp, "custom"
presets) silently drop the field so we don't 400 those servers.
- Add `test_openai_compaction.py` with 4 cases: cloud OpenAI sets
the field verbatim, low-threshold probe passes through (we don't
clamp on the OpenAI side because the API accepts whatever),
non-cloud base drops the field, omitted threshold leaves body
untouched.
Live verified against the real OpenAI API on gpt-5.5:
`context_management:[{type:"compaction", compact_threshold:200000}]`
returns 200 with no error.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: accept Azure OpenAI base URLs + raise compaction floor
Two reviewer follow-ups on the OpenAI compaction PR:
1. The `is_openai_cloud = "api.openai.com" in self.base_url` check
excluded Azure OpenAI Foundry, even though Azure exposes the
same /v1/responses extensions (context_management,
prompt_cache_retention, container shell). Users on Azure saw
their compaction toggle silently no-op. Broadened the check to
also match `*.openai.azure.com` and made it case-insensitive so
URLs copy-pasted from the Azure portal still resolve. Non-cloud
OpenAI-compatible servers (ollama / llama.cpp / vLLM / "custom"
preset) still fall outside the gate.
2. The schema floor on compaction_threshold was ge=1_000, which is
well below the upstream Responses API's effective minimum
(vercel/ai#12486, langchain-ai/langchain#35464 report
`compact_threshold is not enabled` 400s on Azure at 100k; cloud
uses 200k as the canonical example). Raised the floor to 10k
so obvious typos surface as a clean 422 from FastAPI rather than
an opaque upstream 400 the user has to debug from the SSE
stream.
Tests added: Azure base URL carries both context_management and
prompt_cache_retention; mixed-case Azure URLs match; schema rejects
9_999 and accepts 10_000.
* Address review: drop schema-level compaction floor (cross-provider regression)
Codex P2 follow-up on the previous floor bump: ge=10_000 was
enforced globally at the ChatCompletionRequest layer, but the field
is documented as a no-op on every non-cloud OpenAI base and every
non-OpenAI provider. With the global floor, an Anthropic / ollama
/ llama.cpp / custom request that happens to carry compaction_threshold
below 10k was rejected with 422 at request validation time instead
of being silently ignored as the description promised.
Reverted the schema floor to ge=1 (any positive int) and rewrote
the description to call out per-provider routing: OpenAI cloud's
effective floor is around 200k and surfaces upstream 400s below
that; _stream_anthropic clamps sub-50k values up. Per-provider
helpers stay the single source of truth on the floor.
Test updated to pin: zero is still rejected, but every positive
value (1, 5_000, 9_999, 10_000, 200_000) passes schema validation.
* Address CodeQL: hostname-anchored OpenAI cloud detection
CodeQL py/incomplete-url-substring-sanitization fired on
`".openai.azure.com" in _base`. An attacker who controls the
configured base_url could slip cloud-only request body fields
(prompt_cache_retention, context_management compaction, container
shell) to an arbitrary server with:
https://evil.com/api.openai.com/v1https://api.openai.com.attacker.com/v1https://attacker.com/.openai.azure.com/v1https://my-resource.openai.azure.com.attacker.com/openai/v1
Replaced the substring check with a `_is_openai_family_cloud`
helper that runs urllib.parse.urlparse on the URL and matches the
lowercased hostname exactly (`api.openai.com`) or via `endswith`
on the leading-dot suffix (`.openai.azure.com`). Both halves are
host-anchored so path / fake-subdomain bypasses fail.
Test added: every attacker-controlled bypass shape above must NOT
carry context_management OR prompt_cache_retention on the wire.
Existing Azure and openai.com tests still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: scope compaction_threshold description to OpenAI on this branch
Codex P2: the field description on this PR mentioned Anthropic
compaction behavior, but the Anthropic wiring lives on PR 5686
(separate branch). On feat/openai-compaction alone, _stream_anthropic
has no compaction_threshold parameter, so the field is silently
ignored for Anthropic requests and the doc claim was misleading.
Trimmed the description to OpenAI cloud + Azure Foundry only on
this branch. PR 5686 already re-adds the Anthropic clause via its
own change, so the rebase / merge order on main will land the
combined description naturally once both PRs ship.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: wire Anthropic server-side context compaction
Anthropic ships server-side context compaction as a beta
(`compact-2026-01-12`). When the rendered prompt crosses the
configured input-token threshold, Anthropic runs an extra LLM pass
that summarises older turns and the request continues against the
compacted prefix. The response carries the original top-level fields
plus a new `context_management` block (with `applied_edits`) and
`usage.iterations[]` accounting per pass.
Per the docs the feature is currently supported on Opus 4.6, Opus 4.7,
Sonnet 4.6, and Mythos preview. The minimum threshold is 50k tokens;
under-50k requests 400.
Changes:
- Add prefix gate + helper `_anthropic_supports_compaction` plus
constants `_ANTHROPIC_COMPACTION_PREFIXES`, `_ANTHROPIC_COMPACTION_BETA`,
`_ANTHROPIC_COMPACTION_TYPE`, `_ANTHROPIC_COMPACTION_MIN`.
- Add `compaction_threshold: Optional[int]` to ChatCompletionRequest
(50k ge bound, 2M le bound). Thread through `routes/inference.py`
-> `stream_chat_completion` -> `_stream_anthropic`.
- In `_stream_anthropic`, when threshold is set AND the model
accepts compaction, attach `context_management.edits[{type:
"compact_20260112", trigger:{type:"input_tokens", value:N}}]` to
the outbound body. Sub-50k values are clamped up to 50k to keep
the request well-formed.
- Refactor the anthropic-beta header builder to merge any combination
of `code-execution-2025-08-25` + `compact-2026-01-12` flags into
one header value. Unrelated betas added at the registry level still
pass through.
- Add `test_anthropic_compaction.py` with 16 cases: gate matrix
(every doc-listed model), correct body shape, threshold clamping,
beta header merge with code execution, silent no-op on unsupported
models, omitted-threshold pass-through.
Live verified end-to-end against the real Anthropic API:
`compact_20260112` accepted on Opus 4.7, response carries
`context_management.applied_edits` + `usage.iterations[]` as
documented. (The first WebFetch-summarised version of these docs
suggested `compact_20260120`; the actual API only accepts
`compact_20260112`, matching the beta-header date. Worth pinning
behind a test so a future doc update can't drift back.)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: drop ge=50_000 clamp + parse usage.iterations[]
Two reviewer follow-ups on the compaction PR:
1. Pydantic ge=50_000 on compaction_threshold was dead code.
FastAPI rejected sub-50k threshold values with a 422 before the
`max(int(...), _ANTHROPIC_COMPACTION_MIN)` clamp in
_stream_anthropic could ever fire. Relaxed the floor to ge=1 so
the in-helper clamp actually does its job; the schema comment
now explains why this is intentional. Added a regression test
that posts a value of 1 and 49_999 through the real request
schema.
2. Anthropic publishes per-iteration token counts in
`usage.iterations[]` whenever a fresh compaction has run, and
the top-level input_tokens / output_tokens cover only the
`message` iteration -- billing must add the compaction
iterations on top. Aggregate compaction iteration tokens into
`last_usage["compaction_input_tokens" / "compaction_output_tokens"]`
so the cost surface (PR 5690) can read them without re-walking
the array, and surface both figures in the closing stream
summary log. Added two tests: one that pins the aggregation on a
compacted turn and one that pins `None` when no fresh
iterations land (so re-applied compaction blocks don't double-bill).
Sourcing: https://platform.claude.com/docs/en/build-with-claude/compaction
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: round-trip Anthropic compaction blocks across turns
Codex P1: once context_management is enabled and Anthropic runs
server-side compaction mid-stream, the response carries a
`{type:"compaction", content:"<summary>"}` content block on the
assistant message. The translator only handled text_delta and
input_json_delta on content_block_delta, so the compaction block
was silently dropped. Worse, the request schema's ContentPart
discriminated Union didn't accept `type:"compaction"`, and
_build_external_messages didn't pass it through, so even a
hand-crafted assistant message carrying the block would 422 at
parse time. Net result: Anthropic re-compacted from scratch on
every subsequent turn, wasting input tokens and reasoning budget.
End-to-end backend wiring of the round-trip:
1. SSE translator. _stream_anthropic now tracks a `current_compaction`
state slot. content_block_start with type=="compaction" seeds it
(Anthropic may include the summary on the start event AND/OR
stream it via text_delta events on the same block index --
handle both). text_delta inside a compaction block routes into
the compaction buffer instead of the user-visible content
stream, since the summary is opaque internal state, not
assistant prose. content_block_stop emits a `compaction_block`
tool_event carrying the full summary so the chat-adapter can
persist it. compaction_blocks_seen is surfaced in the closing
summary log.
2. Pydantic schema. Added CompactionContentPart with Tag("compaction")
on the ContentPart Union so requests carrying the block parse
cleanly. Required `content` field with a docstring pointing at
the Anthropic docs.
3. Message builder. _build_external_messages forwards compaction
parts on both vision and non-vision paths; the per-provider
stream helper decides whether to forward to the wire (Anthropic
does; other providers ignore the part). When a non-vision route
ends up with a single text part, collapse back to a string
so providers that don't accept content arrays still get the
expected shape.
4. _stream_anthropic outbound translator. {type:"compaction"} parts
on an assistant message land on the wire verbatim. Empty/missing
`content` is skipped so a malformed stored block can't 400
Anthropic.
Tests added (5): stream emits compaction_block tool event with the
summary intact; user-visible content stream does NOT carry the
summary text; outbound body forwards compaction parts verbatim on
the next turn; Pydantic schema accepts the part; builder passes
it through on both vision and non-vision provider routes.
Frontend follow-up: the chat-adapter needs to persist the
compaction_block tool_event onto the stored assistant message so
turn N+1 includes it in payload.messages. Pinned in the PR
description.
Sourcing: https://platform.claude.com/docs/en/build-with-claude/compaction
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: gate compaction-part passthrough to Anthropic only
Codex P1: my previous round-trip change preserved {type:"compaction"}
parts on every provider route in _build_external_messages. That
meant a chat history with prior compaction state silently leaked
the Anthropic-specific block to OpenAI/DeepSeek/Mistral/Gemini/
Kimi/OpenRouter on a provider switch, where generic
/chat/completions passthrough hands the unknown content type to
the upstream API and 400s the whole turn.
Added a `provider_type` kwarg to _build_external_messages and
gated the compaction forwarder on `provider_type == "anthropic"`.
Every other value (including the legacy None for callers that
don't pass it yet) strips the part. The Anthropic stream helper
still maps it to a native `compaction` block on the wire.
Threaded provider_type through from _proxy_to_external_provider's
call site.
Tests updated: vision + provider="anthropic" still forwards; six
non-anthropic providers strip the part; missing provider_type
strips defensively; non-vision + anthropic still forwards; non-vision
+ non-anthropic collapses back to a text string.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: wire Anthropic web_fetch server-side tool
Studio's Anthropic passthrough only forwarded web_search and
code_execution when enabled_tools was set. Asking Claude through Studio
to fetch a URL produced no fetch (the tool was not in the outbound
tools array), so users had to fall back to web_search even when they
already had the exact URL they wanted.
This change opts in web_fetch_20250910 when enabled_tools contains
"web_fetch". The new tool entry is appended alongside any existing
web_search / code_execution entries:
{"type": "web_fetch_20250910", "name": "web_fetch", "max_uses": 5}
No anthropic-beta header is required (web_fetch is GA); the existing
code-execution-2025-08-25 flag continues to merge cleanly when both
tools are enabled in the same turn.
SSE translation mirrors the web_search path. A `server_tool_use` block
with name="web_fetch" emits a `tool_start` _toolEvent carrying the
URL the model asked to fetch; the matching `web_fetch_tool_result`
block emits a `tool_end` _toolEvent whose result string follows the
Title / URL / Snippet shape parseSourcesFromResult on the frontend
already expects, so the source pill renders identically. Error blocks
(`web_fetch_tool_error`) are surfaced as "Error: <error_code>" matching
the code_execution error path.
The final "Anthropic stream complete" log line picks up web_fetch_
requested / web_fetch_invocations / web_fetch_urls so support reports
of "the model did not fetch anything" can be triaged from the log.
Verified end to end against claude-haiku-4-5 with
`enabled_tools=["web_fetch"]`: the model emitted tool_start with
url=https://example.com and tool_end with the page Title + URL +
Snippet, plus the assistant message correctly read back "Example
Domain" as the title.
Tests:
- 5 new unit tests in test_anthropic_web_fetch.py covering tool
registration, the combined web_search + web_fetch + code_execution
request body, the pill-off case, and SSE translation for both
success and error paths.
- All 242 existing Anthropic + OpenAI provider tests still pass.
The enabled_tools field description in models/inference.py is updated
so OpenAPI consumers see the new option.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* web_fetch: title fallback to URL, log parse failures, drop dead checks
Three review nits on the previous commit:
1. `_format_web_fetch_result` left `title` empty when Anthropic omitted
`document.title`. The frontend `parseSourcesFromResult` only emits
a source pill when both `Title:` and `URL:` lines are present, so
fetches against pages without an HTML title tag silently lost
their citation in the UI. Fall back to `title = title or url`,
matching the web_search formatter.
2. The broad `except Exception` around `json.loads(buffer)` for the
web_fetch input swallowed the failure with no trace. Log at debug
so a malformed partial_json buffer can be triaged from the server
log without changing behavior.
3. `inner` was already sanitised to a dict at the matching
content_block_start and `_format_web_fetch_result` always returns
a non-empty string (defaulting to "(fetch complete)"), so the
`isinstance(inner, dict) else {}` guard and the
`result_text or "(fetch complete)"` fallback at the emit site
were dead code. Removed.
Added a test exercising the titleless path so the fallback stays
covered.
* chat-adapter: emit source pills for web_fetch tool calls
`parseSourcesFromResult` was only wired up for tool calls where
`toolName === "web_search"`, so the Title / URL / Snippet block the
backend formatter emits for `web_fetch_tool_result` never reached the
source-pill renderer. Users saw the raw tool result in the tool card
but the dedicated source-pill row at the message tail stayed empty.
Both web_search and web_fetch ship the same text shape today, so the
fix is to broaden the gate.
* Address review: wire web_fetch from Search pill + fix pause_turn truncation
Two reviewer follow-ups on the Anthropic web_fetch PR:
1. The backend tool wiring landed but the frontend chat-adapter
never put `web_fetch` in `enabled_tools`, so toggling the Search
pill only ever attached `web_search` -- web_fetch was unreachable
from the UI. Added providerSupportsBuiltinWebFetch() (Anthropic
today) and paired the entry with the existing Search pill, since
the canonical workflow is "search returns URLs, fetch reads
them" and there is no separate UI toggle yet.
2. `pause_turn` from Anthropic's stop_reason vocabulary fell through
the finish_reason map's "stop" default, which the OpenAI-format
client renders as end-of-message and truncates the answer. Per
the docs pause_turn means "Claude paused a long server-tool
turn (web_search / web_fetch) and will resume". Mapped to None
and skipped the chunk emission so the SSE stream still ends with
[DONE] on message_stop but no terminal finish_reason lands on
the client. While there: added explicit mappings for `tool_use`
(-> tool_calls) and `refusal` (-> content_filter) which were
also falling through to "stop".
Tests added: pause_turn emits no finish_reason, end_turn still
emits "stop", refusal maps to "content_filter".
Sourcing: https://platform.claude.com/docs/en/api/messages#response-stop-reason
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: support Anthropic 1h cache TTL via prompt_cache_ttl field
Anthropic exposes two ephemeral cache pools per request: the default
5-minute pool, and a 1-hour pool selected by attaching `ttl:"1h"` to
the `cache_control` marker. 1h writes are billed at 2x base input vs
1.25x for 5m, but reads stay at 0.1x for both, so a single extra read
landing more than 5 minutes after the write pays off the premium.
Studio hardcoded the 5m pool via `cache_control: {type:"ephemeral"}`
on both breakpoints. For chats with multi-minute idle gaps (people
juggling tabs, long-running tool calls between turns), the cache
expires before the next turn and every read becomes a cache_creation,
not a cache_read -- exactly the case where the 1h pool wins.
Changes:
- Add `prompt_cache_ttl: Optional[Literal["5m", "1h"]]` to
ChatCompletionRequest. Default (None) preserves today's 5m behavior.
- Thread through `routes/inference.py` ->
`stream_chat_completion` -> `_stream_anthropic`.
- Build a shared `cache_marker` dict in `_stream_anthropic`; attach
`ttl` only when the request asks for one of the two valid values.
Unknown TTL strings are silently dropped to avoid sending malformed
markers (the upstream API would 400).
- Apply the same marker to both existing breakpoints (system block at
line 1175 and the latest-message tail at line 1198 / 1213) so the
pool selection is consistent across the whole prefix.
- Add `test_anthropic_cache_ttl.py` with 11 parametrized cases
pinning the outbound body shape: omitted -> default marker;
explicit `5m`/`1h` -> ttl field set; unknown values dropped;
caching off -> no markers at all.
Verified upstream that `cache_control: {type:"ephemeral", ttl:"1h"}`
is accepted by the Anthropic API today; no beta header required.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Relax prompt_cache_ttl to Optional[str] (Codex P1)
Declaring `prompt_cache_ttl` as `Optional[Literal["5m", "1h"]]` made
FastAPI/Pydantic 422 the request before _stream_anthropic could even
see the field. The whole point of the downstream drop-unknown-values
behaviour was to keep a stale frontend from crashing the request;
the strict Literal at the request layer defeated that.
Loosen the schema to Optional[str]; the existing in-helper guard
already restricts forwarded values to {"5m", "1h"} (everything else
is silently dropped). Test suite stays unchanged -- the bogus-value
cases in test_anthropic_cache_ttl.py already pass arbitrary strings
through and assert they are dropped before the wire.
* Address review: confirm extended-cache-ttl beta header is GA
Reviewer asked whether the 1h cache TTL still requires the
`extended-cache-ttl-2025-04-11` anthropic-beta header. Investigated:
- Live-tested api.anthropic.com on claude-opus-4-7 (2026-05-22)
with cache_control={type:"ephemeral", ttl:"1h"} and NO beta
header. Got status 200 and ephemeral_1h_input_tokens populated
on the create turn, plus cache_read_input_tokens populated on
the reuse turn.
- Cross-checked the current prompt-caching docs: no mention of
any beta header on the 1h TTL path.
Conclusion: the gate has been promoted to GA. The code already
does not send the beta header (the cache_marker dict only carries
`type`/`ttl`), so no wire change is needed. Pinned the contract
with two regression tests that assert the header is NOT on the
outbound request, and added a docstring note explaining the
investigation outcome so a future reader does not re-add it.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix: Claude Code Anthropic API tool compatibility
* fix: merge Anthropic server tool selections
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix anthropic /v1/messages server-tool alias misrout
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: harden Anthropic /v1/messages tool validation
* fix: dispatch Anthropic server tools by only
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: reject Anthropic client tools missing 'name' at boundary
AnthropicTool.name was relaxed to Optional[str] to accommodate server-tool
declarations. A client tool with input_schema but no name now parses but
is silently dropped by anthropic_tools_to_openai, leaving tool calling
disabled. Surface as 400 instead.
* fix: reject Anthropic client tools with empty 'name'
isinstance(name, str) accepts an empty string, but anthropic_tools_to_openai
drops entries via 'if not name', producing the same silent-disable
fallthrough the boundary check is meant to prevent. Tighten to also reject
empty name.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
* studio: add --spec-draft-n-max toggle for MTP speculative decoding
Surface llama-server's --spec-draft-n-max as a first-class
LoadRequest field so users can tune the MTP draft tree size from
the chat settings panel. Default behaviour is unchanged: when the
caller omits spec_draft_n_max, the existing platform defaults still
apply (6 on GPU, 3 on CPU/Mac).
Why this matters: on context-constrained loads the draft KV cache
competes with the target model's KV cache for VRAM. Lowering
spec_draft_n_max reduces that pressure, lets a larger user context
fit, and recovers throughput; raising it pays off when draft
acceptance is high enough to amortise the extra cache.
Backend
- LoadRequest gains an optional spec_draft_n_max: int (1..16).
- LlamaCppBackend.load_model accepts and persists the override on
self._spec_draft_n_max, used in place of the hardcoded 6/3 in the
MTP emit branch.
- LoadResponse and InferenceStatusResponse echo the active value
(None when the platform default is in effect) so the UI can
hydrate the input on refresh.
- _already_in_target_state and _request_matches_loaded_settings
compare spec_draft_n_max alongside speculative_type so a value
change triggers a reload rather than no-op'ing.
- strip_shadowing_flags now strips inherited --spec-* extras when
either speculative_type or spec_draft_n_max is in fields_set, so
an inherited --spec-draft-n-max cannot last-wins-override a fresh
request's first-class field.
Frontend
- LoadModelRequest, LoadModelResponse, InferenceStatusResponse
TypeScript shapes get spec_draft_n_max.
- chat-runtime-store gains specDraftNMax / loadedSpecDraftNMax and
a setter, hydrated from /v1/status and /v1/load.
- chat-settings-sheet renders a "Draft Tokens" numeric input
directly under the Speculative Decoding switch when that switch
is on. Toggling the switch off clears the override; the Reset
button restores the loaded value.
Tests
- Four new regression tests cover _already_in_target_state with
matching / mismatching / non-MTP / unset spec_draft_n_max.
- Existing test_llama_server_args.py and test_llama_cpp_mtp_detection.py
green: 141 passed locally.
* studio: add --spec-draft-p-min and --spec-draft-p-split to spec strip set
llama.cpp server documents --spec-draft-p-min (default 0.75, min draft
acceptance probability) and --spec-draft-p-split (default 0.10). Both
are first-class spec-decoding knobs that should travel with the rest
of the --spec-* family when an Apply re-sets speculative_type, so an
inherited override doesn't leak across a fresh load.
* studio/tests: skip MTP capability-probe tests on Windows
The four probe_server_capabilities tests use a bash stub written to
tmp_path/llama-server, which Windows' subprocess can't execute
directly (no shebang resolution, .bat / .cmd would be needed). Mark
them skipif sys.platform == 'win32' so the rest of the MTP plumbing
suite stays green on Windows CI. Unix coverage is unchanged.
* studio: lower MTP GPU default --spec-draft-n-max from 6 to 2
Bench on B200 / Qwen3.6-27B-MTP-GGUF UD-Q4_K_XL across five prompt
types (essay, code, story, math, science) with greedy temp=0:
prompt OFF n=1 n=2 n=3 n=6
essay 79.1 93.4 93.8 84.7 64.6
code 79.1 104.4 116.6 113.5 103.0
story 79.1 99.2 105.7 101.8 88.9
math 79.1 100.8 110.8 111.8 98.2
science 79.1 100.1 110.8 110.8 102.9
The previous hardcoded GPU default of 6 was 17% SLOWER than spec-off
on the essay prompt (64.6 vs 79.1 t/s) and 11-50% slower than n=2 on
the rest. n=2 wins on 4/5 prompts with a 1.18x-1.47x speedup vs OFF;
n=3 wins on the math prompt by a hair. n=6 collapses once acceptance
rate drops past n=3 -- wasted draft decode dominates the per-step
budget.
Matches the dataset README ("n_max=2 is the sweet spot for 36 of 42
quants"). Keeps CPU/Mac default at 3, which empirically tracks the
narrower ngram+MTP chained budget on those platforms.
Users who want the old behaviour can pass spec_draft_n_max in
LoadRequest (the toggle this PR also adds) or --spec-draft-n-max via
llama_extra_args.
* studio: skip MTP auto-promote on sub-2B models, backfill chat usage
Two MTP-visibility fixes uncovered while bisecting llama.cpp post-#22673
on Qwen3.6-27B-MTP-GGUF UD-Q4_K_XL on B200.
Size gate. Direct llama-server bench (no Studio measurement loop) at
n_predict=192 across 9 prompts shows MTP regresses vs spec-off on
sub-2B dense models because draft cost exceeds savings:
Qwen3.5-0.8B Q4_K_XL GPU: 452.0 OFF -> 283.4 t/s n=2 (0.63x)
CPU: 84.5 OFF -> 64.9 t/s n=3 (0.77x)
Qwen3.5-4B Q4_K_XL GPU: 241.0 OFF -> 258.2 t/s n=2 (1.07x)
Qwen3.5-9B Q4_K_XL GPU: 201.6 OFF -> 228.9 t/s n=2 (1.14x)
Qwen3.5-27B Q4_K_XL GPU: 78.8 OFF -> 113.6 t/s n=2 (1.44x)
Qwen3.6-27B Q4_K_XL GPU: 78.8 OFF -> 113.6 t/s n=2 (1.44x)
Qwen3.6-35B-A3B Q4 GPU: 192.3 OFF -> 223.2 t/s n=2 (1.16x)
The 2B inflection is sharp. Skip auto-promote to draft-mtp when the
identifier reports <2.0B params; users can still force via --spec-type
or the Speculative Decoding toggle. Mirror the gate in the
reload-skip check so a sub-2B reload-with-default does not bounce a
spec-off backend.
Chat-completions usage. llama-server's final SSE chunk emits both an
OpenAI-style usage block and a custom timings block. timings.predicted_n
is always populated, but usage.completion_tokens is zero on some
server builds. The Studio chat UI computes generation t/s from
meta.usage.completion_tokens / totalStreamTime, so a zero
completion_tokens makes the UI fall back to wall-clock time
(including SSE / proxy / template overhead) which dilutes MTP gains and
makes ON look the same as OFF.
Add _backfill_usage_from_timings: if usage.completion_tokens is missing
or zero AND timings has predicted_n/prompt_n, synthesize a complete
usage dict. Apply at the streaming metadata yield in
generate_chat_completion and at the three accumulator/yield sites in
generate_chat_completion_with_tools so per-iteration counts are not
silently lost across tool calls.
Tests cover both the gate (sub-2B skips, 2B+ promotes) and the
backfill (zero usage filled, real usage preserved, empty timings
passthrough).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: probe + emit legacy ngram-mod flags for pre-rename llama-server
llama.cpp upstream renamed the ngram-mod tuning knobs:
--draft-max -> --spec-ngram-mod-n-max (and --spec-draft-n-max)
--draft-min -> --spec-ngram-mod-n-min (and --spec-draft-n-min)
--spec-ngram-size-n -> --spec-ngram-mod-n-match
The new names are real flags on post-rename builds and stub removal
entries on the same builds (with description "argument has been
removed"). Pre-rename builds only carry the legacy names as real
flags. Studio was emitting the new names unconditionally, so a user
running a pre-rename llama-server (e.g. an older prebuilt or a
hand-installed binary) would see "unknown argument" errors when the
ngram-mod path engages, or silent drop of the ngram knobs.
Extend `probe_server_capabilities` to parse the help text into
per-flag description blocks and tell real flags apart from removal
stubs by the "argument has been removed" marker. Add three new probe
fields: `ngram_mod_flavor` ("new" / "legacy" / None),
`supports_ngram_mod`, and `spec_draft_n_max_flag` (the actual n_max
flag the binary accepts). Cached by (path, mtime) the same way as
`mtp_token`.
Add `_build_ngram_mod_flags(caps, ...)` that picks the right flag
set, returning [] when neither is usable so callers can drop ngram
chaining entirely on minimal binaries.
Wire both call sites to use the probe-driven flag set:
- CPU/Mac MTP comma-chain (--spec-type ngram-mod,draft-mtp) emits
legacy or new knobs as appropriate. If neither set is available,
degrade to MTP-only (warn but still engage spec).
- Standalone --spec-type ngram-mod branch uses the same helper.
Tests cover post-rename detection, legacy detection, removal-stub
discrimination, minimal-binary case, and all three branches of
`_build_ngram_mod_flags` plus custom n_match/n_min/n_max values.
Verified against three real binaries (Studio bundled 726704a, my
build of 45b455e HEAD, and the MTP merge baseline 2555826) all
correctly reporting ngram_mod_flavor=new.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: sub-3B MTP falls back to ngram-mod, not off
Earlier sub-2B gate disabled speculative decoding entirely for tiny
dense MTP models because the MTP draft head's per-token cost exceeds
the acceptance savings at that scale. The "fully off" fallback was
conservative -- ngram-mod has near-zero idle cost on diverse content
and consistently outperforms both off and draft-mtp at sub-3B.
Clean-methodology bench (each of 9 distinct prompts run once after
two unrelated warmup prompts so the ngram-mod hash pool is
realistically populated but never holds the exact deterministic
output we're about to measure):
Q4_K_XL on B200:
0.8B OFF=451 draft-mtp n=2=263 (0.58x) ngram-only=498 (1.10x)
2B OFF=377 draft-mtp n=2=308 (0.82x) ngram-only=369 (1.00x)
4B OFF=240 draft-mtp n=2=260 (1.08x) -- 4B+ wins with MTP
Q4_K_XL on x86 48 cores:
0.8B OFF= 80 chained n=2= 69 (0.86x) ngram-only= 95 (1.19x)
2B OFF= 62 chained n=2= 51 (0.83x) ngram-only= 63 (1.01x)
4B OFF= 31 chained n=2= 41 (1.33x)
Change:
- Raise the MTP-skip threshold from 2.0B to 3.0B (2B falls below it).
- When skipping the MTP head, fall back to --spec-type ngram-mod via
the probe-driven _build_ngram_mod_flags helper. Works on both
post-rename and pre-rename llama-server builds.
- If the binary advertises neither ngram-mod flavor, fall back to
spec-off (older binaries that don't support ngram-mod at all).
- Mirror the same fallback in _already_in_target_state so a sub-3B
reload-with-default does not bounce a ngram-mod backend.
Tests updated: monkeypatch probe_server_capabilities so the gate
behavior is deterministic regardless of which llama-server happens
to be on the host. +1 new test for the "binary has no ngram-mod
support" branch; renamed prior 2B/0.8B tests to reflect new semantics.
This generalizes the size gate to be probe-driven instead of a hard
"disable spec" branch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: 5-mode Speculative Decoding dropdown (Auto / MTP / Ngram / MTP+Ngram / Off)
Replace the Chat Settings Speculative Decoding on/off Switch with a 5-option
Select. Auto preserves today's platform-aware resolver (MTP on MTP GGUFs,
ngram-mod fallback for sub-3B, --spec-default for non-MTP). The other 3 modes
force the user's choice on BOTH GPU and CPU: MTP emits draft-mtp only (no
ngram chain on CPU), Ngram emits ngram-mod only, MTP+Ngram emits the
ngram-mod,draft-mtp chain on both platforms. Off is the existing fully-off
state, kept so the Switch's "disable" capability isn't lost.
Backend
- New module-level _canonicalize_spec_mode(value) maps any accepted input
(canonical, legacy "default" / "draft-mtp" / "ngram-mod" / "ngram-simple",
or comma-chained "ngram-mod,draft-mtp") onto one of auto / mtp / ngram /
mtp+ngram / off / ngram-simple / None. Lets external callers and old
persisted UI state round-trip without breaking.
- LlamaCppBackend grows a _requested_spec_mode field + requested_spec_mode
property storing the canonical UI mode the user requested. Status
responses round-trip this instead of the resolved internal flag, so the
dropdown restores the picked value after reload / refresh (Auto on a 27B
MTP GGUF resolves to draft-mtp internally but the dropdown stays on
"Auto").
- The resolver block in load_model is extracted into a unit-testable
_build_speculative_flags method. Forced MTP / MTP+Ngram on a sub-3B or
non-MTP GGUF logs a warning and engages anyway (user override > the
Auto-path sub-3B fallback).
- _already_in_target_state and routes/inference._request_matches_loaded_settings
now compare canonical-requested mode, dropping the old auto-promotion
mirror. spec_draft_n_max still gates on the resolved spec so Auto + a
changed n_max still bounces a reload.
Frontend
- chat-settings-sheet.tsx: Switch swapped for Select modeled on the KV
Cache Dtype Select. Items: Auto / MTP / Ngram / MTP+Ngram / Off. Draft
Tokens input only visible when speculativeType is "mtp" or "mtp+ngram".
- chat-runtime-store.ts: initial value flips from "default" to "auto".
- use-chat-model-runtime.ts normalizeSpeculativeType mirrors the backend
canonicaliser so persisted "default" / "draft-mtp" / "ngram-mod" / chain
values hydrate to the right dropdown option.
- types/api.ts: docs the canonical wire vocabulary.
Tests
- 53 new assertions in test_llama_cpp_mtp_detection.py: full
_canonicalize_spec_mode table, a 23-row resolver matrix across
(requested mode) x (GPU/CPU) x (model size class), plus n_max override,
user-extra-args precedence, requested-mode round-trip, and graceful
degrade on an outdated llama-server without an MTP token.
- 165 existing backend tests still green. 218 total in the MTP /
server-args / reload-inheritance suite.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: reset Speculative Decoding to Auto on model switch
When the user switches from model A to a different model B, clear the
runtime store's speculativeType + specDraftNMax (and their loaded*
shadows). The new load request then carries null, the backend
canonicalises that to "auto", and its platform-aware resolver runs
fresh for the new model.
Without this, a non-MTP model loaded with "Off" carried the Off choice
into a subsequent MTP load, suppressing MTP auto-promotion (and the
sub-3B ngram-mod fallback) until the user manually opened settings and
flipped the dropdown back to Auto. The clean-sweep deep probe caught
it as anomaly A-1.
The reset only fires when currentCheckpoint != modelId, so a
same-model reapply or forceReload still honours the user's current
spec choice. End-to-end probe on Qwen3.5-4B-GGUF (non-MTP, Off) ->
Qwen3.5-0.8B-MTP confirms: dropdown shows Auto, /api/inference/status
returns speculative_type=auto, studio.log shows the Auto sub-3B
fallback emitted --spec-type ngram-mod.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: warn when llama.cpp prebuilt is at least 3 days behind
Layered on #5528. Generalises the MTP-specific staleness warning to
every llama.cpp prebuilt update, not just the ones that add MTP. If
the installed prebuilt is at least 3 days old AND its tag differs
from the latest published tag on the helper release repo (default
unslothai/llama.cpp), Studio nudges the user to run
"unsloth studio update".
How it works
Reads the install marker UNSLOTH_PREBUILT_INFO.json that
install_llama_prebuilt.py already writes to install_dir. The marker
carries the installed tag, the helper repo, and an installed_at_utc
timestamp. Studio compares those against the latest published tag
from the GitHub releases API for the helper repo.
GitHub fetch is cached at two levels:
- Process-level memo for /status hot path.
- Disk-level cache (24h TTL) at ~/.unsloth/studio/cache/llama_cpp_freshness/
so cold-start Studio launches do not always hit the API.
On a transient fetch failure (offline, rate-limited) we keep the
last-good disk value alive rather than poisoning the cache with None.
The check fails open: if anything is missing (marker, timestamp,
GitHub response), stale stays False so users never see a misleading
banner.
Surfaced in two places
1. Startup banner (logs + stderr) in main.py:lifespan(), alongside the
MTP capability probe added in #5528. Single line, e.g.:
WARNING: llama.cpp prebuilt is 5 days behind: installed b9190,
latest b9300. Run "unsloth studio update" to refresh.
2. /api/inference/status now returns:
llama_cpp_prebuilt_stale: bool
llama_cpp_installed_tag: str | None
llama_cpp_latest_tag: str | None
so the frontend can render a banner / popup with the actual tag
delta the user is missing.
3-day threshold
Mirrors the typical Unsloth llama.cpp release cadence. Anything
shorter would nag users who restart Studio at the wrong moment;
longer leaves real bugs sitting on the user's machine. Configurable
via the threshold_days kwarg if a future call site wants a different
window.
Tests
17 new cases in tests/test_llama_cpp_freshness.py cover marker
discovery in both cmake and root install layouts, missing / invalid
marker, GitHub fetch caching across process restarts (disk cache hit
after the in-memory cache is reset), the stale / not-stale decision
matrix (tag mismatch + age threshold), fail-open behaviour when
GitHub is unreachable, custom threshold, singular/plural day in the
warning string, and unparseable installed_at_utc. The broader
205-test inference regression suite still passes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: warn when llama.cpp prebuilt is too old for MTP
Layered on #5527. Adds a one-shot llama-server --help capability probe
so users get a clear signal when their prebuilt is missing MTP support,
plus a graceful fallback if they load an MTP GGUF against an outdated
binary.
What's surfaced:
1. Startup log + stderr line in main.py:lifespan() if MTP isn't
advertised:
WARNING: llama.cpp prebuilt is missing MTP support
(--spec-type mtp / draft-mtp). Run `unsloth studio update` to
refresh it. MTP GGUFs will load without speculative decoding.
2. Load-time graceful fallback in load_model's spec block: skip the
auto-emit and log a clear warning instead of letting llama-server
fail with an unknown-flag error.
3. /api/inference/status now returns llama_cpp_supports_mtp: bool so
the frontend can show a banner / popup.
Probe internals:
- Class-level cache keyed on (binary_path, mtime). One subprocess call
the first time, instant thereafter. Touching the binary (e.g. via
`unsloth studio update`) invalidates the cache automatically because
the mtime changes, so the new build is picked up without restarting
the server.
- Recognises both upstream naming forms: the original draft-mtp from
llama.cpp PR #22673 and the renamed mtp variant in later commits.
- Spec block uses whichever token the binary accepts so we emit the
right value regardless of which release the user has.
Tests:
- 6 new cases in test_llama_cpp_mtp_detection.py covering each probe
variant (draft-mtp, renamed mtp, pre-MTP build, missing binary,
mtime-based cache invalidation).
- Existing 38 MTP detection cases still pass; broader 188-test
regression suite (server args, reload inheritance, gguf metadata,
load progress, context fit, model validation) still green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: scope cancel-cleanup to in-flight tmp dirs; walk back tool_call_id
Two follow-ups to #5375's training and chat hardening.
_cleanup_cancelled_checkpoints used to rmtree every checkpoint-N
directory on Cancel. That is the opposite of what the user expects.
A user cancelling an 8h run with save_steps=2000 loses every
completed checkpoint they could have resumed from. The 67 MB residue
the audit memo flagged is the HF Trainer atomic-rename partial
(tmp-checkpoint-N), not the completed ones. The cleanup now targets
only tmp-checkpoint subdirs; completed checkpoint-N directories are
user-owned and stay. Symlinked output_dir and symlinked children are
skipped so the realpath containment cannot be levered into deleting
arbitrary content via a symlink trick.
ChatMessage._validate_role_shape stamped a random secrets.token_hex
id on tool messages with no tool_call_id. That id is uncorrelated
with the prior assistant tool_calls id, so strict passthrough
backends (OpenAI, Anthropic) reject the request as orphaned and
llama.cpp treats the tool result as "no preceding call" and
hallucinates. The synthesis moves up to ChatCompletionRequest, where
the whole conversation is visible: for each tool message missing an
id we walk back to the most recent assistant turn with tool_calls
(stopping at user turns), prefer a function.name match, otherwise
take the first unconsumed tool_call. Synthesis is the fallback when
no candidate assistant turn exists, preserving the prior round-trip
guarantee for orphaned tool messages.
Tests:
- test_cleanup_cancelled_checkpoints.py (new): pins that completed
checkpoint subdirs survive, tmp-checkpoint partials are removed,
non-int suffixes (checkpoint-final, checkpoint-best) are left
alone, output_dir outside outputs_root is refused, symlinked
output_dir and symlinked child are both skipped, missing dir is
a no-op.
- test_inference_model_validation.py: 6 new walkback cases covering
name-match preference, first-unconsumed fallback, explicit-id
passthrough, multi-tool-result pairing, synth-on-no-parent, and
no-cross-user-turn invariant.
- test_openai_tool_passthrough.py: the two ChatMessage-level
synth-on-missing tests are rewritten to assert that the per-
message validator now leaves tool_call_id untouched; resolution
coverage lives in the request-level tests above.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: explicit tool_call_id reserve, numeric tmp-checkpoint suffix only
Reviewer follow-ups to the training-cleanup + tool_call_id walkback PR.
tool_call_id walkback: a mixed assistant turn with [call_a, call_b]
followed by a tool result that carried tool_call_id="call_a" and a
sibling tool result with no id resolved to ['call_a', 'call_a']
because the explicit id never reserved call_a in the consumed set.
Added a pre-pass over the message list that walks back from every
role="tool" message carrying an explicit id and marks the matching
(asst_idx, tc_idx) consumed, then the missing-id walkback runs against
that pre-populated set. The second result now resolves to call_b.
While here, also harden the function-shape check: if a provider
ships a malformed tool_call where `function` is a string rather than
a dict, the old `(tc.get("function") or {}).get("name")` raised
AttributeError on the string's .get; now isinstance-gated so the
walkback falls through to the fallback id without raising.
Cancel cleanup: `tmp-checkpoint-*` is too broad. HF Trainer's
in-flight partials are always `tmp-checkpoint-<integer-step>`, so
constrain the cleanup regex to `^tmp-checkpoint-\d+$`. A user folder
named `tmp-checkpoint-final`, `tmp-checkpoint-backup`, or
`tmp-checkpoint-user-notes` is now preserved.
ChatMessage docstring still pointed at the pre-PR contract that
required `tool_call_id` on every role="tool" message. Updated to say
missing ids are accepted at message scope and resolved at
ChatCompletionRequest scope. Inline comment above the cancel-cleanup
call now describes the actual behaviour (in-flight tmp partials,
completed checkpoints preserved).
Test:
- python -m pytest studio/backend/tests/test_inference_model_validation.py
studio/backend/tests/test_cleanup_cancelled_checkpoints.py
studio/backend/tests/test_openai_tool_passthrough.py -q
-> 76 passed (was 67 before this commit; +2 walkback regression
tests, +1 numeric-suffix preservation test)
* studio: trim verbose comments in cleanup + tool_call_id walkback
Move the HF tmp-checkpoint regex to module scope as a named constant.
Drop the multi-paragraph docstring on _cleanup_cancelled_checkpoints
and the inline call-site rationale; the function name + the test
class already cover the why.
Compress _resolve_missing_tool_call_ids docstring from a six-line
explanation to two. Same logic, fewer in-flow tutorials.
76 tests in cleanup + inference-model-validation + tool-passthrough pass.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio/chat: reuse Anthropic code_execution container across turns
Mirror the OpenAI shell-tool reuse path for Anthropic. Backend latches
`message.container.id` off the message_start SSE event, emits a synthetic
container_ready _toolEvent, and forwards a stored id back on the next
turn via the top-level `container` request field. Stale-id 4xx surfaces
as container_invalidated so the next turn falls back to auto-create.
* studio/chat: temp diag log of Anthropic SSE events when code_execution is on
To locate where the API actually emits container.id on the stream.
* studio/chat: latch Anthropic container id from message_delta, drop diag
Anthropic surfaces container.id on `message_delta.delta.container`, not
on `message_start` (at start the container is not provisioned yet).
Move the latch + container_ready emit to message_delta and remove the
temporary raw-event log.
* studio/chat: fix OpenAI container delete UX (expired filter, TTL cap, idempotent 404, refresh-on-error)
- Filter status="expired" from /containers/list so the picker only
shows usable containers. OpenAI keeps expired entries in the list
indefinitely, which made delete look broken.
- Cap ttl_minutes at 20 (backend Field + frontend TTL_MAX + persistence
clamp). OpenAI's actual hard limit is 20; the prior 10080 cap caused
integer_above_max_value rejections on create.
- Treat 404 on delete as idempotent success in the frontend client so
already-gone containers don't surface a scary error toast.
- Run refresh() in finally for onCreate/onDelete so the picker stays
in sync with OpenAI even when the call errors.
- Add route-level test for the expired filter.
* studio/chat: add diagnostic logging for OpenAI /containers DELETE
Trace what arrives at /external/openai/containers/delete (subject,
container_id, base_url) and what we send to OpenAI (URL, presence
of Authorization, value of OpenAI-Beta) plus the full response
status + body (capped at 300 chars). Helps confirm whether the
beta header is on the wire and whether OpenAI's response actually
reports deleted=true, when users report the delete "not taking".
No secrets are logged — Authorization is reported as a boolean.
* studio/chat: log raw /containers list response from OpenAI
Sibling to the delete diagnostics. After a confirmed delete
(deleted=true on the wire), we want to see whether the very next
list call returns the just-deleted id — that distinguishes
"OpenAI eventually-consistent list" from "frontend stale state".
Logs each entry's id + status only; no names, no timestamps.
* studio/chat: fingerprint decrypted API key for container CRUD
Logs kind (sk-proj-/sk-/other), length, and last-4 chars only —
never the full secret. Lets us compare what the backend actually
uses against the key the user expects, since the same DELETE
request shape can produce different results across keys
(project-scoped containers: list is permissive but delete requires
the owning project's key).
* studio/chat: use fresh httpx client for /v1/containers DELETE
Same key, same headers, same URL via the shared _http_client
returned deleted=true but the container persisted in subsequent
list calls. A fresh httpx.AsyncClient with the identical request
shape (verified with a standalone reproducer) deleted the same
container cleanly. Suspect connection-pool state from earlier
chat-completion streams interferes at the edge — switching to a
per-call client side-steps it entirely. Scoped to delete only;
list/create keep using the shared pool until we can confirm the
same fix is needed there.
* studio/chat: log OpenAI response headers on container DELETE
Adds cf-ray / x-request-id / openai-organization / openai-project /
openai-processing-ms to the delete-response diagnostic line. Lets
us cross-reference a failing delete against OpenAI support (or
against a working standalone reproducer) using the unique
request-id and edge node.
* studio/chat: client-side tombstone for just-deleted OpenAI containers
OpenAI's /v1/containers DELETE returns {"deleted": true} but the
list endpoint can keep returning the same container for several
minutes (replica lag or in-use silent no-op — undocumented per
developers.openai.com/api/docs/guides/tools-shell). Our backend
sends the correct DELETE with OpenAI-Beta: containers=v1 and a
standalone reproducer shows the same behavior, so the right fix
is UI-side rather than waiting on OpenAI.
After a successful delete, the id goes into a per-component
tombstone map with a 5-minute expiry. visibleContainers (now the
single chokepoint feeding sortedContainers, auto-bind, and the
all-containers list) filters those ids out. A 30s sweep clears
expired tombstones so the picker recovers automatically if OpenAI
eventually catches up (or the container's TTL elapses).
* studio/chat: tombstones live for the page lifetime; drop API key fingerprint log
- Tombstones change from Map<id, expiry> to Set<id>: once tombstoned,
the id stays hidden from the picker until page reload. OpenAI's list
can keep returning a deleted id for an undocumented and variable
amount of time; automatically un-tombstoning after a fixed window
surfaces it again and creates more confusion than it solves. The
container's own TTL eventually expires the entry on OpenAI's side,
and the expired-status filter at the backend list route hides it
anyway.
- Remove the periodic sweep effect (dead code without expiries).
- Remove the api-key fingerprint log added during debugging — it
served its purpose (confirmed parity) and isn't needed long-term.
* studio/chat: built-in code execution for Anthropic Claude 4.x
Wire Anthropic's server-side code_execution_20250825 tool to the
existing Code pill in the composer. Pill lights up only for Claude
Opus/Sonnet/Haiku 4.x models that the docs list as compatible; pairs
independently with Search. Backend appends the tool entry plus the
code-execution-2025-08-25 beta header, and translates the SSE
server_tool_use / *_tool_result blocks (bash + text_editor sub-tools)
into the _toolEvent shape the frontend renderer consumes. File
uploads via the Files API are a deliberate follow-up.
* studio/chat: enable code execution pill in in-thread composer too
thread.tsx renders its own composer with a separate CodeToolsToggle
that was still gated on supportsTools only, so the pill stayed
disabled inside an active thread even after picking Anthropic 4.x.
Surface the capability through the runtime store
(supportsBuiltinCodeExecution, set from chat-page alongside
supportsBuiltinWebSearch) and read it in the toggle.
* studio/chat: built-in code execution for OpenAI cloud gpt-5.5
Extend the Code pill to OpenAI cloud's gpt-5.5 / gpt-5.5-pro via the
shell tool on /v1/responses. Per-thread container reuse: capture the
container_id from each response on a synthetic container_ready event,
persist it onto the ThreadRecord, and pass it back as
environment.type="container_reference" on follow-up turns so the
model sees filesystem state from prior turns until OpenAI's idle
expiry. Stale ids surface a container_invalidated event that clears
the thread record so the next turn falls back to container_auto.
Gated strictly on OpenAI cloud (api.openai.com base URL) — Ollama,
llama.cpp, vLLM, and custom OpenAI-compat presets won't see the
shell tool entry even when their providerType collapses to "openai".
* studio/chat: OpenAI shell-tool container management UI
Side-panel section (settings sheet → Code Execution) for managing
OpenAI's shell-tool containers per thread. Three controls:
- New-container idle timeout (provider-level default, pre-fills the
create dialog and is used by the lazy-create path on a thread's
first turn when set to a non-default value).
- Active container picker for the active thread — pick any existing
container or stay on "Auto-create per thread".
- Inline create form (name + idle TTL) and per-row delete actions.
Three new backend endpoints under /api/inference/external/openai/
containers/{list,create,delete} proxy to OpenAI /v1/containers using
the encrypted API key. All three reject non-cloud base URLs up front
so the picker stays scoped to api.openai.com.
Deleting a container clears all thread bindings pointing at it; the
next turn falls back to auto-create.
* studio/chat: inherit container across threads + styled active picker
New threads on the same OpenAI provider now default to the most
recently used container instead of "Auto-create per thread" — both
in the chat-adapter (so a send works even if the side panel was
never opened) and in the side panel itself (auto-binds the active
thread when the dropdown loads on a thread that has no container).
Picker is visually emphasized with an accent panel and the
currently-active row in the list below is highlighted with the same
accent so the two views stay in sync.
* studio/chat: friendly English-word names for auto-created containers
Replaces the "chat-<thread-id-slug>" auto-name with a random
English-word + short hex suffix (e.g. "kestrel-3f9c"). Applies only
to the chat-adapter's lazy-create path; the OpenAI container_auto
path stays unnamed (only fires when no custom TTL is set).
* studio/chat: always pre-create OpenAI containers via frontend
Drops the TTL-based gate on the chat-adapter's lazy-create path so
every code-execution container the user ever sees in the picker has
a friendly English-word name. The backend's container_auto fallback
stays as a safety net (used only if the POST /v1/containers call
fails); in practice that branch should be rare.
* studio/chat: send OpenAI-Beta header for /v1/containers CRUD
Without OpenAI-Beta: containers=v1, OpenAI returns 200
{"deleted": true} for DELETE /v1/containers/{id} but does not
actually remove the container. The list call then keeps returning it,
making it look like Studio's "Delete container" button is broken.
Verified 2026-05-15 against api.openai.com: DELETE with the beta
header returns 200 and removes the container; the same DELETE without
the header returns the same 200 deleted:true body but the container
stays alive.
- Add _container_headers() that merges OpenAI-Beta on top of the
shared auth headers; route list / create / delete through it.
- Verify the DELETE response body reports {"deleted": true}; raise
httpx.HTTPError otherwise so the route surfaces a 5xx instead of
silently reporting success on a silent no-op.
- Add tests covering header propagation and the deleted-flag guard
(true, false, missing key, non-JSON body, 4xx passthrough).
* studio/chat: surface unpersisted-thread picker no-op as a toast
The "Active for this thread" container picker uses
db.threads.update(activeThreadId, ...), which silently returns 0 rows
affected when the thread record isn't yet in IndexedDB. That happens
on a brand-new thread where the user toggles code execution on and
opens settings before sending the first message — the chat adapter
only materializes the thread row on first send. The picker would
appear to ignore the user's selection and snap back to "Auto-create
per thread".
- onPick now awaits the update and toasts an actionable hint
("Send a message first to pin a container to this thread.") when
the update affected zero rows.
- Auto-bind effect comment clarifies why it stays best-effort silent.
The auto-bind effect itself is unchanged: it's a heuristic that
should not nag the user when it can't apply.
* studio/chat: let user pick OpenAI container before first send
Previously the picker silently no-op'd until the user sent the first
message, because Dexie's ThreadRecord is only materialized inside the
runtime-provider's `initialize` hook (assistant-ui's first-message
callback). That kept users from binding a thread to an existing
OpenAI container up front; they had to either send a message and
risk the chat adapter auto-creating one, or accept the cross-thread
inheritance default.
- Export `ensureThreadRecord` from runtime-provider so other surfaces
can materialize the row idempotently.
- In OpenAICodeExecSection.onPick, await ensureThreadRecord before
the update, with modelType="base" (the settings sheet that hosts
this section is only rendered in single-thread mode).
Behaviour after this commit:
- New thread + user picks a container in the sidebar → thread row is
created with that container_id; first send uses it, no auto-create.
- New thread + user does nothing → row still absent; first send goes
through the existing inherit/lazy-create path as before.
- The auto-bind effect remains silent best-effort: it does not
eagerly create the thread row, so it cannot pre-empt the user's
pick on a fresh thread.
* studio/chat: drop "Auto-create per thread" option, default to latest
The dropdown previously offered "Auto-create per thread" as an
explicit value (null in storage), with the chat-adapter then
inheriting from the most recent container at send-time. That made
the picker display disagree with what the backend would actually do:
the picker said "auto", but the backend was reusing an existing
container.
Behaviour after this commit, when code execution is enabled on an
OpenAI cloud provider:
- Containers list non-empty: dropdown defaults to the container with
the latest lastActiveAt, eagerly bound via ensureThreadRecord +
db.threads.update so the bind survives even when the thread row
has not been materialized by the chat adapter yet. User can pick
any other container in the list.
- Containers list empty: render a disabled placeholder "(none yet —
will be created on first send)". The chat-adapter's lazy-create
path (chat-adapter.ts:1040-1082) mints the first container on
first send and writes it back to the thread; the next refresh
surfaces it in the picker.
Expiration mid-operation is unchanged: the existing
container_invalidated _toolEvent clears the thread's stored id and
the next turn re-creates.
* studio/chat: fix picker stuck on "Selecting most recent…" + manual-create binding
Two follow-up fixes to the picker rework in d0cbeb99b.
1) The dropdown was getting stuck on the "Selecting most recent…"
placeholder option even after the auto-bind write completed,
because the select was controlled by `activeContainerId` (whatever
sits in Dexie) and there's a brief window between the auto-bind
firing and useLiveQuery propagating the new row back. Decoupled
the rendered value from the Dexie state: compute the displayed id
locally as `activeContainerId ?? sortedContainers[0]?.id`, so the
most-recent container's name shows up immediately. The auto-bind
effect still writes the bind to Dexie so the chat adapter sees it
on send. Dropped the placeholder option entirely.
2) The manual "Create container" flow (`onCreate`) bound the new
container to the active thread with a bare `db.threads.update`.
On a brand-new thread that hadn't been materialized yet, the
update affected 0 rows; the user's next send then went through
cross-thread inheritance / lazy-create and could land on a stale
container, surfacing as "container does not exist". Same fix as
`onPick`: ensureThreadRecord before update so the bind lands.
* make API key optional for local providers (llama.cpp/vLLM/Ollama)D
* chore: reduce comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* polish: update provider dropdown and rename cloud
* fix: tighten custom provider fallback handling
* fix: external provider fallback typing
* studio: wire the chat Search button to OpenAI's built-in web_search tool
When the active model is an OpenAI external provider and the user
clicks the existing Search pill in the composer, the chat-completion
request now carries the unified enable_tools shorthand:
enable_tools: true
enabled_tools: ["web_search"]
The backend's stream_chat_completion threads enabled_tools through
to _stream_openai_responses, which translates it into the Responses
API tool schema:
body["tools"] = [{"type": "web_search"}]
per the OpenAI Responses tool spec
(https://developers.openai.com/api/docs/guides/tools). OpenAI then
runs the search server-side before the model replies; the search-
informed answer streams back through the existing
response.output_text.delta path. web_search_call lifecycle events
are silently ignored for now — sources / status indicators are
follow-up scope.
Frontend:
- provider-capabilities.ts: new providerSupportsBuiltinWebSearch()
helper. Returns true only for `openai` today; Anthropic
(web_search_20250305), Gemini grounded-search, and OpenRouter
variants can be added later with matching backend translation.
- chat-page.tsx: both model-switch paths (the onChange handler and
the inferenceParams.checkpoint useEffect) set supportsTools to
match the new helper, and force toolsEnabled=false on every
external switch so the Search toggle is opt-in by default.
- chat-adapter.ts: external branch adds enable_tools +
enabled_tools=["web_search"] to the request body when the
toggle is on AND the active provider supports built-in
web-search. Local-model branch is unchanged — it continues to
route the same shorthand through our local tool runtime.
Backend:
- routes/inference.py: forwards payload.enabled_tools to
stream_chat_completion at the proxy site (line 1599).
- external_provider.py: stream_chat_completion gains an
enabled_tools parameter; _stream_openai_responses appends
{"type": "web_search"} to body["tools"] when the list contains
"web_search". Other tools (file_search, code_interpreter,
image_generation, computer_use_preview) are easy follow-ups in
the same block.
Reuses the existing pydantic ChatCompletionRequest.enabled_tools
field, so no schema migrations.
* studio/backend: surface OpenAI server-side web_search in the chat UI
When the user has the chat Search button toggled on and OpenAI's
/v1/responses invokes the built-in web_search tool, _stream_openai_responses
now translates the tool's lifecycle events and citation annotations
into the same _toolEvent shape that local-tool calls use. The result:
the chat UI shows a web_search tool-call card mid-stream, then lists
the cited sources at the end of the message — identical to how local
web_search renders.
SSE event translation:
- response.output_item.added with item.type=web_search_call ->
emit _toolEvent tool_start. Carries item.action.query as args
when OpenAI ships it on the added event.
- response.output_item.done with item.type=web_search_call ->
backfill the query if it only arrives on the done variant. The
existing reasoning branch on the same event is preserved as an
if/elif under a shared isinstance guard.
- response.output_text.annotation.added with type=url_citation ->
collect into the most-recent web_search_call.citations list.
- response.output_text.delta with inline annotations[] (older
API variant) -> same collection path, so both wire shapes work.
- response.completed -> emit _toolEvent tool_end per call with
citations formatted as
Title: <title>\nURL: <url>\nSnippet: <snippet>
blocks joined by `\n---\n`. The frontend's
parseSourcesFromResult already lifts this format into source
content parts at end-of-stream.
- response.incomplete -> close out web_search cards with whatever
citations had landed, so a truncated response does not leave a
perpetually "running" tool card in the UI.
Both reasoning and web_search work simultaneously on the same turn —
the body sends `reasoning: {effort, summary}` and `tools: [{type:
"web_search"}]` independently, and the SSE handler tracks them
through separate channels.
Diagnostic: finally-block logger now reports per stream
web_search_requested - whether the client asked for it
web_search_invocations - how many calls OpenAI actually made
citations - total URLs cited
queries - the search queries the model issued
reasoning_emitted - whether <think> content was streamed
so reports of "I clicked Search and nothing happened" can be triaged
from the backend log without browser devtools.
* studio/backend: fix empty query + per-card '(no sources cited)' on OpenAI web_search
Two display bugs on the OpenAI Responses web_search → chat-UI bridge:
1. Tool cards showed "Searching for ''" — query missing.
OpenAI's response.output_item.added for web_search_call does not
reliably populate action.query across API versions; the canonical
place is output_item.done. The previous code emitted tool_start
at added with empty args and tried to backfill at done, but the
frontend's _toolEvent: tool_start is a one-shot push (no update
mechanism), so the args stayed empty.
Fix: defer both tool_start *and* a placeholder tool_end emission
to output_item.done, where action.query is guaranteed populated.
added now just initialises tracking. Frontend then renders one
card per call with the right "Searching for: <query>" label.
2. Every card showed "(no sources cited)".
The previous code tried to attribute url_citation annotations
to individual web_search_call invocations, but OpenAI's
annotations carry no link back to a specific search call —
they're just URLs the model cited from the aggregated search
pool. With N invocations and M annotations, the previous logic
bucketed all M into the last call and stamped "(no sources
cited)" on the rest.
Fix: collect citations into a single shared all_url_citations
list, dedup by URL. At response.completed (and
response.incomplete) overwrite the *last* web_search_call's
tool_end result with the aggregated Title:/URL:/Snippet:
blocks. The frontend's parseSourcesFromResult already flatMaps
every web_search result, so one non-empty result is enough to
surface the full source-pill set at the message tail. Other
tool cards get an empty result string (no '(no sources)' text).
Diagnostic log unchanged in shape; total_citations now reads
len(all_url_citations) directly.
* studio/chat: split Code and Search pill gates so external models cannot enable Code
The previous wire-up set supportsTools=true for OpenAI external
models to light up the Search pill, but supportsTools also gates the
Code pill, so Code became clickable for OpenAI even though external
providers have no local code execution.
Separate the two gates so each pill reflects what's actually
available:
- chat-runtime-store: new `supportsBuiltinWebSearch: boolean` flag.
Distinct from supportsTools — that one still means "runtime has a
local tool sandbox" (Code, python, our DuckDuckGo web_search).
This one means "the active external provider exposes a server-side
web_search tool we can opt into" (OpenAI's /v1/responses today).
- chat-page model-switch (both code paths): for external models,
supportsTools is now forced to false (no local Code path) and
supportsBuiltinWebSearch follows providerSupportsBuiltinWebSearch.
Local-model paths are unaffected — they only set supportsTools.
- shared-composer: Search pill gates on
`searchDisabled = !modelLoaded || !(supportsTools ||
supportsBuiltinWebSearch)`. Code pill gates on
`codeDisabled = !modelLoaded || !supportsTools` — strictly the
local runtime, so external models keep Code greyed out.
A `toolsDisabled = codeDisabled` alias is left in place for any
later-touched call site that may still reference the old name.
No backend changes — chat-adapter already calls
providerSupportsBuiltinWebSearch directly, independent of the store
flags, so the request shape and the backend translation are
unchanged.
* studio/chat: default external reasoning effort to medium, not the carry-over
When switching to an external model with reasoning support, the effort
dropdown was inheriting whatever value the user had set on a prior
model — frequently "xhigh" left over from a previous Opus/gpt-5
session. That meant every fresh OpenAI/Anthropic selection started at
Extra High, burning tokens unintentionally.
Both model-switch sites in chat-page (the useEffect on
inferenceParams.checkpoint and the onChange callback) now pick
"medium" whenever the new model's level list contains it, instead of
the clamped carry-over. The clamp still fires as a fallback for the
narrow case where a model doesn't expose medium (e.g. gpt-5.3-chat-
latest which only has medium anyway — no change there). Users can
still pick another level explicitly via the Think dropdown.
* studio/chat: also light the Search pill in the welcome-screen composer
There are two composers in the chat feature. shared-composer.tsx
renders inside an active thread, and assistant-ui/thread.tsx has its
own WebSearchToggle / CodeToolsToggle that ship the welcome-screen
"Send a message…" composer (visible before the first user message).
The previous fix split supportsTools and supportsBuiltinWebSearch in
shared-composer but never touched the welcome-screen toggles in
thread.tsx — they both still gated on supportsTools alone, so the
Search pill stayed greyed on the welcome screen even for OpenAI
external models that legitimately support web_search server-side.
Mirror the shared-composer rule in WebSearchToggle:
disabled = !modelLoaded || !(supportsTools || supportsBuiltinWebSearch)
CodeToolsToggle is left as-is — its current
`disabled = !(modelLoaded && supportsTools)` is correct: external
models have no local code-execution sandbox, so Code stays greyed
when supportsTools=false (which is what chat-page now writes for
external selections).
* studio/backend: wire Anthropic server-side web_search end-to-end
Mirrors the OpenAI web_search integration for Anthropic's
web_search_20250305 tool. When the user toggles Search on with an
Anthropic model selected, the request now carries the documented
tool entry:
tools: [{type: "web_search_20250305", name: "web_search",
max_uses: 5}]
on /v1/messages, and the SSE translation surfaces tool cards +
source pills in the chat UI exactly the same way as OpenAI.
stream_chat_completion now forwards enabled_tools into the
Anthropic branch (was only doing this for the OpenAI Responses
branch). _stream_anthropic gains an enabled_tools parameter and
the web_search request-body block plus three additional event
handlers:
- content_block_start with type=server_tool_use, name=web_search:
start tracking a new call. id becomes the tool_call_id.
- content_block_delta with type=input_json_delta inside a
server_tool_use block: buffer the partial_json so we can read
out the search query when the block closes.
- content_block_start with type=web_search_tool_result: capture
the per-call result list (urls + titles) that Anthropic ships
inline.
- content_block_stop: closes whichever block we're inside —
* server_tool_use -> emit _toolEvent: tool_start with the
parsed query as args.
* web_search_tool_result -> emit _toolEvent: tool_end with
Title:/URL: blocks the frontend's parseSourcesFromResult
lifts into source pills.
* thinking block -> existing </think> close.
Unlike OpenAI we get per-call results directly, so no aggregated-
last-call fallback is needed — each tool card carries its own
citations.
Diagnostic log on stream completion now reports
web_search_requested / invocations / total_results / queries,
matching the OpenAI shape.
Frontend providerSupportsBuiltinWebSearch returns true for
'anthropic' as well, so the Search pill lights up on Claude
models the same way it does on OpenAI. The existing chat-adapter
external branch already sends enabled_tools=['web_search'] based
on this helper — no adapter changes needed.
* studio: wire OpenRouter built-in web search via :online model suffix
OpenRouter exposes a universal "add web search to any model" shortcut:
append `:online` to the model id and the gateway runs the search
server-side, streaming citations back as annotations on text deltas.
Documented at https://openrouter.ai/docs/features/web-search
Hook the existing Search toggle into that path:
Backend (external_provider.py, default OAI-compat branch):
- When provider_type == 'openrouter' and enabled_tools contains
'web_search', rewrite body['model']:
openai/gpt-4o -> openai/gpt-4o:online
anthropic/claude-sonnet-4-5:free -> anthropic/claude-sonnet-4-5:online
Any existing `:variant` (`:free`, `:nitro`, etc.) is replaced —
OpenRouter variants are mutually exclusive.
- `openrouter/free` is skipped: it's a meta-router and `:online` is
not a valid suffix on it (the gateway 400s).
- A one-line INFO log fires whenever the rewrite happens so the
diagnostic backend log shows exactly which model id the request
was promoted to.
Frontend (provider-capabilities.ts):
- providerSupportsBuiltinWebSearch now returns true for 'openrouter'
alongside 'openai' and 'anthropic'. The Search pill lights up and
the existing chat-adapter external branch already forwards
enabled_tools=['web_search'] based on this helper — no adapter
changes needed.
No new SSE event handling: OpenRouter does not emit a separate
web_search_call event the way OpenAI/Anthropic do. Citations come
back as text annotations via the existing reasoning_details path
the adapter already parses, so source data flows through without
extra translation. A per-call tool-card UX ("Searching for: …")
would require synthesizing one client-side; deferred to a follow-up
if the bare-citation flow feels too minimal.
* studio: wire Mistral built-in web search connector
Same shape as OpenAI's web_search tool, lives on
/v1/chat/completions instead of /v1/responses. When the chat
Search pill is toggled on with a Mistral model selected, the
backend now appends
{"type": "web_search"}
to body["tools"] before the request goes out. Idempotent —
won't double-append if a future call site adds it first. Models
in the registry allowlist that don't support the connector
(codestral, devstral, ministral, mistral-tiny) will surface a
400 from upstream; the existing default-path error log captures
it. Mistral's docs:
https://docs.mistral.ai/capabilities/agents/connectors/websearch
Frontend providerSupportsBuiltinWebSearch returns true for
'mistral' now, alongside openai / anthropic / openrouter. The
Search pill lights up for Mistral models and the existing
adapter branch already sends enabled_tools=['web_search'] off
this helper — no adapter changes.
No SSE translation yet — Mistral streams citations inline as
text annotations or `references` in the final assistant content,
not as a separate web_search_call event. Citations flow through
to the message body as text; a per-call tool-card UX with
"Searching for: …" indicators is a follow-up if needed.
* studio/backend: fix OpenRouter web_search to use plugins shape + synthesize tool card
Two changes against the actual OpenRouter docs at
https://openrouter.ai/docs/guides/features/plugins/web-search:
Request shape:
The previous commit appended :online to the model id, which works on
concrete model ids but rejects on meta-routers like openrouter/free —
and that's exactly the model the user was testing with, so neither
the request rewrite nor the diagnostic log fired. Switch to the
universal plugins shape:
body["plugins"] = [{"id": "web"}]
Per the docs this is "exactly equivalent" to :online but works on
every model id including openrouter/free and openrouter/auto. No
model suffix manipulation, idempotent if added twice.
Tool-card synthesis:
OpenRouter doesn't emit a structured web_search_call event the way
OpenAI/Anthropic do — citations come back only as `annotations` of
type=url_citation on delta/message objects. To match the chat-UI
tool-card UX the user expects ("Searching for: …" indicator,
source pills at message tail), synthesize the events client-side
in the default OAI-compat stream loop:
- On stream open (after the 200 status check): yield a synthetic
_toolEvent: tool_start with tool_name=web_search, fixed id
"openrouter_web_search". The chat-UI then renders the running
tool card before any text streams.
- During the SSE loop: scan every chunk's choices[].delta and
choices[].message for `annotations: [{type: "url_citation",
url_citation: {url, title, content}}]` entries. Dedup by URL
into a citations list. Handles both the nested-url_citation
shape OpenRouter documents and the flat-on-annotation shape
some upstreams ship.
- On [DONE] (or stream-close without [DONE]): emit synthetic
tool_end carrying the citations as
Title: …\nURL: …\nSnippet: …\n---\n…
blocks the existing parseSourcesFromResult lifts into source
pills at message tail.
Diagnostic log on completion now also reports
web_search_requested + citation count alongside the existing
chosen-model / event-count telemetry.
* studio: drop Mistral built-in web_search — connector lives on Agents API only
Mistral's web_search is exclusively on /v1/agents + /v1/conversations;
sending it on /v1/chat/completions returns
"WebSearchTool connector is not supported". Wiring it would require a
dedicated Agents streaming path. Remove from the frontend capability map
and revert the chat-completions tool injection.
* studio: wire Kimi $web_search builtin via two-call round-trip
Kimi's $web_search lives on /v1/chat/completions but requires a client
round-trip per https://platform.kimi.ai/docs/guide/use-web-search:
the first call returns tool_calls with function.arguments populated;
the caller echoes those arguments back as a role=tool message; the
second call streams the final answer with search results incorporated.
The docs also mandate thinking=disabled while the builtin is active.
Backend: new _stream_kimi_web_search helper dispatched from
stream_chat_completion when provider_type=='kimi' and 'web_search' in
enabled_tools. Buffers tool_calls across deltas, falls back to a plain
stream if the model declines to search, and synthesizes tool_start
(with parsed query) / tool_end (with any url_citation annotations) so
the chat UI's web-search card behaves the same as other providers.
Frontend: kimi added to providerSupportsBuiltinWebSearch so the Search
pill lights up in the composer.
* studio/chat: mutual exclusion of Think + Search on Kimi composer
Kimi's $web_search builtin requires thinking=disabled per
https://platform.kimi.ai/docs/guide/use-web-search, so the two states
cannot coexist. Make the pills mutually exclusive in both composers
(shared and welcome-screen): clicking Search turns Think off; clicking
Think back on turns Search off. Default Think to on when a Kimi model
is selected — k2.6/k2.5 ship with thinking enabled out of the box.
* studio/chat: fix wrong provider var name in onChange branch
selectedProvider, not provider — TS2304 in tsc -b.
* studio/backend: add diagnostics to Kimi $web_search round-trip
Log the actual function.arguments from the first call (so we can see
the model's search query) and the second call's usage.prompt_tokens +
any annotation type names that came through. prompt_tokens spiking
above the input message length is direct proof the server injected
search results into context. annotation_types lets us learn the shape
Kimi uses for citations if/when they emit any.
* studio: per-provider defaults — Anthropic xhigh + Search on, OpenAI high + Search on, Opus 4.7 gains max
Anthropic: Think effort defaults to the highest level the model
supports (xhigh on 4.6/4.7, high on 4.5) and Search starts on, since
the web_search_20250305 tool returns structured citations end-to-end.
OpenAI: Think effort defaults to 'high' (the gpt-5.x reasoning sweet
spot for /v1/responses + web_search) and Search starts on.
Opus 4.7: 'max' added as an effort level above 'xhigh' in both
backend (_ANTHROPIC_THINKING_SPECS) and frontend (ANTHROPIC_REASONING_MODELS).
Kimi diagnostics: emit tool_end immediately after tool_start so the
web-search card transitions to 'complete' before the second-call
answer streams, log first-call args + second-call usage/prompt_tokens
+ any annotation type names, request stream_options.include_usage so
the second call exposes usage in SSE.
* studio/backend: harden Kimi fallback path with HTTPError handler + manual aiter_lines loop
Addresses PR review feedback (#5443): the no-search fallback streaming
path was using `async for response.aiter_lines()` and had no
`httpx.HTTPError` guard around the POST. Switch to the manual
__anext__ loop pattern used elsewhere in this module (avoids the
Python 3.13 + httpcore 1.0.x GeneratorExit propagation issue) and wrap
the whole request in a try/except so network failures surface as a
proper SSE error frame instead of a raw traceback.
* feat: prompt caching frontend for openai/anthropic
* studio/chat: route vLLM provider to /v1/chat/completions, not /v1/responses
vLLM's /v1/responses rebuilds messages through the loaded model's chat
template, which 400s on strict-alternation templates like Gemma 3
("Conversation roles must alternate user/assistant/..."). Stop collapsing
vllm -> openai in the frontend so the backend sees the real provider type
and falls through to the standard chat-completions path. Register vllm as
a hidden entry in PROVIDER_REGISTRY so supports_vision and provider-create
validation work without surfacing it in the cloud-provider dropdown.
* studio/chat: wire prompt caching for OpenAI and Anthropic external providers
Backend half of the prompt_caching toggle that already exists in the chat
settings panel. Scoped to OpenAI cloud (/v1/responses) and Anthropic
(/v1/messages); every other provider plumbs the flag as a no-op.
- Anthropic: attach cache_control={type:ephemeral} to the system block so
the static prefix is reused across turns. Without the marker Anthropic
caches nothing, so this is the only way to make the toggle do real work
on /v1/messages.
- OpenAI: opt into prompt_cache_retention="24h" — same price as the
default in_memory policy per the OpenAI docs, but the cache survives
~24 hours of idle instead of ~5-10 minutes. The model picker is
registry-scoped to gpt-5.x / o3 / gpt-4.5, all of which accept the
parameter (gpt-5.5+ already defaults to "24h" so it's a no-op there).
- Treats `enable_prompt_caching=None` as enabled to match the frontend
default for both providers; pass `false` explicitly to opt out.
* studio/chat: log cache token counts on OpenAI and Anthropic stream completion
Surface cache usage in the existing "stream complete" info logs so
prompt-caching behavior can be verified by tailing the studio backend
log instead of opening the provider dashboard.
- Anthropic: latch usage from message_start (input + cache_creation +
cache_read counts) and message_delta (output_tokens), then include in
the per-request summary. cache_read_input_tokens > 0 confirms the
cache_control marker on the system block is doing its job.
- OpenAI Responses: latch usage from response.completed and
response.incomplete, extract usage.input_tokens_details.cached_tokens
(the /v1/responses field name, not prompt_tokens_details). A non-zero
value on turn N proves prompt_cache_retention="24h" let the prefix
hit the cache instead of being recomputed.
* studio/backend: strip temperature/top_p for Claude 4.7 family
Anthropic Opus 4.7 removed temperature, top_p, and top_k as a launch
breaking change ("Sampling parameters removed" in the 4.7 release notes
at https://platform.claude.com/docs/en/about-claude/models/whats-new-claude-4-7).
Setting any of them to a non-default value returns 400
"<param> is deprecated for this model". The existing guard only handled
top_k; temperature was still being sent unconditionally and is now
breaking opus-4-7 requests.
Rename _ANTHROPIC_TOP_K_DEPRECATED to _ANTHROPIC_4_7_SAMPLING_REMOVED to
reflect the broader scope, omit temperature from the base body on 4.7,
and skip the thinking-mode temperature=1 override on 4.7 (still applied
on 4.5/4.6 where it's required). Existing thinking_translation tests
target 4.5/4.6 / mock the wire so they're unaffected.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/chat: anchor Anthropic prompt cache on the latest message too
A system-only cache_control marker is a no-op when the system prompt is
empty or shorter than Anthropic's ~1024-token cache floor — caching
silently does nothing (both cache_creation and cache_read return 0).
Add a second cache_control breakpoint on the final block of the latest
conversation message so the entire prefix (system + prior turns + new
user turn) becomes eligible for caching. On turn N+1, Anthropic
rehydrates everything up through turn N's marker instead of recomputing
it. Up to 4 breakpoints are allowed per request; we use at most 2
(system + tail). Tail rebuild avoids mutating the caller's content list
so an image-bearing turn still slots cleanly into the cached prefix.
* studio/chat: gate vLLM reasoning toggle on provider config
Add a "This server runs a reasoning model" checkbox on the vLLM
provider config. When off (default), the chat Think pill stays
hidden and no enable_thinking ever reaches vLLM. When on, the
pill renders, per-turn state flows through the existing
enable_thinking plumbing, and the backend proxy lifts it onto
chat_template_kwargs.enable_thinking so vLLM's Jinja template
honours it.
* chore: clean vLLM reasoning-toggle comments
* studio/chat: gate prompt_cache_retention to actual OpenAI cloud requests
Addresses Codex P1 review on _stream_openai_responses. The frontend
only sends enable_prompt_caching for the openai/anthropic UI provider
types, so ollama/llama.cpp/"custom" requests reach this helper with
the flag as None. The previous `is not False` check treated None as
enabled and injected prompt_cache_retention="24h" into every request
including those bound for non-OpenAI servers, which would 400 on
servers that implement /v1/responses but not the retention parameter.
Match the public OpenAI host (api.openai.com) on the client base_url
before adding the field so it only lands on actual OpenAI cloud
requests. Studio's openai picker is already registry-scoped to
gpt-5.x / o3 / gpt-4.5, all of which accept the parameter.
---------
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: drop unused max_grad_value schema + route plumbing
The MLX worker hardcodes max_grad_value to 5.0 after PR #5340. The
schema field, frontend payload type, route forwarder, and start_training
kwarg threading were all left in place as a transitional buffer for old
clients. The field is now genuinely unused everywhere except inside the
MLX worker, so the schema, route forwarder, and config-build entries can
go. Pydantic still tolerates older clients that send max_grad_value
because TrainingStartRequest's model_config defaults to extra=ignore.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* mlx fixes
* Fix studio integration, local dataset files, chat templates without the torch gpu imports
* pass grad norm in mlx worker
* fix(studio): pass MLX grad clipping settings
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* mlx: update grad value
* fix(mlx): address ci and clipping review
* fix backward compatibility and CI tests
* unsloth local is mlx function
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* dont reference runtime
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio mlx: hardcode value clipping, drop max_grad_value from frontend
Simplifies the MLX grad-clipping plumbing now that we are standardising on
elementwise value clipping at [-5, 5] for the compiled MLX path and norm
clipping disabled. The MLX worker no longer reads max_grad_norm /
max_grad_value from the request; both are pinned in one place. Frontend
stops sending the field at all, and the TypeScript request type drops it
to match. Non-MLX (CUDA/AMD/Intel) is untouched and continues to pick up
HF TrainingArguments' default max_grad_norm = 1.0.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* studio: add external provider support for chat inference
Adds the ability to connect to OpenAI, Mistral, Google, Cohere, Together,
Fireworks, and Perplexity from the Studio chat interface.
- Provider configs stored in SQLite (no API keys persisted)
- RSA-2048 key pair generated at startup for client-side key encryption
- httpx proxy client streams SSE responses in OpenAI-compatible format
- New /api/providers routes: registry, CRUD, test, models
- /v1/chat/completions routes to external provider when provider fields present
- Integration test suite covering CRUD, connection, model listing, and inference
- Frontend spec doc with full API contract
* remove frontend spec doc from branch
* fix auth fixture: handle forced password change on fresh install
* fix tests: default port 8000, allow 400 for no-model-loaded
* fix: update Cohere models to current (command-r retired Sept 2025)
* feat: add OpenRouter as 8th provider
* feat: add native Anthropic provider with Messages API translation
* fix: correct Anthropic base URL and drop top_p (conflicts with temperature)
* feat: add DeepSeek provider (deepseek-chat, deepseek-reasoner)
* feat: rename google -> gemini, refresh model list to 2.5 series
* feat: remove together, fireworks, perplexity providers
* feat: multimodal image support for external providers
- Add _build_external_messages() that preserves image_url parts for
vision-capable providers instead of stripping them
- Update _proxy_to_external_provider() to use new helper
- Translate image_url content parts to Anthropic native image format
in _stream_anthropic()
- Add TestVisionInference pytest class (1x1 PNG smoke test)
* test: use sloth photo URL for vision test, add Anthropic remote URL support
* fix: update Mistral model to mistral-small-2506
* update mistral default model to mistral-large-2512
* fix gemini vision test: download image as base64 data URI instead of remote URL
* add gemini-3-flash-preview as default gemini model
* fix gemini truncated reply (max_tokens 16->64) and suppress GeneratorExit on client disconnect
* increase vision test max_tokens to 215
* fix GeneratorExit: aclose stream generator before closing httpx client
* fix httpcore GeneratorExit: explicitly aclose aiter_lines before response closes
* fix duplicate [DONE] and suppress httpcore RuntimeError on Python 3.13 asyncgen cleanup
* fix: call response.aclose() before lines_gen.aclose() to prevent httpcore RuntimeError on Python 3.13
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Potential fix for code scanning alert no. 36: Clear-text logging of sensitive information
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* review: add comments for manual iteration rationale, mask password in test print, clarify Anthropic URL/models support
* perf: use shared module-level httpx client for connection pooling across requests
* studio: add API provider UI and integrate wiring (#4737)
* feat: expose external models in selector and chat settings
* feat(chat): wire external providers to backend + RSA key flow
- Fetch registry/configs; create/update/delete saved providers
- Encrypt API keys (Web Crypto RSA-OAEP) for test/models/chat
- External model selection + chat payload (provider_id/type, external_model, encrypted key, optional base URL)
- Local storage for keys + provider list; small UX/copy and guardrails
* add missing providers-api.ts file by Imagineer99
* fix: address PR review comments — system prompt visibility, retry loop, test logging
* feat(studio): encrypt external provider API keys at rest in localStorage
API keys for external providers (OpenAI, Mistral, etc.) were stored as
plaintext in localStorage, vulnerable to browser extensions and XSS.
Add password-derived AES-256-GCM encryption: on login the user's password
is used via PBKDF2 (100k iterations, SHA-256) to derive an in-memory
encryption key. API keys are encrypted before writing to localStorage and
decrypted on read. The derived key is never persisted — cleared on logout,
re-derived on next login.
Legacy plaintext keys are transparently migrated on first access. Password
changes re-encrypt all stored keys. No backend changes required — the
existing RSA-OAEP transit encryption is unaffected.
* fix: cast PBKDF2 salt to BufferSource for strict TypeScript lib types
* fix: persist session password in sessionStorage to survive page refreshes
* feat(studio): preserve image parts in external provider chat requests
toOpenAIMessage() now returns multimodal content arrays (OpenAI vision
format) when messages contain images, instead of always flattening to
plain text. This enables vision-capable external providers (OpenAI,
Gemini, Anthropic, etc.) to receive user images. The backend already
handles image_url content parts in _build_external_messages().
* studio: fix external models selectable in chat-only mode (#4779)
* fix: external models selectable in chat-only mode
* fix: model selector tabs default to active model kind
* Studio: API external provider registry + curated catalogs (HF/OpenRouter) and chat UX (#4787)
* fix: external models selectable in chat-only mode
* fix: model selector tabs default to active model kind
* feat(studio): expand provider registry, curated catalogs, and chat UX
- Add Hugging Face, Kimi, Qwen; remove Cohere; reorder registry
- model_list_mode curated for HF/OpenRouter; lightweight /models check
- API returns default models for curated providers; expose model_list_mode
- Frontend: provider logos in model picker, providerType on external models
- Chat providers dialog: curated vs remote flows, motion polish
- Thread: LayoutGroup + composer motion alignment with app easing
* fix(studio): disable Anthropic tool-calling flag and preselect curated defaults
* feat(studio): add external provider logos and ApiProviderLogo helper
* Studio: Polish API Providers dialog (#4899)
* fix: lower verbage in API providers page
* fix: fix(studio): tune API Providers dialog width with rem-based responsive caps
* feat: add custom provider support (#4902)
* fix: replace crypto.subtle with node-forge for HTTP compatibility
crypto.subtle is only available in secure contexts (HTTPS/localhost),
which breaks provider API key encryption when Studio is accessed over
plain HTTP on remote GPU VMs. Switch to node-forge for RSA-OAEP and
AES-256-GCM operations — same algorithms, works on any origin.
* fix: store provider API keys as plaintext in localStorage
Drop AES-256-GCM at-rest encryption for provider API keys. The
session-password-derived encryption broke on auto-login via refresh
token (password never captured), causing keys to silently vanish.
API keys are still RSA-encrypted in transit via node-forge. At-rest
encryption in localStorage added no real security since the
decryption key also had to live client-side.
Removes crypto-storage.ts, session password plumbing, and
reEncryptAllKeys.
* fix: use max_completion_tokens for OpenAI provider
Newer OpenAI models (gpt-4o, gpt-5.x) reject the max_tokens param
and require max_completion_tokens instead. Other providers still use
max_tokens.
* fix: skip empty assistant messages in external provider requests
Some providers (Mistral) reject assistant messages with empty content.
Filter them out when building the message list for external providers.
* Update model-selector.tsx
* Update model-selector.tsx
* Update model-selector.tsx
* Update chat-adapter.ts
* Update chat-adapter.ts
* Update chat-page.tsx
* Update chat-settings-sheet.tsx
* Update chat-settings-sheet.tsx
* Update chat-settings-sheet.tsx
* Update chat-providers-dialog.tsx
* feat: polish providers settings form UI
* style: polish provider row icon sizing and alignment
* style: stabilize provider layout
* style: add provider API key visibility toggle
* fix: add provider render on empty list
* studio/frontend: sync package-lock.json with package.json
npm ci was failing because node-forge and @types/node-forge were
declared in package.json but missing from the lockfile. Ran
npm install to regenerate.
* studio/backend: fix backend CI failures for providers router
- test_desktop_auth: include providers_router in the routes stub so
studio.backend.main imports cleanly under the monkeypatched module
- test_providers_api: skip the whole module when STUDIO_TEST_PASSWORD
is unset (it is an integration test against a live Studio server,
same shape as the already-ignored test_studio_api.py)
* studio/chat: drive ChatSettingsPanel from a per-provider capability map
Replace the binary isExternalModel toggle in the sampling section with a
provider-aware capability map. Each external provider type advertises
which of top_k / min_p / repetition_penalty / presence_penalty its
chat-completions API actually accepts, so the panel only renders the
knobs that map onto the active provider's request body.
Anthropic now exposes top_k; DeepSeek hides presence_penalty (deprecated
in their docs); OpenRouter and custom providers continue to show every
knob (OpenRouter drops unsupported server-side, custom assumes
OpenAI-compat or a permissive vLLM/Ollama backend). Local models are
unaffected — null capabilities means 'show everything'.
chat-adapter.ts now forwards top_k / presence_penalty to the external
proxy only when the active provider's capabilities permit it, so the
request body matches what the UI shows.
* studio/backend: forward top_k to Anthropic; filter OpenAI model list
Two paired changes so the frontend capability map has matching backend
behaviour:
1. ExternalProviderClient.stream_chat_completion now accepts top_k and
forwards it to the Anthropic Messages body. OpenAI-compat providers
(which all reject unknown sampling params) still receive only the
fields they document. The proxy route in routes/inference.py passes
payload.top_k through, so a UI request with top_k actually reaches
Anthropic instead of being silently dropped at the boundary.
2. PROVIDER_REGISTRY['openai'] gains a model_id_allowlist regex that
scopes the /models picker to current-gen ids (gpt-5.5 / gpt-5.4 /
gpt-5.3 / gpt-4.5 / o3 families). The remote /v1/models listing
otherwise returns dozens of historical snapshots, fine-tunes and
non-chat models (embeddings, TTS, image, moderation) that we never
want in the chat UI. default_models is refreshed to match.
* studio/chat: relax presence_penalty to optional on OpenAIChatCompletionsRequest
Followup to 1fbf445a — chat-adapter now omits presence_penalty for
providers that do not accept it (Anthropic / DeepSeek), but the
request type still required it as a non-optional number, breaking
tsc. The backend pydantic model already defaults presence_penalty
to 0, so making it optional client-side matches reality.
* studio/backend: route OpenAI traffic through /v1/responses
OpenAI's new flagship models (gpt-5.x) return 404 'This is not a chat
model' on /v1/chat/completions and are only reachable via /v1/responses.
Add a dedicated _stream_openai_responses path in ExternalProviderClient
that:
- Translates outbound messages into the Responses shape: system messages
are folded into the top-level 'instructions' field, user/assistant
messages become {role, content} items with input_text / input_image
content parts (data URLs and https URLs both pass through).
- Drops presence_penalty / top_k / frequency_penalty, none of which the
Responses contract accepts.
- Translates inbound SSE events back into OpenAI Chat Completions
chunks so the frontend keeps a single SSE shape:
response.output_text.delta -> delta chunk with content
response.completed -> chunk with finish_reason='stop'
response.incomplete -> chunk with finish_reason='length'
response.failed / error -> propagated error SSE line
Stream terminates with data: [DONE] (Responses emits this verbatim).
stream_chat_completion dispatches all provider_type='openai' calls to
this path; other OpenAI-compatible providers (mistral, gemini, etc.)
continue to use /v1/chat/completions.
Frontend provider-capabilities map updated to hide presence_penalty for
OpenAI in the chat settings panel, matching the new request contract.
Includes unit coverage in tests/test_openai_responses_translation.py
exercising the request body translation, image-part rewriting, and
SSE-to-chat-completions translation via httpx.MockTransport.
* studio/chat: clamp external max_tokens to 32k to stay within provider caps
The chat settings slider already capped maxTokens at 32768 for external
models, but a value persisted from a prior local-model session (where
the cap can be 128k+) was sent verbatim to the provider — Claude Opus
returns 'max_tokens: 131072 > 128000' on requests like that, and other
providers have stricter limits still.
Expose EXTERNAL_MAX_OUTPUT_TOKENS from provider-capabilities (32k) and
use it both for the slider max and as the clamp inside chat-adapter's
external-request body. 32k sits below the tightest declared output
limit across the providers we ship and well above what a typical chat
reply needs; the local-model path is unaffected.
* studio: drop temperature/top_p for OpenAI reasoning models
gpt-5.x / o3 / gpt-4.5 are reasoning-class models served via
/v1/responses, and reject temperature and top_p with
'Unsupported parameter' 400s. The OpenAI registry allowlist already
scopes the picker to those families, so neither knob ever applies on
this branch.
- external_provider._stream_openai_responses no longer puts
temperature or top_p in the request body (kept on the method
signature for API symmetry with the other stream methods).
- ProviderCapabilities gains temperature/topP flags; OpenAI sets both
to false. ChatSettingsPanel hides the sliders for OpenAI so the user
does not see inert controls.
- chat-adapter omits temperature/top_p from the external request body
when the active provider does not advertise them.
- OpenAIChatCompletionsRequest type marks both as optional, matching
the new chat-adapter shape.
- test_responses_request_body_uses_input_and_instructions: assertions
flipped to confirm temperature / top_p are absent from the body.
* studio: stop forwarding top_k to Anthropic
Claude 4.x (Opus / Sonnet / Haiku 4.x) returns 400 'top_k is
deprecated for this model' on any request that includes top_k. It
was always optional on the older 3.x line, so dropping it
unconditionally for every Anthropic call is the simplest path —
no per-model gate to maintain.
- external_provider._stream_anthropic no longer adds top_k to the
Messages body (kept on the method signature for API symmetry).
- provider-capabilities sets anthropic.topK = false so the chat
settings panel hides the Top K slider for Anthropic providers
and chat-adapter does not send top_k in the external request.
* studio: gate Anthropic top_k drop to Claude 4.7 only
Previous commit (b5aa6ffd) dropped top_k for every Anthropic call,
but only Claude 4.7 (Opus/Sonnet/Haiku) actually rejects it. 4.6, 4.5,
and the 3.x line still accept top_k and use it as documented.
Backend: _stream_anthropic matches the model id against
^claude-(opus|sonnet|haiku)-4-7(-|.|$) and only strips top_k when it
hits. Every other Claude generation continues to receive the value
from the chat settings panel.
Frontend: anthropic.topK is restored to true so the Top K slider is
visible again — the backend handles the per-model drop, and the
4.7 case is silent (request still succeeds without top_k).
* chore: hide dated openai models in provider select
* studio/providers: apply model_id_denylist when listing remote models
The OpenAI registry entry gained a model_id_denylist regex matching
dated snapshot ids (-YYYY-MM-DD) in 048d73bf, but the list-models
route was never consulting it, so the snapshots still showed up
alongside their canonical ids (gpt-5.5 and gpt-5.5-2026-04-23 both
listed). Apply the denylist with .search() right after the allowlist
filter so dated entries are dropped before the response is built.
* studio/chat: seed registry default_models for remote providers in picker
The Anthropic provider runs in remote model-list mode, so the picker
started with an empty availableModels until the user clicked
'Load Models'. If that /api/providers/models call fails (e.g. the
known transient decryption error during key rotation), the user sees
no models at all — claude-haiku-4-5 in particular was missing from
the dialog even though it is seeded in the registry.
Always pre-populate availableModels with the registry's default_models
when a provider type is selected (curated and remote alike), and have
loadModels() return the union of defaults + the live /models response
so registry-seeded ids are reachable regardless of what the provider's
endpoint returns or whether the call succeeds at all.
* studio/backend: diagnostic logging on provider key decryption
Decryption failures currently log just 'Failed to decrypt API key:
Decryption failed', which leaves no way to tell whether the cause is
a stale public key in the browser, a corrupted ciphertext, an
unexpected exception class, or a server-side keypair rotation. That's
the gap the next reproduction needs to close.
- key_exchange now publishes a short SHA256 fingerprint of the public
key PEM. init_key_pair logs the fingerprint on generation and warns
if it is ever called a second time (re-init silently invalidates
every browser that cached the previous public key).
- decrypt_api_key wraps both the base64 decode and the RSA decrypt
in dedicated try/excepts that log exception type, ciphertext byte
length (RSA-2048 should be exactly 256), input string length, and
the current public-key fingerprint.
- GET /api/providers/public-key returns the fingerprint alongside the
PEM so the frontend can correlate a future encrypt-time fingerprint
against the decrypt-time fingerprint and prove or rule out a
keypair rotation as the cause.
- The /test and /models route-level decrypt warnings now include the
exception class name (alongside the existing message).
* studio/providers: hide dated Anthropic snapshots from the model picker
Anthropic's /v1/models returns dated snapshot ids (e.g.
claude-3-5-sonnet-20241022, claude-3-5-haiku-20241022) alongside
the canonical names users actually want to pick. Same intent as
the OpenAI denylist added in 048d73bf, just a different date
format — Anthropic uses -YYYYMMDD (no dashes) while OpenAI uses
-YYYY-MM-DD.
- Add model_id_denylist = re.compile(r'-\d{8}$') to the anthropic
registry entry. The /api/providers/models route already applies
any denylist after fetching, so dated ids drop out automatically.
- Strip the dated 3.5 ids from default_models so the seeded picker
no longer surfaces them; keep claude-opus-4-7 and the 4.5 family
as the curated set.
Net effect: the picker shows opus-4-7 / opus-4-5 / sonnet-4-5 /
haiku-4-5 only, regardless of whether the remote /models call
succeeds or fails.
* fix: provider dialog and mistral short list
* style: fix provider dialog curated list styling
* fix: provider dialog curated model ids placeholder reference
* style: rename Providers to Cloud and tighten dialog header spacing
* UX: rename Providers to Cloud, remove header shortcut
* studio/chat: normalize structured delta.content from reasoning providers
Mistral's magistral (and similarly-shaped reasoning models) stream
chat-completion deltas where choices[0].delta.content is an array of
structured parts rather than a plain string, e.g.
[{ type: 'text', text: '...' }, { type: 'thinking', thinking: '...' }]
The accumulator did 'cumulativeText += delta', which coerced each
part to '[object Object]' and produced output like
'[object Object][object Object]...Hey there!'.
Add extractDeltaText() to normalize delta.content before append:
- string → returned as-is
- array of parts → text/output_text parts contribute their .text or
.content; thinking/reasoning parts are re-wrapped inline as
<think>...</think> so the downstream parseAssistantContent lifts
them into a reasoning part the same way it does for providers that
emit thinking inline. magistral keeps its thinking panel; no other
provider's output shape changes.
- unknown shapes → dropped rather than stringified, so a stray field
cannot pollute the rendered chat with '[object Object]'.
* Studio: restore Cloud icon shortcut in chat header
Brings back the header chip that opens Settings -> Cloud (external
providers) directly from the chat view. Same button as before the
bf24e604 removal: single-mode only, opens useSettingsDialogStore on
the 'connections' tab, tooltip 'API providers'.
* studio/chat: strip trailing template literal from external provider streams
Mistral's magistral occasionally appends a literal '${response}' token
after its actual answer — likely a training-format artifact, since it
keeps happening with an empty system prompt and only on that model.
Apply a tight strip in the chat-adapter SSE accumulator: when the
active provider is external, drop a trailing '${...}' template literal
(with optional whitespace) from cumulativeText after each chunk. The
regex anchors to end-of-string, so mid-stream fragments ('${re')
remain untouched and only collapse once the closing brace arrives.
Local-model output is unaffected.
* studio/providers: scope Kimi picker to kimi-k2.6 / kimi-k2.5
Mirror what the live Kimi docs surface as the current models
(https://platform.kimi.ai/docs/models). Everything else the
remote /v1/models call returns — moonshot-v1-* legacy ids and
dated k2 previews like kimi-k2-0711-preview — is filtered out.
- default_models: ['kimi-k2.6', 'kimi-k2.5'] (was four
legacy moonshot-v1 ids plus the dated k2 preview)
- model_id_allowlist: ^kimi-k2\.[56]$ applied in the
/api/providers/models route after the live fetch
- doc-link comments point at platform.kimi.ai overview /
models / list-models for the next refresh
* studio: drop temperature/top_p for Kimi reasoning models
Kimi k2.5/k2.6 are reasoning-class. The API locks temperature and
top_p to fixed defaults and 400s on any other value with
'invalid temperature: only 1 is allowed for this model'.
The frontend capability map already gated these knobs out of the
external request body, but the OpenAI-compat path on the backend
unconditionally re-adds them from the pydantic ChatCompletionRequest
defaults (temperature=0.7 etc), so the gate was bypassed end-to-end.
Add a generic body_omit hook on the provider registry that
stream_chat_completion consults after building the body, and use it
to strip temperature/top_p for Kimi. Frontend provider-capabilities
flips kimi.temperature and kimi.topP to false so the sliders are
hidden in the chat settings panel as well.
* studio/providers: scope Gemini picker to current 3.x + *-latest aliases
Google's /v1beta/openai/models returns dozens of historical,
experimental, and non-chat ids that we never want in the chat UI.
Cap the picker to the current curated set:
- gemini-3.1-pro-preview
- gemini-3.1-flash-lite
- gemini-3-flash-preview
- gemini-pro-latest
- gemini-flash-latest
- gemini-flash-lite-latest
Default_models seeded with these, model_id_allowlist applied in
the /api/providers/models route to drop anything else the live
fetch returns.
* studio/providers: switch Hugging Face to remote model listing
Per the Inference Providers docs
(https://huggingface.co/docs/inference-providers/index),
GET https://router.huggingface.co/v1/models returns the full
chat-model catalog across all providers, including per-provider
metadata. The OpenAI-compatible endpoint we already use for
chat completions accepts the same Bearer token, so flipping
model_list_mode from 'curated' to 'remote' lets users discover
models via the existing list_models() path without any new
wiring.
- model_list_mode: 'remote' (was 'curated')
- default_models refreshed with current popular ids
(gpt-oss-120b, DeepSeek-V3, Llama-3.3-70B, Qwen2.5-72B) so the
picker still has a sensible seed if /v1/models fails
- notes updated to reference the docs page and clarify the
endpoint is chat-only
* UX: chat cloud icon changed to model select signifier
* studio/providers: org allowlist + count cap for HF Inference picker
The HF /v1/models response is the full cross-provider catalog (hundreds
of ids — community fine-tunes, mirrors, fp8 variants, dated snapshots).
Scope the picker to the first-party org repos worth surfacing and cap
the post-filter list.
- model_id_allowlist matches the org prefixes openai/, deepseek-ai/,
google/, meta-llama/, Qwen/, moonshotai/, mistralai/, zai-org/.
Anything outside those orgs is dropped.
- model_id_limit (new registry field) caps the post-filter list. The
list-models route now slices [:limit] after allowlist/denylist; set
to 15 for HF Inference. Other providers leave it unset and behave
exactly as before.
- default_models stays as the seed so the flagship ids users care
about (gpt-oss-120b, DeepSeek-V3, Llama-3.3-70B, Qwen2.5-72B) are
always reachable regardless of the API's response order.
Dedup is already handled in loadModels() via Set, so no additional
work needed there.
* style: adjust cloud icon right margin with rem spacing
* Studio: cloud openai reasoning level toggle (#5402)
* feat: cloud openai reasoning level toggle
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: honor enable_thinking=false
* fix: prevent local reasoning toggle regressions and align OpenAI effort levels
* fix: isolate external OpenAI reasoning toggle state
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
* fix: clamp reasoning effort
* fix: align OpenAI reasoning effort
* fix: clear stale GGUF badge state
* ui: new badge on cloud setting
* fix: separate selected models from cached provider model list
* Studio: anthropic effort by model family (#5412)
* feat: external thinking control and Anthropic effort mapping
* fix: anthropic thinking constraints and 4.6 max effort mapping
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: harden Anthropic thinking params and effort mapping
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio/backend: drop top_p from Anthropic body when thinking is enabled
PR 5412 added body['top_p'] = max(0.95, min(top_p, 1.0)) inside the
thinking branch of _stream_anthropic, but Anthropic returns 400 on
extended/adaptive thinking when both temperature and top_p are set:
invalid_request_error: temperature and top_p cannot both be
specified for this model. Please use only one.
(Observed on Claude Opus 4.6.) The contract for thinking-enabled
requests is temperature=1 with neither top_p nor top_k allowed.
Replace the body['top_p'] = ... line with body.pop('top_p', None).
Defensive pop rather than a bare delete: the base body construction
above does not currently set top_p, but a future edit that adds it
would silently reintroduce the regression.
* studio/chat: force reasoningEnabled=true on local reasoning-effort models
Followup to PR 5402 / 5412. The model-status refresh path in
use-chat-model-runtime carried reasoningEnabled forward verbatim for
every reasoning-capable model. That left one observable edge case:
1. user picks an external model that supports Off (gpt-5.x, Claude
4.x), clicks Off — store sets reasoningEnabled=false
2. user switches back to a local reasoning-effort model
(gpt-oss / Harmony-style) which does NOT support Off
3. composer's effectiveReasoningEnabled override paints the UI as
'Think: <level>' (on)
4. chat-adapter sees reasoningEnabled=false on the local branch
and sends '{}', so the backend's _request_reasoning_kwargs
returns None and the Harmony template falls back to its own
default effort instead of the displayed level
Mirror the composer's override in the store on load: for local
reasoning-effort models (where supportsReasoningOff is false), force
reasoningEnabled=true so the store and the UI agree on every send.
Other reasoning styles still inherit prior state — only the
reasoning-effort family changes.
* studio/backend: align Anthropic thinking with the extended-thinking docs
Two compliance fixes against
https://platform.claude.com/docs/en/build-with-claude/extended-thinking
1. Adaptive-mode effort field shape
The docs spell adaptive thinking as:
{'thinking': {'type': 'adaptive'}, 'effort': {'type': '<level>'}}
We had been sending the legacy 'output_config: {effort: <level>}'
shape, which Anthropic appears to silently ignore — adaptive ran
at the server default effort regardless of the user's selection.
Rename to 'effort: {type: <level>}'.
2. thinking_delta event translation
The Messages-API streams reasoning content as
content_block_delta events with delta.type == 'thinking_delta',
which our SSE loop was dropping entirely. On Claude 4.5/4.6 with
display=summarized (the default), the user would see the answer
text but never the reasoning panel. Wrap thinking_delta.thinking
as inline <think>...</think> chunks (same pattern as the OpenAI
Responses path) so the frontend's parseAssistantContent lifts it
into the reasoning channel. The </think> closer fires on the
first text_delta transition, on content_block_stop for the
thinking block, on message_delta, and on message_stop —
whichever arrives first — so no model path can leak an
unclosed <think> into chat output.
signature_delta events are left as no-ops; they carry
verification metadata, not user-visible content.
Adds test_anthropic_thinking_translation.py with httpx.MockTransport
coverage of: effort shape on adaptive (Claude 4.6), budget_tokens
shape on manual (Claude 4.5), thinking_delta wrapping with signature
suppression, and thinking-only turns (display=omitted on Opus 4.7).
* studio/backend: revert Anthropic adaptive effort to output_config nesting
The previous commit (0a664df4) moved the adaptive-thinking effort
field to a top-level 'effort: {type: <level>}' based on a misread of
the docs page. The actual Messages API schema nests it under
output_config:
thinking: optional ThinkingConfigParam ({type: 'adaptive'})
output_config: optional OutputConfig
effort: optional 'low' | 'medium' | 'high' | 'xhigh' | 'max'
Sending the top-level field produced:
400 invalid_request_error: effort: Extra inputs are not permitted
Restore the body to:
body['thinking'] = {'type': 'adaptive'}
body['output_config'] = {'effort': effort}
This was the shape PR 5412 originally shipped (and the author
validated against live APIs). My 'compliance fix' was a regression.
The companion thinking_delta SSE translation added in 0a664df4 stays
— that part WAS missing from the previous shape and is unchanged
by this revert. Test pinning the body shape flipped to assert
output_config.effort, top-level effort is asserted absent.
* studio/backend: opt in to summarized thinking display on adaptive
Per the adaptive-thinking docs, the 'display' field on the thinking
config defaults to 'omitted' on Claude Opus 4.7 (and Mythos Preview).
With 'omitted' the API still emits a thinking content block, but its
'thinking' field is empty — only the signature_delta arrives.
Our SSE handler would then surface a stray '<think></think>' for the
empty block and the reasoning panel would stay blank for the entire
response. Set 'display': 'summarized' explicitly on the adaptive
thinking config so Opus 4.7 emits thinking_delta events the same way
Opus 4.6 / Sonnet 4.6 do (where 'summarized' is the default, making
the explicit setting a no-op there).
The manual-thinking branch (Claude 4.5) is unaffected — its default
is also 'summarized', and we have no reason to override it.
* studio/backend: log Anthropic SSE event counts for thinking diagnostics
Reports of 'no reasoning panel content on Anthropic' have two
distinct causes that produce the same symptom:
1. Anthropic streamed thinking_delta events but our frontend
dropped them somewhere on the rendering side.
2. Anthropic did not emit thinking_delta at all (adaptive mode
can skip thinking for simple prompts even with effort=high,
and display=summarized only re-enables the *content* — it
does not force thinking to happen).
Tally each event type for the duration of one stream and log the
counts in the finally branch, so the next 'no reasoning content'
report shows immediately whether thinking_delta was even on the
wire. Zero counts → upstream (model/effort/prompt choice).
Non-zero counts → triage moves to chat-adapter / parse-assistant
-content / the reasoning component.
* studio/backend: route external_provider logs through structlog
The studio backend wires structlog as the active logger (via
LogConfig.setup_logging at main.py:262), but external_provider.py
was using stdlib logging.getLogger(__name__) for every diagnostic.
The stdlib root logger defaults to WARNING with no handlers
attached, so plain logger.info('...') and logger.debug('...') from
this module were being silently dropped — including the
'Proxying chat completion to <url>' and the new
'Anthropic stream event counts' lines. Only WARNING/ERROR survived
(via the implicit fallthrough that the user actually observed
when an Anthropic call 400'd).
Switch the module-level logger to structlog.get_logger(__name__),
matching the routes/providers.py and routes/inference.py pattern.
All existing call sites use printf-style positional args, which
structlog accepts unchanged — no other edits needed.
* studio/backend: disable read timeout on SSE streams to external providers
Anthropic Opus 4.7 (adaptive thinking) and OpenAI gpt-5.x (/v1/responses)
can pause for tens of seconds between bytes while the model is
internally reasoning. httpx's read timeout is the *gap* between
successive reads, not a wall clock on the whole request — so the
shared 120s default was cutting streams mid-response:
log: Anthropic stream event counts (... text_delta: 11)
Read timeout from anthropic
(eleven text deltas in, no content_block_stop, no message_stop)
Add a separate _stream_timeout on ExternalProviderClient with
read = None (no gap timeout) and the same 10s / 120s connect/write/
pool bounds, then use it at the three SSE streaming call sites:
default OpenAI-compat chat completions, _stream_anthropic, and
_stream_openai_responses. Non-streaming call sites (chat_completion,
list_models, verify_models_endpoint_lightweight) keep self._timeout
because a stuck non-streaming response should still fail fast.
* studio/backend: log outbound Anthropic request shape for thinking debug
After bumping to Xhigh effort the user still saw zero thinking_delta
events and only one content_block_start, meaning Anthropic Opus 4.7
opened no thinking block at all. Per the effort docs that should be
impossible — Xhigh always thinks. Two open hypotheses:
1. Our adaptive branch is not wiring output_config.effort onto the
outbound body for this code path (regex miss, frontend never
propagated reasoning_effort, etc).
2. Anthropic is silently accepting output_config as an unknown
field and falling back to high default effort regardless.
Add a single-line structlog INFO right before the stream POST that
echoes the keys actually present on the body (thinking, output_config,
temperature, presence of top_p / top_k, max_tokens). Messages are
deliberately excluded to keep PII out of the log. With this in place
the next 'no thinking on 4.7 at Xhigh' report shows immediately
whether we sent the effort knob — separating client bug from
provider behaviour.
* studio/chat: surface delta.reasoning_content from Kimi / DeepSeek thinking
Kimi (kimi-k2.6, kimi-k2-thinking) and DeepSeek's reasoner stream
their thinking content via a separate top-level field on the
chat-completion delta — choices[0].delta.reasoning_content — rather
than as a structured part inside delta.content. Per Kimi docs:
In streaming output (stream=True), the reasoning_content field
will always appear before the content field.
Our chat-adapter SSE loop only read delta.content (via
extractDeltaText), so the entire reasoning channel from these
providers was being silently dropped — kimi-k2.6 thinks by default
yet the chat UI showed no reasoning panel.
In the adapter:
- Read both delta.content and delta.reasoning_content per chunk
- When reasoning_content arrives, open a <think> block in
cumulativeText (mirrors how the backend wraps Anthropic
thinking_delta and OpenAI Responses reasoning summaries)
- When content arrives after reasoning, close </think> first
- On stream end, force-close any still-open <think> so
parseAssistantContent can lift it into a reasoning part cleanly
Anthropic and OpenAI Responses paths are unaffected — they already
wrap as <think> on the backend and never set reasoning_content.
* studio: Kimi thinking toggle + 16k max_tokens floor
Two coordinated changes so Kimi's thinking is user-controllable and
the response budget meets the docs' floor.
Toggle (frontend + backend):
- getExternalReasoningCapabilities now handles provider=='kimi':
kimi-k2.6 -> reasoning_style=enable_thinking, reasoningOff allowed
kimi-k2-thinking -> always on (reasoningAlwaysOn=true, no off)
kimi-k2.5 (and anything else) -> no reasoning controls
- chat-adapter already forwards enable_thinking on the
enable_thinking-style branch, so the user toggle reaches the
backend without additional wiring there.
- external_provider stream_chat_completion now translates the
boolean into Kimi's wire shape on the default OAI-compat path:
enable_thinking=True -> body['thinking'] = {type: enabled, keep: all}
enable_thinking=False -> body['thinking'] = {type: disabled}
kimi-k2-thinking ignores the toggle so the API never gets a
disabled value it would reject. Other providers on the same
path are unaffected (gated on provider_type == 'kimi').
Max tokens floor:
- New EXTERNAL_MIN_OUTPUT_TOKENS_BY_PROVIDER table and
getExternalMinOutputTokens helper. Kimi entry = 16000 per docs:
'Set max_tokens >= 16,000 to ensure the full reasoning_content
and final content can be returned without truncation.'
- chat-adapter clamps the outbound max_tokens to
min(max(stored, providerMin), EXTERNAL_MAX_OUTPUT_TOKENS),
so a stored value of 4096 still becomes 16000 when sending to
Kimi (other providers unaffected, min stays effectively 64).
- chat-settings-sheet's Max Tokens slider min mirrors the same
floor when an external Kimi model is selected, so the slider
cannot show a value lower than what we'd actually send.
- chat-page threads activeExternalProviderType down to the panel.
* fix: stabilize external reasoning controls for Anthropic 4.6 and OpenAI o3
normalize Anthropic 4.6 reasoning effort handling by accepting max as an alias and mapping it to xhigh, while keeping Sonnet/Opus 4.6 in default model suggestions.
broaden reasoning effort typing across backend/frontend and migrate persisted max selections to xhigh for compatibility.
remove reasoning.summary=\"auto\" from OpenAI /v1/responses payloads to avoid o3 eligibility/gating errors.
tighten provider model filtering to hide retired gpt-5.3 IDs and add exact/prefix filtering support in provider routes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: add openrouter/free + full reasoning passthrough on OpenRouter
Four-layer wire-up so the OpenRouter free-router model (which picks
a free model at random per request, filtered by needed capabilities)
shows up in the picker and its reasoning channel surfaces in the
chat UI.
Registry:
- providers.py: openrouter/free seeded at the top of openrouter
default_models. Curated list, so picker shows it immediately.
Frontend capability map:
- provider-capabilities.ts: getExternalReasoningCapabilities now
treats openrouter as enable_thinking style with off support. The
Think dropdown appears for every OpenRouter model; the gateway
silently no-ops the parameter for models that do not reason, so
surfacing one toggle on every model is safe.
Backend reasoning passthrough:
- external_provider.py stream_chat_completion (default OAI-compat
branch): for provider_type=='openrouter', translate the request:
reasoning_effort in {low,medium,high} -> body['reasoning'] =
{'effort': <level>}
enable_thinking=True -> body['reasoning'] = {'enabled': True}
enable_thinking=False -> body['reasoning'] = {'enabled': False}
Matches the documented shape at
https://openrouter.ai/docs/guides/best-practices/reasoning-tokens
with effort and max_tokens mutually exclusive.
Frontend SSE reader:
- chat-adapter.ts: OpenRouter streams reasoning as a third shape we
did not handle yet: delta.reasoning_details is an array of parts
like {type: 'reasoning.text', text: '...'}. Pull text from every
part, merge with the existing delta.reasoning_content channel
used by Kimi/DeepSeek, and feed the combined string through the
same <think>...</think> wrap path so parseAssistantContent lifts
it into the reasoning panel. Anthropic/OpenAI Responses paths
already wrap on the backend, so they never set this field — no
cross-provider interference.
* studio/backend: surface OpenRouter SSE errors and router-chosen model in logs
The frontend showed 'Provider returned error' for some openrouter/free
requests with nothing on the backend side to triage from — the
existing 4xx error log only fires when the upstream returns a non-200
status code, but OpenRouter (and most OAI-compat providers) return
200 OK and emit the actual failure as an SSE error event mid-stream,
which our default-path stream loop forwarded verbatim without
logging.
Best-effort diagnostics on the default OpenAI-compat stream path:
- Peek at every `data:` line in the inner forward loop, parse JSON
best-effort (silently skip on failure so nothing is dropped).
- Count event types: delta / error / done.
- On any chunk containing an `error` field, emit a structlog WARNING
with the provider type and the error payload — same trail the
user would otherwise have to dig out of browser devtools.
- Latch the first non-empty `chunk.model` field. OpenRouter reports
the router-picked underlying model there per request, so the
finally-block summary log shows which free model handled the call.
In the finally block:
'openrouter stream complete (model=openrouter/free,
chosen=google/gemini-2.5-flash, events={delta: 47, done: 1})'
Zero overhead for non-error streams (a json.loads per chunk +
dict-key lookups). The structlog logger is already configured at
INFO; ERROR and WARNING surface in JSON logs without further setup.
Hoists `import json as _json` to module top so the default path can
reuse it; the existing in-function imports in _stream_anthropic and
_stream_openai_responses are now redundant but harmless.
* studio/chat: show router-picked model after 'openrouter/free:' in chip
When the user picks openrouter/free, the gateway routes each request
to a different underlying free model. Until now there was no way to
tell which one actually replied without reading the backend logs.
Surface the picked model in the active-model chip:
- chat-runtime-store gains lastOpenRouterChosenModel: string|null
plus a setter. Reset on every model switch unless the user stays
on openrouter/free.
- chat-adapter SSE loop latches chunk.model into the store on
every chunk whose top-level model differs from
openrouter/free, gated on the active checkpoint being
openrouter/free under an OpenRouter provider.
- chat-page externalModels useMemo appends :<chosen> to the display
name for the openrouter/free option when the store has a value,
so ModelSelector renders e.g.
'openrouter/free:google/gemini-2.5-flash'
in the chip. Other models unaffected.
- Model-switch callback in chat-page clears the cached value when
the user moves to any model other than openrouter/free, so the
chip never shows a stale suffix from a previous session.
* studio/chat: shorten openrouter/free chip to openrouter:<short-chosen>
The full display name in use was:
openrouter/free:inclusionai/ring-2.6-1t-20260508:free
The `:free` suffix on the underlying id already conveys 'free model',
which made the leading `/free` on the router id redundant, and the
`inclusionai/` org prefix was just noise crowding the chip.
Trim both. Now the chip renders as:
openrouter:ring-2.6-1t-20260508:free
Strictly a display change in chat-page externalModels useMemo — the
backend wire id stays `openrouter/free`, the runtime store still
caches the full `inclusionai/...:free` value, and the model-switch
clearing logic is unchanged.
* studio/providers: switch OpenRouter to remote listing with org allowlist + cap
Same shape as Hugging Face Inference. The curated list had only four
entries; remote listing fetches OpenRouter's full ~300-model
catalog via /v1/models and the new allowlist + limit scope it back
down to a usable picker.
- model_list_mode: remote (was curated)
- model_id_allowlist matches the prefixes:
openrouter | openai | anthropic | google | meta-llama | qwen
| mistralai | deepseek | moonshotai | inclusionai | zai-org
| z-ai
Anything outside drops out.
- model_id_limit: 20 — first 20 post-filter matches from the live
fetch; default_models stays seeded so the most useful canonical
ids are always visible regardless of API response order.
- default_models seed extended from 4 to 6 (openrouter/free,
openai/gpt-4o, anthropic/claude-sonnet-4-5, google/gemini-2.5-flash,
mistralai/mistral-large-2411, deepseek/deepseek-r1).
openrouter/free remains the first entry, so the dialog's
loadModels() union-merge (registryDefaults first, then remote,
deduped via Set) keeps it at the top of the picker.
* feat: external mistral thinking toggle
* studio/chat: fix TS2540 by replacing readonly ContentPart instead of mutating
The ContentPart type from @assistant-ui/react marks `text` as readonly,
so the coalesce-adjacent-same-type-part optimization in
parseAssistantContent failed the tsc build with:
parse-assistant-content.ts(15,10): error TS2540: Cannot assign to
'text' because it is a read-only property.
parse-assistant-content.ts(25,10): error TS2540: ...
This broke npm run build, the Studio installer's `building frontend...`
step, and every downstream CI job that runs against an installed
Studio (Mac/Windows/Linux variants of Studio API CI, GGUF CI, UI CI,
Tauri CI, Wheel CI).
Replace the last element with a fresh merged object instead of
mutating its `text` field. Same allocation profile as the previous
path (one object swap per merge), type-safe under the readonly
declaration. Behaviour unchanged.
* studio/backend: restore summary='auto' on OpenAI Responses reasoning body
A recent refactor dropped the `summary: 'auto'` field from the
reasoning config we send to /v1/responses. Without it OpenAI does
not emit reasoning summary events on most reasoning models, which
means our SSE handler has no <think>…</think> to wrap and the chat
reasoning panel stays blank for any gpt-5.x / o3 response.
The expected wire shape is:
body['reasoning'] = {'effort': '<level>', 'summary': 'auto'}
Two backend tests pin this:
- test_responses_reasoning_effort_included_when_requested (high)
- test_responses_reasoning_effort_xhigh_passthrough (xhigh)
Both were failing with AssertionError because the produced body
omitted `summary: auto`.
Restore the field. Skip it only for the explicit "off" case
(effort: 'none'), where summaries serve no purpose. The
enable_thinking=True fallback (no explicit effort) also pairs
medium effort with summary='auto' so that branch produces
reasoning text too.
* chat: external reasoning, OpenRouter curation, Think toggle fixes
* fix: opus and sonnet 4.6 xhigh --> max
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* studio: allow huggingface.co and datasets-server.huggingface.co in CSP connect-src
The security hardening pass (0881a7a5) added connect-src 'self', which
blocked the Training page's direct browser calls to HuggingFace. Model
search (@huggingface/hub listModels/modelInfo/whoAmI -> huggingface.co)
and dataset subset/split discovery (datasets-server.huggingface.co/splits)
both returned nothing as a result.
Extend connect-src to permit the two HF hosts the SPA actually talks to.
No other directive changes; HF tokens still stay client-side.
* studio: format FastAPI 422 detail arrays in training error messages
readError in train-api.ts stringified payload.detail directly. On a 422
the detail is an array of {loc, msg} objects, which JS coerces to
'[object Object],[object Object]' -- the UI showed that instead of the
actual validator message.
Format the array into 'field.path: msg; ...' so the offending field and
the validator's message surface in the UI and toast.
* studio: allow num_epochs/max_steps = 0 sentinel through TrainingStartRequest
The hyperparameter validators added in the security pass rejected 0 for
both num_epochs and max_steps. But Studio's steps-vs-epochs toggle uses
0 as a sentinel: when training by max_steps the frontend sends
num_epochs=0, and when training by epochs it sends max_steps=0. The
trainer expects this and ignores the zeroed field.
Widen both validators to [0, MAX]. They still catch the actual
out-of-range and non-integer inputs they were added for.
* studio: reject TrainingStartRequest when num_epochs and max_steps are both 0
Each field's validator accepts 0 as a "use the other one" sentinel, but
on their own they don't catch the case where both are 0 (or max_steps
is None and num_epochs is 0). That payload would otherwise produce a
no-op training job. Add a model-level validator that rejects it with a
clear 422 message.
* studio: add Optional[int] type hints to _check_max_steps and _check_warmup_steps
Brings these two validators in line with the rest of the TrainingStartRequest
validators in the same file, which all carry explicit cls/v/return hints.
* studio: contain export and dataset paths under their configured roots
resolve_under_root and resolve_dataset_path previously returned absolute
paths unchanged, so an authenticated client could supply
save_directory="/tmp/escape" (or any other absolute path) and have the
exporter drop adapter files anywhere the server user could write. This
turned up during a recent audit pass where an authenticated POST to
/api/export/export/lora with save_directory="/tmp/lora_escape_test"
returned 200 and wrote adapter_model.safetensors, adapter_config.json,
and tokenizer files under /tmp.
The fix is two-layered:
storage_roots.py adds an _assert_contained(resolved, root) helper that
runs after path resolution and rejects any result whose realpath does
not sit under realpath(root). resolve_under_root now rejects '..'
segments and null bytes outright, and only accepts absolute inputs when
they are already inside the configured root (internal call sites that
re-resolve a stored absolute path stay idempotent;
worker.py:resolve_output_dir(output_dir) etc. continue to work).
resolve_dataset_path picks up the same containment rule, scoped to the
three dataset roots.
models/export.py adds field_validator("save_directory", mode="before")
to ExportCommonOptions and ExportGGUFRequest so bad input fails fast at
422 with a clear message rather than a 500 deep inside the resolver.
The validator rejects empty/whitespace, null bytes, control chars,
strings longer than 255 chars, absolute paths, and '..' segments.
routes/export.py:_export_details now returns os.path.relpath(output_path,
exports_root()) so the Export Complete dialog and /api/models/loras no
longer leak the absolute install prefix to the UI; the basename is
used as a last-resort fallback.
Verified end to end:
- POST /api/export/export/lora {"save_directory":"/tmp/foo"} -> 422
"save_directory must be a name or relative path under the export
root; absolute paths are rejected". /tmp/foo is not created.
- "../../etc/escape" -> 422 "may not contain '..' segments".
- save_directory="my_subdir" -> still accepted (400 only because the
test had no checkpoint loaded yet, not because of validation).
- Internal idempotent re-resolve via resolve_export_dir(absolute path
that is already under exports_root) returns the same path unchanged.
* studio/sandbox: harden bash + python tool execution
The sandboxed Bash and Python tool channels in Chat ran with a thin
preexec hook (PR_SET_NO_NEW_PRIVS + RLIMIT_FSIZE only). Bash had a
small word blocklist; Python had an AST safety pass aimed at
signal-tampering and shell-escape primitives. An audit pass showed
several gaps that a tool-calling model could trigger inadvertently:
- bash curl/wget/nc reached AWS IMDSv2 and returned live STS
credentials for the instance role.
- python "import socket; s.connect((169.254.169.254, 80))"
reached the same endpoint regardless of the bash blocklist.
- "cat /etc/passwd" was blocked at the bash side (because "passwd"
is in the blocklist), but "open('/etc/passwd').read()" in Python
happily returned its contents.
- "chr(115)+chr(117)+chr(100)+chr(111)" style dynamic-arg
construction slipped through the AST shell-escape check.
- The supervisor used proc.kill() on timeout, which only signals
the immediate pid; bash-backgrounded children survived. A fork
bomb could spawn for the full 300s timeout window.
- Session work directories under ~/studio_sandbox/<id>/ were
created with default umask (0o755), so any other UID on the host
could enumerate them.
- session_id sanitisation used a one-shot str.replace("..",""),
which is non-iterative and a small footgun.
This commit takes a conservative middle path: the sandbox still
runs as the Studio UID with no namespace tricks where the kernel
disallows them, but every chokepoint is tightened.
_sandbox_preexec now:
- calls os.setsid() so children share a process group; the
supervisor uses os.killpg(SIGKILL) on timeout/cancel so
backgrounded children die with the parent (new _kill_process_tree
helper, wired into _cancel_watcher and both _bash_exec /
_python_exec timeout branches).
- calls os.umask(0o077) so files the child writes default to 0o600.
- applies PR_SET_PDEATHSIG=SIGKILL so an orphaned child dies if
Studio exits.
- best-effort unshare(CLONE_NEWNET) for a private network namespace
(failure is logged and swallowed; defense-in-depth is still in
place via the bash blocklist and the AST checker below).
- sets RLIMIT_NPROC=10000 (tunable via UNSLOTH_STUDIO_SANDBOX_NPROC),
RLIMIT_AS=8GB, RLIMIT_CPU=300, RLIMIT_NOFILE=1024. The 10k NPROC
figure is chosen to sit well above the ~500 LWPs a healthy Studio
+ llama-server combination already uses while still capping a
runaway fork bomb. NPROC counts LWPs per real UID, so a lower
figure (e.g. 256) starves legitimate bash forks
("bash: fork: retry: Resource temporarily unavailable").
_get_workdir:
- rejects session_id that doesn't match [A-Za-z0-9_-]{1,64};
non-matching values bucket into a shared "_invalid" dir.
- chmod 0o700 on both the workdir and on ~/studio_sandbox/ so
other UIDs cannot read another session's contents.
_BLOCKED_COMMANDS_COMMON gains: doas, pkexec, halt, poweroff, curl,
wget, nc, ncat, netcat, socat, ssh, scp, sftp, rsync, eval, source.
The intent is to keep general bash usage working (echo, ls, pipes,
loops, for, head, etc.) while denying the obvious egress and
escalation paths.
The AST checker (_check_signal_escape_patterns) is split into the
existing shell/signal/loop checks plus a new narrow IO denylist:
- Always flag non-literal args to anything in _SHELL_EXEC_FUNCS,
not just _STRING_SHELL_FUNCS. Closes the dynamic-arg bypass.
- Reject calls to socket.create_connection, socket.socket().connect,
urllib.request.urlopen, http.client.HTTP*Connection, requests.*,
httpx.* whose literal host argument is in a cloud-metadata
denylist (169.254.169.254 + 169.254.* + 100.64.*, plus the
GCP/Alibaba/ECS metadata hostnames and IPv6 link-local). Public
hosts (example.com, huggingface.co, ...) still work. Dynamic
hosts cannot be statically blocked; mitigated by the bash
blocklist + the netns where the kernel allows it.
- Reject literal open("/etc/passwd"), /etc/shadow, /etc/sudoers,
/etc/ssh/*, and /proc/<pid>/environ. Other files
(/etc/os-release, /etc/hostname, /tmp/*, user dirs) still work.
The _check_code_safety summariser is updated to include the new
network_calls and sensitive_file_reads buckets in its error string.
Regression-checked: echo, sleep, ls /tmp, for loops, piped helpers
(echo a | tr a A), urllib.request.urlopen("http://example.com"),
socket.getaddrinfo("example.com",80), open("/etc/os-release"),
open("/tmp/...","w") all still succeed. curl, wget, nc, ssh, rm,
socket.create_connection(("169.254.169.254",80)),
open("/etc/passwd"), open("/proc/self/environ") all correctly
blocked.
* studio: rate-limit login, rotate refresh tokens, add logout, security headers, gate bootstrap injection
A pass over the auth surface found a cluster of related issues that this
commit closes together.
Login (routes/auth.py):
- Add an in-memory per-IP login rate limiter. Five failed POSTs to
/api/auth/login inside a 60s window produce 429 with Retry-After.
A successful login clears the bucket. Previously 30 wrong passwords
in under one second was accepted as 30x 401, which combined with
the (now fixed) admin-username leak from /api/auth/status made
brute-force trivial against a small password.
Logout (routes/auth.py):
- New POST /api/auth/logout returns 204 and calls
storage.revoke_user_refresh_tokens(subject) so the refresh token
is no longer valid. Previously POST /api/auth/logout returned 405
and there was no way to invalidate refresh tokens short of
changing the password. Frontend session.ts already calls
clearAuthTokens() to drop localStorage; the new endpoint lets the
client also tell the server to revoke server-side state.
Refresh-token rotation (routes/auth.py + auth/storage.py):
- New storage.consume_refresh_token(token) atomically validates +
deletes a refresh token, returning (username, is_desktop). The
/api/auth/refresh handler now mints both a new access AND a new
refresh token; the supplied token becomes invalid. Replaying a
consumed refresh returns 401 "Invalid or expired refresh token".
The previous refresh_access_token helper is left in place for
callers that intentionally want the non-rotating shape; nothing
in the route layer uses it now.
/api/auth/status no longer leaks default_username (models/auth.py +
routes/auth.py):
- AuthStatusResponse.default_username becomes Optional[str] with a
None default; the handler always returns None. The frontend already
hardcodes HIDDEN_LOGIN_USERNAME = "unsloth" (auth-form.tsx:82), so
no UI change is required.
window.__UNSLOTH_BOOTSTRAP__ no longer auto-injects (main.py):
- _inject_bootstrap is now opt-in via the
UNSLOTH_STUDIO_INJECT_BOOTSTRAP env var. The previous default
(inject whenever requires_password_change is true) embedded the
plaintext bootstrap password into the first-boot HTML for any
caller that hit /, /change-password, or any unknown SPA path.
Browser extensions and any XSS payload on the page could read it
trivially. With the new gate the bootstrap password lives only in
the auth/.bootstrap_password file (mode 0o600) where it has always
been; users typing it into a current-password field is the right
UX. routes/auth.py:change_password also clears
app.state.bootstrap_password defensively.
Security headers + server fingerprint (main.py + run.py):
- New SecurityHeadersMiddleware adds Content-Security-Policy,
X-Frame-Options: DENY, X-Content-Type-Options: nosniff,
Referrer-Policy: no-referrer,
Permissions-Policy: camera=(), microphone=(), geolocation=(),
interest-cohort=(), and stamps server: unsloth-studio so the
generic uvicorn banner no longer fingerprints the stack. The
uvicorn.Config gains server_header=False so it stops emitting its
own Server header.
/api/health minimisation (main.py):
- Unauthenticated GET /api/health returns just
{"status":"healthy","timestamp":...} so load-balancer liveness
probes keep working without leaking version, device_type,
chat_only, desktop_protocol_version, or studio_root_id to
arbitrary callers. A request that presents a valid Bearer token
still gets the full diagnostic payload so internal launchers and
sibling-Studio detection (which compares studio_root_id) keep
working.
Verification:
- 30 wrong-password POSTs to /api/auth/login -> first 5 = 401, 6th
through 30th = 429.
- POST /api/auth/logout with a fresh token -> 204. The matching
refresh token then fails 401.
- Login -> R1; /api/auth/refresh with R1 -> new access + R2 (R2 !=
R1); /api/auth/refresh with R1 again -> 401; /api/auth/refresh
with R2 -> still succeeds once and rotates again.
- curl /api/auth/status -> default_username: null.
- curl http://127.0.0.1/ does not contain __UNSLOTH_BOOTSTRAP__.
- curl -I / shows CSP, X-Frame-Options: DENY,
X-Content-Type-Options: nosniff, Referrer-Policy: no-referrer,
Permissions-Policy, and server: unsloth-studio.
- curl /api/health unauthenticated -> {status, timestamp} only.
curl with Authorization: Bearer <valid> -> full payload.
- Existing /api/system, /api/models/list, /api/train/status,
/api/inference/status, /api/auth/api-keys, login flow, SPA root
all still return 200 after the changes (regression smoke).
* studio: add SecurityHeadersMiddleware, MaxBodyMiddleware, /recipes redirect, gate _inject_bootstrap, minimise /api/health
This commit lands the main.py-side changes that share a single
middleware-registration spot. They are kept together because every
change here is either (a) a top-level middleware definition that has
to be added next to LoggingMiddleware, or (b) a route handler at the
same file-level.
SecurityHeadersMiddleware (Content-Security-Policy, X-Frame-Options:
DENY, X-Content-Type-Options: nosniff, Referrer-Policy: no-referrer,
Permissions-Policy, server: unsloth-studio). The previous responses
emitted no CSP, no XFO, no Referrer-Policy and were stamped
server: uvicorn.
MaxBodyMiddleware rejects POST/PUT/PATCH on the inference / dataset /
data-recipe / train / export prefixes when Content-Length exceeds
UNSLOTH_STUDIO_MAX_BODY_MB (default 100). The audit hit this by
attaching a 50 MB plain-text file to a chat message and watching
Studio base64-encode it into the JSON body; uvicorn has no enforced
cap so the only previous guard was the per-file 50 MB ceiling that
data-recipe upload routes already enforce. The new middleware extends
that ceiling to the OpenAI-compat path that the Chat attachments
flow through. Verified: a 200 MB JSON POST to /v1/chat/completions
returns HTTP 413 "Request body too large (209,715,264 bytes; max
104,857,600)". A small valid request continues to reach the handler.
_inject_bootstrap is gated behind UNSLOTH_STUDIO_INJECT_BOOTSTRAP.
The previous default was to inline window.__UNSLOTH_BOOTSTRAP__ =
{username, password} into the first-boot HTML whenever
requires_password_change was true, which exposed the plaintext
bootstrap password to any browser extension, page script, or LAN
caller on -H 0.0.0.0. The bootstrap password remains in the on-disk
.bootstrap_password file (mode 0o600) where it has always lived;
users typing it into a current-password field is the right UX.
/api/health unauthenticated returns {"status":"healthy","timestamp":
...} only; the previous payload (version, device_type, chat_only,
desktop_protocol_version, supports_desktop_auth, studio_root_id,
native_path_leases_supported) is preserved for callers that present
a valid Bearer token, so internal launchers and sibling-Studio
detection (which compares studio_root_id) keep working.
/recipes -> /data-recipes 308 redirect. The Data Recipes page lives
at /data-recipes; users typing /recipes hit the SPA catch-all and
saw "Not Found". The redirect also preserves any tail path, so
/recipes/<rest> -> /data-recipes/<rest>.
Verified end to end with curl: CSP / XFO / X-Content-Type-Options /
Referrer-Policy / Permissions-Policy all present on /, server header
is now unsloth-studio (uvicorn's own banner is suppressed via
server_header=False in run.py from the auth-batch commit). Followed
the /recipes redirect lands on the SPA HTML.
* studio: bound TrainingStartRequest hyperparameters at the schema level
POST /api/train/start accepted any value for learning_rate, batch_size,
max_steps, max_seq_length, warmup_steps, warmup_ratio, num_epochs,
save_steps, weight_decay, gradient_accumulation_steps, lora_r,
lora_alpha and lora_dropout, including -1, 0, 1e9, and non-numeric
strings like 'abc' or 'two' (which silently coerce to 0 in the
trainer). Probing showed the API returning 200 to learning_rate=-1
and batch_size=0; only max_steps had any partial clamping.
This commit adds field_validator on every numeric hyperparameter.
Bounds are chosen wide enough to span realistic single-host
configurations (B200 with 180 GB of memory comfortably fits the
upper end) while rejecting the values that always produce broken
training:
- learning_rate: parses str/float, requires 0 < lr < 1.0. Non-numeric
input raises with "learning_rate must be parseable as float (got
'abc')" instead of silently coercing to 0.
- batch_size: [1, 1024].
- gradient_accumulation_steps: [1, 4096].
- num_epochs: [1, 1000].
- max_steps: [1, 1_000_000].
- max_seq_length: [1, 131072].
- warmup_steps: [0, max_steps].
- warmup_ratio: [0.0, 1.0].
- save_steps: [0, 1_000_000].
- weight_decay: [0, 10] (typical 0..0.1).
- lora_r: [1, 512].
- lora_alpha: [1, 1024].
- lora_dropout: [0.0, 1.0).
Each validator names the offending field in its ValueError message
so the 422 response body identifies which input is bad. The
learning_rate validator returns its result as str (the schema field
type is str("2e-4") for backwards compatibility) so existing call
sites that float() the value continue to work.
Verified:
- learning_rate=-1 -> 422 "learning_rate must be > 0 (got -1.0);
typical range is 1e-6 .. 1e-3".
- learning_rate='abc' -> 422 "must be parseable as float".
- batch_size=-1 / 0 / 999999 -> 422 "batch_size must be in [1, 1024]".
- batch_size='two' -> 422 (pydantic int parser).
- max_steps=0 / -5 -> 422 "must be a positive int".
- max_seq_length=200000 -> 422 "must be in [1, 131072]".
- warmup_ratio=2.5 -> 422 "must be in [0.0, 1.0]".
- lora_dropout=1.5 -> 422 "must be in [0.0, 1.0)".
- Valid request with learning_rate='2e-4', batch_size=1, max_steps=5
passes validation and the training run starts as normal.
* studio: redact image-decode errors, clean checkpoint dirs on cancel, tolerate Stop-button + tool-result message shapes
Three small fixes that fall under "do not let the audit findings
become user-visible papercuts".
routes/inference.py - image-decode error redaction (the audit hit
this with a 0-byte / malformed / wrong-extension image upload). The
three image-normalise sites previously raised HTTPException(400,
detail=f"Failed to process image: {e}"). When PIL raised
UnidentifiedImageError(io.BytesIO(raw)) the message string included
"<_io.BytesIO object at 0x7e40a5d7bf60>", leaking both the Python
class name (confirming the PIL/io stack) and a heap address (mildly
useful for ASLR-bypass chaining if another memory-corruption bug is
ever found). Each site now catches UnidentifiedImageError and
returns the generic "Unsupported or corrupt image format"; the
fall-through generic except returns "Failed to process image". No
exception-repr is interpolated into a response body anywhere along
these paths.
core/training/training.py - checkpoint cleanup on cancel. When a
user clicks Cancel Training, the trainer flips _cancel_requested=True
and the supervisor force-terminates the subprocess. The trainer
writes checkpoint-<step> directories under output_dir every
save_steps; previously these survived the cancel and accumulated on
disk (the audit recorded ~67 MB stuck after a 200-step cancel with
save_steps=20). New helper _cleanup_cancelled_checkpoints(output_dir)
globs checkpoint-<int> entries and removes them. It is gated by a
realpath containment check against outputs_root() so it cannot
accidentally rmtree anything outside the configured outputs root.
force_terminate() invokes the helper after the subprocess join when
_cancel_requested is true. Stop-and-Save runs are unaffected because
that path keeps _cancel_requested=False.
models/inference.py - chat message shape tolerance. Two related
frontend interactions used to crash the request validator:
- After the Stop button truncates a generation, the frontend
retained {role:"assistant", content:""} in the conversation
history and replayed it on the next send. ChatMessage previously
required role="assistant" to have non-empty content or tool_calls,
so the next message returned 422 and the thread was permanently
broken. The validator now normalises empty assistant content to
None so the request round-trips and the trailing empty turn can
be ignored downstream.
- The frontend's second-round tool POST drops the streamed
tool_call_id, hitting the strict-spec check "role=tool requires
tool_call_id". The validator now synthesises an opaque id
(call_<8 hex>) when missing, so the request reaches the handler
and the model's final summarising response gets generated. The
proper fix lives in the frontend (carry the streamed id through
the second POST) and will follow.
Verified end to end with curl: HTTP 400 (model not loaded) on both
the empty-assistant history shape and the tool-result-without-id
shape, instead of HTTP 422 from the schema validator.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten code comments from security-hardening pass
Trim verbose docstrings and inline finding references added in the
previous commits in this branch. Functionality unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: await get_current_subject in /api/health and make refresh-token consumption atomic
The /api/health auth probe called get_current_subject(creds) without
awaiting it. The coroutine object is truthy, so any caller presenting a
Bearer header (valid or not) received the full diagnostic payload
including version, device_type, studio_root_id, etc. Await the coroutine
and treat HTTPException as 'fall back to the minimal liveness payload'.
consume_refresh_token did SELECT then DELETE WHERE id under default
autocommit isolation. Two concurrent POST /api/auth/refresh requests
could both win the SELECT before either DELETE ran, defeating
single-use refresh-token rotation. Replace with a single
DELETE ... WHERE token_hash = ? AND expires_at >= ? RETURNING ...
statement so the validate-and-delete lands as one atomic op under
SQLite's write lock (3.45.1 supports RETURNING; min was 3.35).
* studio: enforce body cap on chunked uploads and drop unsafe-inline from script-src
MaxBodyMiddleware previously only inspected the declared Content-Length
header; clients omitting it or sending Transfer-Encoding: chunked
bypassed the cap and could still drive an OOM via the downstream
JSON / file readers on /v1/chat/completions, /api/inference, /api/data-recipe,
/api/datasets, /api/train, /api/export. Rewrite as a raw ASGI middleware
that drains and counts http.request frames, replies 413 once the running
total exceeds UNSLOTH_STUDIO_MAX_BODY_MB before invoking the FastAPI
handler, and replays the buffered body to downstream so route code that
calls request.json() / await request.body() works unchanged.
CSP previously included 'unsafe-inline' on script-src, which defeats the
main XSS protection. The frontend bundle does not need inline scripts;
the only inline <script> the backend ever emits is _inject_bootstrap,
which is opt-in via UNSLOTH_STUDIO_INJECT_BOOTSTRAP. Drop 'unsafe-inline'
from script-src by default; when _inject_bootstrap fires, generate a
per-response nonce, embed it on the inlined <script>, and have
SecurityHeadersMiddleware splice 'nonce-XXX' into the CSP for that one
response (the internal x-internal-script-nonce header is popped before
the response leaves the server). 'unsafe-inline' stays on style-src for
Vite-injected styles.
* studio: drop empty assistant sentinel before passthrough
ChatMessage._validate_role_shape normalises role="assistant", content=""
(the post-Stop sentinel emitted by the frontend) to content=None so the
in-process path can drop it via _extract_content_parts. The passthrough
path then ran m.model_dump(exclude_none=True), which strips the now-None
content key entirely, sending {"role":"assistant"} to llama-server / the
OpenAI-compat backend. That fails upstream and leaves the user without a
recoverable Stop->resume.
Add _drop_empty_assistant_sentinels and call it at both passthrough
message origins: _openai_messages_for_passthrough (covers
/v1/chat/completions and the Responses API which routes through it) and
the anthropic_messages_to_openai output before
_anthropic_passthrough_*. Assistant messages that carry only tool_calls
(no content) are preserved.
* studio/tests: cover audit-fix surfaces and rebase pre-existing tests
Adds and updates pytest coverage for the four bot-flagged audit fixes
landed earlier in this branch and rebases two pre-existing tests that
were broken by the relaxed-validator and /api/health auth-gate changes.
studio/backend/tests/test_middleware.py (new)
MaxBodyMiddleware: small protected, large declared, unprotected
passthrough, chunked-upload-over-cap rejection (the regression for
the original Content-Length-only gap), and chunked-under-cap replay.
SecurityHeadersMiddleware: script-src no longer carries
'unsafe-inline', style-src still does, default headers
(XFO/XCTO/Referrer-Policy/Permissions-Policy/server), and the
internal x-internal-script-nonce header is consumed by the
middleware and converted to 'nonce-XXX' in the CSP.
/api/health: no auth -> minimal, invalid Bearer -> minimal
(the await regression), valid Bearer -> full diagnostic payload.
studio/backend/tests/test_desktop_auth.py
consume_refresh_token: second-call returns None, expired returns
None, and a 64-thread concurrent pile-up against the same hash
produces exactly one successful consumer (regression for the
SELECT-then-DELETE race).
test_health_response_reports_desktop_capability_fields: rebase
against the new health_check(request) signature by going through
TestClient with a real bearer instead of asyncio.run-ing the
handler directly.
studio/backend/tests/test_openai_tool_passthrough.py
Pin the new ChatMessage tolerance: assistant without content or
tool_calls is tolerated (normalises content -> None), empty-string
and empty-list assistant content normalise to None, and a missing
/ empty tool_call_id on role='tool' is synthesised as call_<hex>
rather than raising. Tests for _drop_empty_assistant_sentinels
cover the three drop shapes (empty string, empty list, missing
content key), preservation of assistant text and tool_calls-only
messages, and end-to-end through
_openai_messages_for_passthrough.
studio/backend/main.py
SecurityHeadersMiddleware.dispatch used response.headers.pop(...)
for the nonce-header handoff; Starlette's MutableHeaders has no
pop. Read-then-del so the internal handoff header is still
stripped before the response leaves the server.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio/tests: rebase three more pre-existing CI tests against this branch
CI on PR #5375 was red on three tests that were tuned for behaviour
predating this branch. Updates each so the assertions match what the
audit fixes intentionally changed; no production code touched.
studio/backend/tests/test_trained_model_scan.py
test_scan_trained_models_includes_lora_and_full_finetune_outputs
passed an absolute tmp_path through scan_trained_models, which now
runs resolve_output_dir / _assert_contained against outputs_root().
Repoint outputs_root() at tmp_path via monkeypatch so the fixture
dirs land under the configured root and the realpath containment
check passes.
tests/test_studio_install_workspace_guard.py
test_health_endpoint_exposes_studio_root_id_not_raw_path read
the first 1500 bytes after @app.get("/api/health") and asserted on
the studio_root_id literal. The handler grew (unauth short-circuit
+ await dependency gate) and the literal slid past the byte window.
Replace the fixed window with a slice up to the next top-level
@app.* decorator so the test surveys the whole handler regardless
of size.
tests/studio/studio_api_smoke.py
The "login burst (5x wrong pw) -> 401 each" assertion was tagged
"When/if we add one, this assertion updates in the same PR." We
added the per-IP rate-limit in routes/auth.py
(_LOGIN_MAX_FAILS=5/60s) but missed the assertion update. Rewrite
the burst probe to observe the new invariant: at least one 401,
eventual transition to 429, and Retry-After present on the 429.
Adds a small _login_with_headers helper since the existing login()
helper drops response headers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci(studio-ui): set UNSLOTH_STUDIO_INJECT_BOOTSTRAP=1 for Playwright Studios
The Chat UI Playwright test drives the first-boot change-password
form, which (per playwright_chat_ui.py step "1. Change-password
through the UI") pre-seeds the hidden current_password field from
window.__UNSLOTH_BOOTSTRAP__. That global is only emitted when the
backend's _inject_bootstrap path fires, which since the security
pass on this branch is gated behind UNSLOTH_STUDIO_INJECT_BOOTSTRAP
and defaults to off. Without the global, the React form's
current_password validator never satisfies, the submit button stays
disabled, and the composer.wait_for() probe times out on
/change-password.
Re-enable injection only for the CI Studios that drive the chat UI
across linux/mac/windows. Production deployments are unaffected: the
env var has to be explicitly opted into, and the on-disk
auth/.bootstrap_password remains the source of truth for human users
typing the password in by hand.
Covers all eight Studio launch sites: the primary chat-ui boot and
the "extra UI tests" boot for each of the three OSes, plus the
pipeTransport JSON-crash retry relaunches in the macOS workflow that
re-spawn Studio mid-job.
A follow-up frontend PR will add a visible current_password input so
the form satisfies its own validator without needing the bootstrap
auto-fill at all; once that lands this CI knob can come back out.
* studio/sandbox: drop unshare(CLONE_NEWNET); add trusted-host allowlist; block sandbox file uploads; raise CPU rlimit default to 600 s
CLONE_NEWNET inside _sandbox_preexec silently killed every outbound
HTTP request from sandboxed Python whenever the kernel allowed
unprivileged user namespaces. requests.get('https://huggingface.co'),
urllib.request.urlopen('https://en.wikipedia.org/wiki/...'),
socket.connect(('arxiv.org', 443)) all failed despite the AST visitor
intending to allow them. The bash blocklist (curl / wget / nc / ssh /
scp / sftp / rsync / socat / eval / source) plus the AST-level
metadata-host denylist still carry the network policy after this
change; CLONE_NEWNET was redundant with both.
Add _TRUSTED_PUBLIC_HOST_LITERALS + _TRUSTED_PUBLIC_HOST_SUFFIXES
(~100 informational hosts: Wikipedia language subdomains, Wikimedia,
Wikidata, Google search, Bing, DuckDuckGo, HuggingFace, GitHub,
raw.githubusercontent.com, arXiv, StackOverflow / Stack Exchange,
MDN, docs.python.org, PyTorch / TensorFlow / NumPy / pandas docs,
pypi / files.pythonhosted.org / npmjs / crates.io, ReadTheDocs,
arXiv, Britannica, BBC / Reuters / Nature / Science, NASA / CDC /
NIH / WHO open data, api.weather.gov). The visitor now blocks
literal hosts that are neither metadata nor trusted with a short
LLM-readable string so the model can retry with an allowed source
instead of choking on a multi-line error.
Block upload-shape calls regardless of host: requests.post / put /
patch / delete / request with files= or data=open(...) /
data=bytes_literal; httpx equivalents; urllib.request.urlopen /
Request with data=...; HuggingFace upload_file / upload_folder /
upload_large_folder / create_commit (module-level FQ paths AND
method-name match on any receiver). Message: "Blocked: file upload
disallowed in sandbox".
Bump UNSLOTH_STUDIO_SANDBOX_CPU_S default 300 -> 600 s so long
agentic chains that span multiple tool calls don't get SIGXCPU'd
mid-stride. Env-var override path is unchanged.
Host normalisation now strips trailing dot, userinfo @, and explicit
port before allowlist / denylist comparison so trailing-DNS-dot,
userinfo-smuggling, and explicit-:443 URLs are decided correctly.
* studio: raise default request-body cap from 100 MB to 500 MB
UNSLOTH_STUDIO_MAX_BODY_MB default goes 100 -> 500 to comfortably
cover vision + audio + multi-recipe-batch JSON payloads. The
MaxBodyMiddleware stream-counting logic from this branch's earlier
06ec088 already handles chunked bodies up to the new cap; env-var
override path is unchanged for callers that want a tighter limit.
* studio/auth: restore /api/auth/status.default_username to 'unsloth'
This branch's earlier b39e9a4 changed default_username to None on the
public /api/auth/status endpoint so the username field didn't leak to
unauthenticated callers. In practice this regressed third-party
clients (and the in-tree React login form's pre-fill UX) without
adding meaningful security: the bootstrap password is the actual
secret, and the username 'unsloth' is the documented default.
Pin default_username to storage.DEFAULT_ADMIN_USERNAME ('unsloth')
and tighten the response model so the field is required rather than
Optional. Anyone who needs anonymisation can still reach for an
allow-list deployment with auth disabled.
* studio/training: raise max_seq_length / batch_size / lora_r / lora_alpha caps
This branch's 7102815 introduced field validators with conservative
caps. The follow-up loosens them so long-context experiments and
high-rank LoRA exploration aren't gated at the schema layer:
_MAX_BATCH_SIZE 1024 -> 4096
_MAX_SEQ_LENGTH 131_072 -> 2_000_000 (2M tokens)
lora_r cap 512 -> 16_384 (_MAX_LORA_R)
lora_alpha cap 1024 -> 32_768 (_MAX_LORA_ALPHA)
_MAX_GRAD_ACCUM / _MAX_STEPS / _MAX_EPOCHS / lora_dropout /
warmup_ratio / weight_decay are unchanged. Hardware (VRAM, host
RAM, kernel launch latency) is now the binding constraint at the
new caps, which is the correct ordering -- the validator stays a
sanity check on -1 / 0 / 'abc' style garbage, not a usability gate.
* studio/tests: cover sandbox allowlist + upload block + raised training caps
studio/backend/tests/test_sandbox_tools.py (new):
TestMetadataHostDenylist -- short "Blocked: cloud-metadata host"
message on AWS IMDS, GCP metadata,
Alibaba ECS, AWS IPv6 IMDS, 169.254/16.
TestTrustedHostAllowlist -- Wikipedia (any language subdomain),
Google, DuckDuckGo, HF, raw GitHub,
arXiv, StackOverflow / family,
MDN, docs.python.org, pypi, BBC,
api.weather.gov, NumPy / PyTorch docs.
TestUntrustedHostBlock -- example.com / random unlisted host
rejected with the short "Blocked: host
not in sandbox allowlist; use an
allowed informational source" message.
Dynamic URLs (computed var) still pass
-- documented limit of static analysis.
TestHostNormalization -- trailing dot, explicit :443, uppercase,
userinfo-@-smuggle all decided
correctly without false-block /
false-pass.
TestUploadDenylist -- requests / httpx / urllib.urlopen with
files= / data=open / data=bytes,
HfApi().upload_file / upload_folder /
create_commit, module-level
huggingface_hub.upload_folder. POST
json= to trusted host still passes.
TestSandboxCpuRlimitDefault -- pin UNSLOTH_STUDIO_SANDBOX_CPU_S=600
default and confirm CLONE_NEWNET
source line is gone.
TestMaxBodyDefault -- pin UNSLOTH_STUDIO_MAX_BODY_MB=500
default.
studio/backend/tests/test_studio_train_validation.py (new):
Pin at-cap-accepts / over-cap-rejects boundaries for
max_seq_length=2_000_000, batch_size=4_096, lora_r=16_384,
lora_alpha=32_768 so a future regression that tightens them back
without explicit user opt-in is caught.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten code comments across the security-hardening pass
* studio: always inject bootstrap credentials on first boot
The UNSLOTH_STUDIO_INJECT_BOOTSTRAP gate added an extra
terminal-to-browser copy-paste on every fresh install. In practice
the LAN credential leak it guarded against is narrow: the password
is one-time, the user rotates it on the very next click, the
default Studio bind is 127.0.0.1, and -H 0.0.0.0 already exposes
the entire API surface. Drop the gate so the inject fires whenever
a bootstrap password is still pending. The CSP nonce wiring stays
in place; the inline script remains the only inline script the
backend ever emits.
The three Playwright UI smoke workflows lose their
UNSLOTH_STUDIO_INJECT_BOOTSTRAP=1 lines along with the explanatory
comment blocks since the inject now happens by default.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* Dark theme refactor, right sidebar redesign, and chat UI polish
- Dark theme refactor
- Redesign right sidebar
- Further left sidebar adjustments
- Wider chat and content area; layout tweaks for chat content
- Rounded corners across elements for consistency
- Show chat message menu icons on menu-area hover, not only on message hover
- Assistant message menu icons now always visible; user messages keep on-hover
- Redesigned copy icon used consistently across chat blocks and messages
- Redesigned trash icon, applied consistently
- Unified icon sizing and style with the sidebar
- Adjusted icon colors across chat
- Fix on-hover background design for chat icons
- Fix tooltip from 'more' button staying visible after clicking elsewhere
- Adjust position and design of generation speed info text below messages
- Adjust design of token speed info popup
- Adjust sidebar scrollbar to cover recent chats only
* Recents sidebar rename, UI/theme refactor, layout and chat polish
UI & Theme:
- Dark theme refactor
- Consistent rounded corners across elements
- CSS polish and cleanup
- Remove unused logo image assets
Recents sidebar:
- Add 'more' button for options menu
- Support renaming conversations and training runs
- Confirmation dialog before deleting chats
- Add optional display_name column to training_runs (idempotent ALTER TABLE) so renaming doesn't lose model_name/dataset_name from the run config
- New PATCH /api/train/runs/{run_id} endpoint accepts { display_name: string | null }; empty/whitespace clears the override
- Sidebar shows display_name ?? model_name and exposes Rename in the row's More menu, mirroring the chat rename flow
- Cache last list response in localStorage and hydrate from it on mount, so recents paint instantly on F5 / route revisit; cached items are shape-validated and dropped if malformed
- Optimistic updates on rename and delete (apply locally + cache before background refresh)
- Visible toast on rename/delete failure instead of swallowed errors
Layout:
- Redesigned right sidebar
- Further left sidebar adjustments
- Updated chat content layout; chat and content area slightly widened
- Sidebar scrollbar covers recent chats only
Icons:
- Redesigned copy icon, unified across chat blocks and messages
- Redesigned trash icon to match
- Consistent icon sizing and style across chat and sidebar
- Adjusted icon colors across chat
- Fix icon on-hover background design
Chat messages:
- Menu icons now appear on hover over the menu area, not just the message
- Assistant message menu icons always visible; user messages keep on-hover (next/previous response stays visible for edited prompts)
- Repositioned and restyled generation speed info text below messages
- Restyled token generation speed popup
Tooltips:
- Removed tooltip on hover for previous/next assistant response icons
- Unified tooltip design across sidebars and chat
- Removed tooltip animations (also fixes related lag)
Model & Chat Template config:
- Merged Chat Template config into Model Configuration section
- Added revert-to-original for chat template
- Fix Chat Template config disappearing on page refresh until model reload
Performance & scroll:
- Removed chatbox movement animations across pages/navigation (fixes related UI lag)
- Fix scroll flicker at end of streaming when a code block is the final element
- Additional chat scroll improvements
Bug fixes:
- Fix 'more' button tooltip remaining visible after clicking elsewhere
* Remove sidebar localStorage cache and optimistic updates
Drops the localStorage hydration and optimistic rename/delete logic from the recents sidebar; reverts to fetching fresh on mount.
* Fix missing cn import in shared-composer (regression from merge)
* chore(sidebar): import sidebar deps from feature indexes
Re-export deleteChatItem / renameChatItem / useChatSidebarItems / SidebarItem / useChatSearchStore / ChatSearchDialog from @/features/chat, and removeTrainingUnloadGuard from @/features/training. Switch app-sidebar.tsx to consume them via the public feature indexes instead of deep paths, clearing the no-restricted-imports eslint errors. No behavior or UX change.
* fix(studio/frontend): reload training Recents sidebar after F5 refresh
The Recents sidebar showed empty after a hard refresh. The hook's inFlightRef dedup guard collided with React StrictMode's double-mount in dev: the second mount's fetch returned silently with no error, no retry, and no toast — leaving the sidebar empty until navigation.
Replace skip-if-busy dedup with abort-previous via a hook-level AbortController. This also fixes a latent race where a slow poll could resurrect a just-deleted row by clobbering the optimistic update.
Changes (all in use-training-history-sidebar.ts):
- fetchRuns aborts any in-flight request before starting a new one; post-await signal.aborted check drops stale responses.
- Optimistic helpers (applyRunUpdate, removeRun) abort in-flight fetches so they don't depend on caller discipline to invalidate stale data.
- Initial load gets bounded retry-with-backoff (500ms / 1.5s / 3.5s) and surfaces a sonner toast with a Retry action on final failure.
- Failure toast auto-dismisses on any successful load (initial retry, Retry click, or polling recovery).
- Polling pauses while the tab is hidden and catches up on visible, avoiding wasted requests during long training runs.
- Both effects own their teardown explicitly (abort + clear timer).
* Apply unified tooltip design and behavior across remaining pages for consistency
* UI polish: spacing, tooltip on source icons, letter spacing, smaller icons, consistent edit icon
- Adjust tiny spacing between elements around the UI for subtle polish
- Redesign tooltip on source icons for web search / tool use, consistent with the new design
- Adjust chat text letter spacing
- Smaller icon sizes
- Replace 'edit message' icon in chat with the new Rename icon used in Recents for consistency
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Adjust CSS for right sidebar
* Fix scrollbar UI compatibility across browsers
* fix: preserve chat preset settings on model load
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): remove duplicate chat template status field
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* chore: remove creative preset assumption
* fix(studio): align speculative decoding default
* fix(studio/chat): snap numeric param inputs to step grid
- Type a value in any param input (Temperature, Top K, Max Tokens, etc.)
now clamps to [min, max] and snaps to the slider's step grid, killing
off-grid values like 1.051234 and FP residue from slider drags.
- Branch picker chevrons share the action bar's 32px height + 10px radius
via a new .aui-branch-chevron-btn utility; hover area aligns visually
while staying narrower than the sibling icon buttons.
* fix(studio/chat): keep training-run polls converging and drop dead preset code
- Keep training-run polls converging when responses outrun the 5s interval
(don't unconditionally abort prior in-flight; skip if one is still pending,
mutation race still guarded).
- Drop dead Creative/Precise preset code paths (remove 'builtin-fixed' source
variant + unreachable branches).
* fix(studio): training-run cards show custom name + model + dataset
- Training-run cards now display custom display_name + model + dataset,
with cross-view sync on rename/delete.
- Enhance clarity of borders and colors in dark theme on export etc.
* fix(studio): match active state green to unsloth brand color
* fix(studio): preserve can_resume on training rename
* fix(studio): keep GGUF chat template override distinct
* fix(studio): treat audio input models as multimodal
* fix(studio): cancel numeric draft on Escape
* fix(studio): use default speculative mode on toggle
* fix(studio): detect GGUF audio VLM input models
* fix(studio): address final PR review findings
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): refresh sidebar/history when a new training run starts so it appears without a manual reload
* fix: API and svg
* fix(studio/sidebar): align run rename dirty check with displayed baseline
* fix(studio/sidebar): use leading-tight on account block to prevent descender clipping with truncate
---------
Co-authored-by: sneakr <hauzin@hotmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: shine1i <wasimysdev@gmail.com>
* feat(studio): add Continued Pretraining (CPT) support
Implements CPT as a first-class training method in Unsloth Studio,
resolving feature request #4565.
Changes:
- frontend/src/types/training.ts: add 'cpt' to TrainingMethod union
- frontend/src/lib/vram.ts: add 'cpt' to VramTrainingMethod (fp16 footprint)
- frontend/src/features/export/constants.ts: add CPT to METHOD_LABELS
- frontend/src/features/training/api/mappers.ts: map 'cpt' -> 'Continued Pretraining',
force packing=true and train_on_completions=false for CPT payloads
- frontend/src/features/studio/sections/model-section.tsx: add 'Continued Pretraining'
option (purple dot) to Method selector; update tooltip
- frontend/src/features/onboarding/.../model-selection-step.tsx: add CPT to
onboarding wizard method dropdown
- backend/models/training.py: update training_type field description
- backend/core/training/worker.py: detect is_cpt flag, force packing=True,
train_on_completions=False, pass is_cpt to _train_worker
- backend/core/training/trainer.py: _train_worker reads is_cpt kwarg, forces
packing on, skips train_on_responses_only for raw-text pretraining
CPT behaviour:
- Full model weights (no LoRA adapters), same as Full Finetuning
- Sequence packing always enabled for GPU efficiency
- Trains on every token (no chat-format masking)
- VRAM estimated at fp16 (2.0 bytes/param)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update mappers.ts
* Add CPT raw dataset support and UI fixes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add missing training methods module
* Handle invalid raw-text rows and expose raw in onboarding
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Etherll <61019402+Etherll@users.noreply.github.com>
Co-authored-by: Etherll <mrmrmidessam@gmail.com>
ashzak
