Round 13 follow-up: on Windows Path('/etc/passwd').is_absolute()
returns False because POSIX absolute paths read as drive-relative,
which let the traversal check fall through to resolve(strict=True)
and crash with a raw FileNotFoundError instead of the friendlier
RuntimeError. Add a PurePosixPath check + explicit leading-separator
guard and wrap the resolve() in try/except so a missing path inside
the chosen repo is reported as 'Local repo path does not contain ...'
on every OS.
Pre-existing 59 diffusion backend + route tests still pass; staging
Windows Diffusion CI was failing on this exact case.
Round 13 reviewer aggregate (logs/review_round13_aggregate.md):
P1 fixes:
- routes/export.py load_checkpoint refuses (409) when an export job
is currently active, mirroring the chat/diffusion/training handoff
guards. ``is_export_active`` absence is tolerated for older / mocked
backends.
- core/inference/diffusion.py local-path GGUF loader now accepts
relative directories (Studio exports surface as ``exports/my-flux``)
and confines ``gguf_filename`` to the chosen repo via
``_resolve_local_gguf_child``: absolute filenames, ``..`` segments,
and Windows separators are rejected before any file is opened.
- core/inference/diffusion.py status() exposes ``active_gguf_filename``
alongside the pending variant so delete guards can pair each owned
repo with the GGUF variant it actually owns.
- routes/models.py cache delete + finetuned delete adopt a shared
``_diffusion_owned_targets`` + ``_variant_delete_is_safe_for_owned_gguf``
helper. Per-variant deletes during a swap-in-flight cannot remove
the active variant while the pending variant is loading.
- core/inference/llama_cpp.py publishes ``loading_model_identifier``
before ``_download_gguf`` starts and clears it in ``finally``. Cache
delete (routes/models.py) and the cross-workload release helpers
(routes/inference.py::_release_llama_for and
diffusion.py::_release_chat_backend_for_diffusion) consult it so a
multi-GB HF download cannot be rmtree'd or be ignored by /images/load
while still in flight.
P2 fixes:
- core/inference/diffusion.py adds
``generate_image_with_metadata`` + ``async_generate_with_metadata``;
/images/generate uses it so the response model/family reflect the
pipeline that actually produced the image even if an unload races
the route.
- core/inference/diffusion.py: ``base_repo`` only applies when picking
a GGUF quant. Filling Base diffusers repo while loading a full
diffusers repo no longer silently swaps the load target.
- core/inference/diffusion.py: failed device placement / offload now
drops pipe + transformer references explicitly before drain so
partial allocations cannot keep VRAM around.
- core/inference/diffusion.py: torch/diffusers imports surface as a
clear RuntimeError naming the missing dependency.
- core/inference/diffusion.py: _smart_base_repo splits on both POSIX
and Windows separators so ``C:\\Users\\me\\base\\FLUX.2-klein-4B-GGUF``
no longer picks the Base 4B variant via the parent dir.
Tests:
- 6 new regression cases (Windows leaf, traversal/backslash rejection,
relative-dir local load, metadata snapshot, lock serialisation).
- All 59 diffusion backend + route tests pass.
Round 12 reviewer findings.
Backend correctness (P1)
* core/inference/diffusion.py load_model: GGUF branch now
handles an absolute local directory passed as repo_id by
joining Path(repo_id) / gguf_filename directly instead of
handing the path to hf_hub_download (which raises
HFValidationError because the path is not 'namespace/repo').
Closes round 12 review #1 -- the load request advertised
'local path' support but actually only worked for Hub repo ids.
Delete guard precision (P1)
* routes/models.py /delete-finetuned + /delete-cached:
diffusion guard now consults gguf_filename from status()
and ALLOWS per-variant deletes that target a different quant
than the one the loaded pipeline is reading. Loading
'Q4_K_S' no longer blocks deleting 'Q8_0' from the same
repo / export directory (round 12 reviews #3 and #4).
Accelerator (P2)
* core/inference/diffusion.py _drain_cuda_cache: also calls
torch.mps.empty_cache() when the MPS backend is the
active accelerator. Apple Silicon swaps now actually return
held VRAM instead of leaving it pinned in the Metal
allocator (round 12 review #10).
Smart base repo (P2)
* core/inference/diffusion.py _smart_base_repo: only inspects
the LAST segment of the repo id / path for the 'base' / '9b'
tokens. A namespace like baseorg/FLUX.2-klein-4B-GGUF or
a parent directory like /home/me/.cache/base/... no
longer falsely selects the Base variant (round 12 review #9).
Round 11 reviewer findings.
Backend lifecycle (P1)
* core/inference/diffusion.py _release_other_gpu_owners_for_
diffusion: now re-checks is_export_active() locally before
calling _shutdown_subprocess. The route layer already 409s on
active exports, but defence-in-depth means direct backend
callers (tests, scripts, future routes that forget the
higher-level guard) can no longer terminate an in-flight
export and corrupt the user's partial output.
* routes/inference.py standard chat-load path: the duplicate
inline 'if exp_backend.current_checkpoint -> _shutdown_subprocess'
block was removed. _release_export_for above already handles
settled checkpoints and skips active ones; the inline block
was the round 11 #2 asymmetric fix surface.
Routing / error mapping (P2)
* routes/training.py start_training: except HTTPException:
raise was inserted before the broad except Exception:
handler so the 409 raised by _raise_if_training_active /
_raise_if_export_active reaches the client intact instead of
being swallowed into a 500.
State publishing (P2)
* core/inference/diffusion.py load_model: success path now
clears _loading + _pending_* under _lock BEFORE returning
self.status(), so the response payload reports the resident
pipeline cleanly (no stale is_loading=true / pending_*). The
finally block remains idempotent for error / early-raise paths.
* core/inference/diffusion.py status(): nulls family /
pipeline_class while a swap is in flight (pending_repo set
and != active_repo). Previously the response paired pending
model B's repo_id with model A's family, producing a
combination that never existed.
Validation
* models/inference.py: DiffusionLoadRequest.repo_id and
base_repo length caps bumped from 256 to 1024; gguf_filename
bumped from 256 to 512. The earlier caps rejected realistic
Studio export paths (deeply nested outputs / exports
directories, especially on Windows).
Dependencies
* pyproject.toml huggingfacenotorch + studio/backend/
requirements/no-torch-runtime.txt: floor gguf at >=0.10.0
to match the diffusers requirement. Unconstrained pin allowed
a resolver to install older gguf releases that raise at
single-file load time.
Round 10's training-side _raise_if_export_active call broke
existing test mocks and older ExportBackend builds that only
expose current_checkpoint -- they raised AttributeError on
exp.is_export_active() and the outer guard converted that into
a 503, causing the prior Backend CI to fail
test_inference_route_returns_400_for_invalid_gpu_ids,
test_training_route_returns_400_for_invalid_gpu_ids, and
test_training_route_forwards_embedding_learning_rate.
Both _raise_if_export_active and _release_export_for now detect
the missing method with getattr(...) and treat absence as
'no async-job tracker available' (effectively 'not active').
The 503 fail-closed path still fires when the method exists but
the call itself raises, so production backends (the
ExportOrchestrator subclass that does expose is_export_active)
keep their stronger guard.
Round 10 reviewers found the round 9 export helpers had a
destructive bug: _release_export_for treated is_export_active=True
as a shutdown condition, so any caller (training, chat, diffusion)
could terminate an in-flight export and corrupt the user's output.
Conversely _raise_if_export_active raised 409 on a settled
checkpoint, blocking idle cleanup.
Backend (P1)
* routes/inference.py: split the export-active surface in two:
_raise_if_export_active() now ONLY raises when
is_export_active() is True. A settled current_checkpoint is
treated as held GPU memory, not an active job.
_release_export_for() now ONLY shuts down when
current_checkpoint is set AND is_export_active() is False
(i.e. a previously completed checkpoint just holding memory).
An unknown / unverifiable is_export_active is treated as
'might still be active' so the helper refuses to drop.
* routes/training.py: now calls _raise_if_export_active before
_release_chat_for / _release_export_for, mirroring the chat
and diffusion paths. The previous code went straight to
_release_export_for and would kill an in-flight export.
* routes/inference.py: split _release_chat_for into
_release_llama_for and _release_safetensors_chat_for so the
GGUF chat-load path can release only the OTHER chat backend
(round 10 review #4: the previous inline 'if active_model_name'
check skipped loading_models and let an in-flight safetensors
load race the new GGUF allocation).
* routes/inference.py: _raise_if_export_active now fails CLOSED
(503) when is_export_active() raises, not only when
get_export_backend() raises. Round 10 review #7.
Dependencies (P1)
* pyproject.toml huggingfacenotorch extra: pin gguf. The
Studio Images default curated picker is GGUF-only and
diffusers.GGUFQuantizationConfig + from_single_file require
the standalone gguf package at runtime; missing it would 500
on the first /api/inference/images/load with
'gguf>=0.10.0 is required'.
Round 9 reviewer flagged a pile of handoff asymmetries: every
GPU-owning lifecycle change (training, export, chat, images) needed
its own bespoke unload sequence and they had drifted out of sync.
Some skipped llama-server is_active; some missed safetensors
loading_models; export and training did not check is_export_active.
Backend handoff (P1)
* routes/inference.py: new _release_chat_for / _release_export_for
helpers. Both treat llama-server as held when is_loaded OR
is_active, safetensors as held when active_model_name OR
loading_models is non-empty, and export as held when
current_checkpoint OR is_export_active. Both helpers run their
unloads in worker threads so async routes do not block the
event loop.
* routes/training.py: replaces its bespoke inline llama / safe /
export unload sequence with await _release_chat_for / _release_
export_for.
* routes/export.py: same swap for the chat unload chain (export
still does NOT call _release_export_for on itself).
* routes/inference.py GGUF + standard chat-load paths: now use
_release_export_for to drop a settled export, and the standard
path's llama unload now also handles is_active=True (round 9
review #8).
Backend reject-on-active export (P1 #5)
* routes/inference.py: new _raise_if_export_active. Symmetric
with _raise_if_training_active: a long-running export is
refused with HTTP 409 instead of being silently killed when
/images/load or /load arrives. Diffusion / images load and
both chat-load paths call it.
* core/inference/diffusion.py _release_other_gpu_owners_for_
diffusion: no longer tears down an in-flight export job. Only
drops a SETTLED export checkpoint (current_checkpoint
populated, is_export_active False). Round 9 review #5 -- the
previous behavior could terminate an in-flight export and
leave a partial output artifact.
Token leak via logger.exception (P1 #6)
* core/inference/diffusion.py: load-failure logging now uses
logger.error(..., exc_msg) with the already-scrubbed string
and exc_info=False. logger.exception() with the raw Exception
would expose any hf_... token that diffusers / huggingface_hub
embedded in the message or traceback locals, defeating the
earlier in-flight scrub.
Dependency pinning (P1 #11)
* pyproject.toml: huggingfacenotorch optional extra now pins
diffusers>=0.37.0. Previously the floor was only set in
studio/backend/requirements/no-torch-runtime.txt, so a normal
pip install would resolve diffusers 0.36.0 (no
Flux2KleinPipeline) and the default curated FLUX.2 klein
Images model would fail at runtime.
Cache-delete exact match (P1 #14)
* routes/models.py /delete-cached: llama.cpp and safetensors
guards now match on exact repo-id (case-insensitive) instead
of prefix. Diffusion guard already does this; the chat guards
were the remaining surface where loading org/model-v2
blocked deleting org/model.
Round 8 reviewer surfaced event-loop stalls (blocking unload from
async routes), incomplete VRAM handoff coverage (is_active /
loading_models / is_export_active not checked), token leaks via
exception messages, /v1 exposure, and several fail-open paths.
Async / event-loop
* routes/inference.py /images/unload, GGUF chat-load handoff,
safetensors chat-load handoff: blocking DiffusionBackend.unload
pushed onto asyncio.to_thread. unload takes _load_lock +
_generate_lock and can block for the full duration of an
in-flight load / generation, which was freezing the FastAPI
worker, SSE stream, and hardware poller for minutes.
* routes/export.py + routes/training.py: same to_thread wrap on
diffusion unload during checkpoint / training start.
GPU-owner handoff completeness
* core/inference/diffusion.py _release_chat_backend_for_diffusion:
llama-server now also unloaded when is_active=True (mid-download
/ startup), not only when is_loaded; flushed in-flight
safetensors loads from loading_models too.
* core/inference/diffusion.py _release_other_gpu_owners_for_
diffusion: export shutdown now also fires when
is_export_active() returns True (checkpoint not yet assigned).
Security / scrubbing
* core/inference/diffusion.py: load failure paths now scrub
hf_token from both _last_error AND the raised RuntimeError
message (the previous scrub only cleared frame locals).
Falls back to a regex strip of hf_[A-Za-z0-9]{20,} to
catch tokens that came in via huggingface_hub default caching.
* routes/inference.py: image lifecycle endpoints moved from
router to studio_router so they no longer answer under
the /v1 OpenAI-compat prefix. Studio-only side effects
(download multi-GB GGUFs, unload chat, etc.) should not be
reachable via an OpenAI-compat client.
* models/inference.py: control-char validator now also rejects
tab. Some log sinks split fields on tab; allowing it left a
log-injection surface.
Fail-closed delete guards
* routes/models.py /delete-cached: llama.cpp and safetensors
branches now fail closed with 503 when their status check
raises (matches the diffusion-side guard added earlier).
* routes/export.py: split the try/except around the training
backend so import failure falls back to 'skip' (no
core.training in this build) while a runtime failure of
get_training_backend()/is_training_active() fails closed.
* routes/models.py /delete-finetuned: diffusion guard now also
compares against relative path candidates (Path.resolve() works
on relative input). Previously a load with a relative repo_id
bypassed the guard.
CUDA cleanup ordering
* core/inference/diffusion.py: split _release() (drops local +
gc.collect) from _drain_cuda_cache() (torch.cuda.empty_cache).
Callers now drain AFTER nulling every reference so the
allocator actually reclaims the freed slabs (previously
empty_cache ran while caller still held a local, which left
the cache pinned).
Generate response (P2 #16)
* routes/inference.py: response uses status()['active_repo_id']
instead of the UI-facing repo_id, so a queued /images/load
promoting a pending model cannot mislabel the just-rendered
image with the new model's identity.
Test wiring
* tests/test_diffusion_routes.py: mount inf.studio_router on the
test app so /images/* routes are reachable now that they live
on the Studio-only router.
Round 7 reviewer surfaced a handful of swap-window races, fail-open
guards, and seed precision mismatches. This commit closes them.
Lifecycle / state (P1)
* core/inference/diffusion.py: status() now emits active_repo_id,
active_base_repo, pending_repo_id, pending_base_repo, and
pending_gguf_filename alongside the existing UI-facing fields.
During a swap (model A loaded, model B loading) the previous
coalesced 'repo_id or pending_repo_id' hid the loading target
from delete guards. Splitting the fields lets guards block
deletion of either repo currently owned by the backend.
* core/inference/diffusion.py: generate_image() now takes
_generate_lock BEFORE snapshotting _pipe / _device. Snapshotting
outside the lock let a concurrent unload/load clear or replace
the backend between the snapshot and the forward, so the freed
or swapped pipeline would still run.
Symmetric handoffs (P1)
* routes/export.py: training-active check now runs BEFORE the
chat / inference / diffusion unload helpers, so a 409 does not
leave the user's chat session torn down for nothing. Also
explicitly fails CLOSED with 503 when is_training_active()
raises.
* routes/inference.py: _raise_if_training_active now fails closed
with 503 when the training backend is importable but its status
check raises. The previous best-effort log-and-continue could
let chat / diffusion loads collide with unverifiable training.
Delete guards (P1)
* routes/models.py /delete-cached: chat guard now also blocks
when llama-server is_active (i.e. mid-download) and when the
inference backend's loading_models set contains the target.
Round 7 review #7 flagged that the PR's diffusion-side loading
guard had no chat-side parallel, so deleting a chat repo while
it was downloading could still race the cache.
* routes/models.py /delete-cached: diffusion guard iterates the
new active_* + pending_* status fields so a delete during a
swap is refused on either repo.
* routes/models.py /delete-finetuned: same active_+ pending
handling, plus the guard now also refuses deletes of a parent
directory that contains the loaded pipeline (round 7 review #6:
rm -rf /exports/flux-model/ could unlink model_index.json that
the live pipeline is reading via mmap).
Seed precision (P2)
* models/inference.py + routes/inference.py: DiffusionGenerate-
Response now carries seed_str alongside the existing numeric
seed. Seeds above Number.MAX_SAFE_INTEGER are rounded by
JSON.parse in the browser; seed_str ships full decimal
precision for display and reproduction.
* frontend/api.ts: DiffusionGenerateResponse types seed_str;
images-page.tsx prefers seed_str over seed in the figure
caption so the displayed value reproduces the image.
* frontend/api.ts: stringifyWithBigInt no longer regex-replaces
sentinel strings over the full JSON output. It pulls the seed
BigInt out, JSON-serialises the remaining payload, and splices
the seed's decimal digits into the resulting object literal at
the known position. Avoids the round 7 #10 case where a
user-supplied prompt equal to '__bigint__:123' was rewritten
into a JSON integer and rejected as a non-string prompt.
Custom HF repo (P2)
* frontend/images-page.tsx: custom panel now exposes a 'Base
diffusers repo' input that maps to DiffusionLoadRequest.
base_repo. Required when a private / mirrored GGUF needs a
non-default base (e.g. a 9B Klein transformer would otherwise
fall back to the 4B base default).
Round 6 reviewers identified several races between load / unload /
generate and several fail-open delete guards. This commit closes
them by widening the lock scope, publishing the pending load
target through status(), and switching delete guards to
fail-closed.
Lifecycle (P1)
* core/inference/diffusion.py: load_model now also takes
_generate_lock. Previous behavior released and reallocated the
pipeline while a generation forward was still iterating
denoising steps, corrupting scheduler state and stacking VRAM.
The forward only briefly touches _lock, so taking it on the
load path does not introduce a deadlock.
* core/inference/diffusion.py: unload_model now also takes
_generate_lock. Without it, /images/unload returned
is_loaded=False while a slow forward was still running, which
let chat / training / export handoffs allocate VRAM on top of
the still-resident pipeline.
* core/inference/diffusion.py: previous pipeline release now
happens BEFORE from_single_file / from_pretrained. Switching
FLUX.2 klein 4B -> 9B on a 16-24 GB GPU was failing because
the new transformer allocation overlapped the old pipe's
residency.
* core/inference/diffusion.py: failed pipeline from_pretrained
now explicitly releases the just-loaded transformer; previously
its weights stayed pinned to GPU until GC and made the next
load more likely to OOM.
Pending-target / delete guards (P1)
* core/inference/diffusion.py: load_model now publishes
_pending_repo_id / _pending_base_repo / _pending_gguf_filename
under _lock at the start of the call (and refreshes
_pending_base_repo when the smart-base / repo defaults resolve).
status() exposes those as 'repo_id' / 'base_repo' /
'gguf_filename' during is_loading=True so delete guards can see
the target before _repo_id is set on success.
* routes/models.py /delete-cached + /delete-finetuned: diffusion
status check now fails CLOSED (HTTP 503) when status() raises.
Both guards previously logged and continued, which could let a
delete proceed against a repo whose status was unverifiable.
* routes/models.py: is_loading is also blocked on both guards
so a mid-download / mid-from_pretrained rmtree is refused.
Symmetric handoffs (P1)
* routes/export.py: /load-checkpoint now refuses with HTTP 409
when training is active instead of calling stop_training().
Chat and /images/load did the same after round 5; export was
the remaining asymmetry that would silently kill a long
training run.
* routes/training.py, routes/inference.py (GGUF and standard
chat), routes/export.py: diffusion handoff now treats
is_loading as is_loaded. The diffusion backend's unload waits
on _load_lock + _generate_lock so an in-flight load completes
first.
Requirements (P1)
* requirements/studio.txt: pin python-multipart explicitly. The
Studio routes package's eager router imports include
routes/datasets.py whose FastAPI UploadFile/File validation
crashes with RuntimeError without it in fresh test envs.
Frontend (P2)
* features/images/api.ts + images-page.tsx: seed handling now
accepts the full [-2^63, 2^64 - 1] range via BigInt. The
previous safe-integer cap rejected valid uint64 seeds the
backend accepts. A small stringify helper emits BigInts as JSON
integers without touching the rest of the payload.
Tests
* test_diffusion_routes.py: load routes/inference.py via
importlib.spec_from_file_location to avoid triggering
routes/__init__.py (which would pull in training / datasets /
data_recipe imports unrelated to diffusion tests).
* test_diffusion_backend.py: status() during is_loading shows
pending repo + base; unload waits for in-flight generation.
Round 5 reviewer findings, mostly symmetric-lifecycle and input
validation gaps the earlier rounds left open.
Backend lifecycle (P1)
* routes/training.py: training start now also unloads the GGUF
llama-server subprocess; was previously only unloading the
safetensors backend, so starting training while a GGUF chat
model was loaded kept the subprocess pinned to VRAM.
* routes/inference.py: new _raise_if_training_active helper. Both
GGUF and standard chat loads, plus /api/inference/images/load,
now refuse with HTTP 409 when training is active instead of
silently stopping training to free VRAM.
* core/inference/diffusion.py: _release_other_gpu_owners_for_
diffusion no longer stops active training. The route layer
refuses the request first, so reaching the helper with training
live would only happen from programmatic backend calls; better
to surface OOM than terminate a long training run.
* core/inference/diffusion.py: BF16 dtype is now gated on
torch.cuda.is_bf16_supported. Pascal/Turing GPUs report
is_available()=True but lack BF16 ALUs; FLUX kernels then fail
inside from_pretrained. Falls back to FP16 instead of refusing.
* core/inference/diffusion.py: GGUF transformer allocation and
pipeline allocation now run AFTER releasing chat/export GPU
owners; previously from_single_file ran first and could OOM
before the intended VRAM handoff happened.
* routes/models.py: /delete-cached now also blocks delete when
diffusion is_loading=True (not just is_loaded); concurrent
delete during hf_hub_download / from_single_file would have
raced the rmtree.
* routes/models.py: /delete-finetuned now also checks the
diffusion backend before unlinking a Studio outputs/exports
path. A user who exported a FLUX LoRA locally and loaded it via
/images/load could previously rmtree the directory the
diffusion backend was reading from.
Backend correctness / safety (P2)
* core/inference/diffusion.py: _FAMILY_EXCLUDE for qwen-image now
also covers qwen_image_edit / qwenimageedit underscore spellings
so '...qwen_image_edit-GGUF' no longer misdetects as Qwen-Image.
* core/inference/diffusion.py: detect_family now scans
_FULL_REPO_FAMILIES in addition to _FAMILIES, so SDXL repos
(stabilityai/stable-diffusion-xl-base-1.0) are auto-detected
instead of failing with 'Could not infer a diffusion family'.
* core/inference/diffusion.py: generate_image now uses a separate
_generate_lock for the pipeline forward instead of holding
_lock for the whole call. status() polls and concurrent unload
requests no longer block for the full minutes-long generation.
* routes/models.py: diffusion delete guard now uses exact repo-id
match instead of prefix match; previously loading 'org/model-v2'
would block deleting unrelated cached 'org/model'.
* models/inference.py: DiffusionLoadRequest now rejects ASCII
control characters in repo_id / gguf_filename / base_repo /
family via field_validator (closes log-injection surface from
authenticated callers). Also caps lengths at 256 chars.
* models/inference.py: DiffusionGenerateRequest seed is now
bounded to the int64/uint64 range; previously a huge seed
(e.g. 2**100) passed Pydantic then crashed inside
torch.Generator.manual_seed with 'Overflow when unpacking long
long'.
Frontend (P2)
* features/images/images-page.tsx: Custom HF repo panel now
exposes a Pipeline family override dropdown; previously the
backend supported it via DiffusionLoadRequest.family but the UI
had no way to send it, so custom repos whose names did not
contain a hard-coded substring failed to load.
* features/images/images-page.tsx: handleLoad now re-fetches
status on error. The backend clears its old pipeline before
allocating the replacement; a failed swap previously left the
UI showing 'Loaded:' with Generate enabled until manual
refresh.
Tests (10 new)
* underscore qwen-image-edit exclusion + SDXL full-repo detection
* BF16 fallback when is_bf16_supported() returns False
* status() does not block while generate_image holds _generate_lock
* route layer rejects control chars in repo_id
* route layer rejects 2**100 seeds (uint64-max boundary accepted)
* route layer happy-path with negative-prompt true_cfg_scale
forwarding (Qwen/Flux) and skip-when-no-neg (distilled CFG)
The Studio backend's no-torch-runtime.txt is installed via
pip --no-deps so the diffusion stack's transitive imports must
be pinned explicitly. huggingface_hub's blob downloader (used by
diffusers.GGUFQuantizationConfig and by every from_single_file
call) imports requests + urllib3 + charset_normalizer at module
load time; a fresh --no-deps install would 500 on the first
/api/inference/images/load with PackageNotFoundError: 'requests'.
Adds requests, urllib3, and charset_normalizer to the
transitive-deps block next to the existing httpx chain.
QwenImagePipeline and FluxPipeline treat guidance_scale as the
distilled CFG factor and expose true_cfg_scale as the real
classifier-free guidance knob. Negative prompts only steer the
output when true_cfg_scale > 1, so forwarding only guidance_scale
left Qwen-Image on the default true_cfg_scale=4.0 and the user's
slider value silently ineffective for negative prompts.
When the loaded pipeline accepts both negative_prompt and
true_cfg_scale and the caller supplies a non-empty negative
prompt, forward guidance_scale through both kwargs so the
negative prompt actually steers generation. When no negative
prompt is supplied, true_cfg_scale is left at the model default
to avoid switching distilled CFG models into real-CFG mode (which
would double inference cost and degrade quality).
Adds two regression tests covering the forward-when-negative and
skip-when-no-negative paths.
- DiffusionBackend.status() now takes _lock so frontend polling
cannot observe a torn snapshot mid-swap.
- Scrub hf_token / pipe_kwargs / single_file_kwargs from frame
locals before logger.exception() so rich tracebacks and structlog
formatters that render locals do not leak hf_... tokens into logs.
- routes/models.py delete_cached_repo: refuse to delete the cache
underlying a currently-loaded diffusion pipeline (both the GGUF
repo and the matching diffusers base_repo). Symmetric with the
existing chat-load + GGUF guard.
- Frontend seed validation: reject non-integer and out-of-safe-
integer-range inputs instead of silently rounding via Number(),
which would otherwise send a different seed than what the user
typed.
When a swap load fails after the previous pipeline is released,
status() previously reported is_loaded=false on top of the OLD
repo/family/base_repo metadata, which the frontend then rendered
as a misleading 'still loaded: X' label. Clear all metadata
atomically with the pipe drop so a failed swap reports a clean
empty status plus last_error. Add regression test.
- detect_family adds _FAMILY_EXCLUDE so 'stable-diffusion-3.5' no
longer matches the SD3 Medium family and 'qwen-image-edit' no
longer matches Qwen-Image. Both were misleading silent loads.
- from_single_file now forwards config=<effective_base>,
subfolder='transformer', and the HF token. Diffusers-format GGUFs
(FLUX.2 klein, Qwen-Image, SD3) need the matching base config or
the transformer load picks the wrong shapes; gated GGUFs need the
token both for download and config read.
- Move _release_chat_backend_for_diffusion + new
_release_other_gpu_owners_for_diffusion to AFTER the GGUF download
and pipeline class lookup so a typo or transient Hub error does
not kill the user's currently-loaded chat model. Peak VRAM still
stays at one model's worth because the releases run right before
from_pretrained.
- _release_other_gpu_owners_for_diffusion: shut down the export
subprocess and any active training subprocess before a diffusion
load. Symmetric with the export load path.
- routes/training.py: unload diffusion before starting training so
the new subprocess does not race FLUX/Qwen for VRAM.
- routes/export.py: also unload the GGUF llama-server before export
load (the existing inference-backend unload only covered the
safetensors path).
_release_chat_backend_for_diffusion was importing
get_inference_backend from core.inference.inference (the in-subprocess
class) and calling unload_model() without the required model_name
argument. The TypeError was swallowed and the active chat model
stayed resident, defeating the chat-to-diffusion lifecycle handoff.
Switch to the orchestrator's accessor at core.inference and pass
active_model_name through, mirroring the GGUF chat-load path. Add a
regression test that stubs both backends and verifies unload_model
is called with the active model name.
- routes/export.py load_checkpoint now unloads the diffusion
pipeline alongside the existing inference + training unloads, so
an export load after Images does not OOM the export subprocess.
- Remove the 'sd3.5' alias from the stable-diffusion-3 family.
SD3.5 needs its own family + base_repo (and its own smoke test);
pairing it with the SD3 Medium base produced a misleading load.
- _smart_base_repo: pick 9B base for unsloth/FLUX.2-klein-9B-GGUF
and -base- variants per the repo id, instead of always falling
back to the 4B family default.
- pipe_kwargs use_safetensors=True so diffusers refuses pickle .bin
weights at load time (defends against compromised base_repo).
- Release the previous pipeline BEFORE allocating the new one so
peak VRAM stays at one model's worth instead of two on swap.
- Reject empty gguf_filename when repo_id ends with -GGUF; the prior
behavior tried from_pretrained on a GGUF-only repo and 500'd deep
in diffusers with a confusing model-index error.
- Status returns gguf_filename (basename) instead of gguf_path so
the local cache path / username does not leak to authenticated
Studio sessions.
- requirements/no-torch-runtime.txt: pin diffusers>=0.37.0 so older
installs cannot resolve a version without Flux2KleinPipeline.
- Frontend curated distilled klein entries now point at the
matching non-base diffusers repos (FLUX.2-klein-4B / -9B) per
the published model cards. Update api.ts to mirror the renamed
status field.
_release_chat_backend_for_diffusion now unloads both the GGUF
chat backend (llama-server) and the safetensors / HF chat backend
(get_inference_backend) before a diffusion load. Mirror the
behaviour on the chat-load side: both the Unsloth/transformers
load path and the GGUF load path now unload the diffusion pipeline
before claiming GPU memory. Closes the OOM-on-swap path flagged
by reviewers in both directions.
- unload_model now takes _load_lock so it cannot race with an in-flight
load_model and have the load thread overwrite cleared state after
unload returned is_loaded=false.
- Move stable-diffusion-xl out of _FAMILIES into _FULL_REPO_FAMILIES.
SDXL uses a UNet (no transformer GGUF path is wired); listing it in
the GGUF families panel was misleading. SDXL full-repo loads still
work via family_override='stable-diffusion-xl'.
- Result gallery now uses h-auto + object-contain so portrait /
landscape outputs render at their true aspect ratio instead of
being cropped into a square thumbnail.
Backend
- Fix FLUX.2 klein family default base_repo: black-forest-labs/FLUX.2-klein
does not exist on the Hub. Point at the Apache 2.0 4B Base instead so
the from_pretrained call works out of the box for ungated users.
- Serialise concurrent load_model calls with a dedicated _load_lock so
two /images/load requests cannot both reach pipeline_cls.from_pretrained
at the same time (would double-spend VRAM and corrupt _pipe).
- When the caller passes a full diffusers repo (no gguf_filename),
use repo_id directly instead of silently substituting the family
default. Closes the load-the-wrong-model regression flagged by review.
- Drop negative_prompt from the pipeline call when the loaded pipeline
does not accept it (FLUX.2 / FLUX.2 klein). Inspect __call__ via
inspect.signature so we do not maintain a manual class list.
- Best-effort unload the chat backend (llama-server) before a diffusion
load so a 24 GB consumer GPU can swap between chat and diffusion
without manual unload steps.
Frontend
- Replace the four curated entries with the actual filenames published
on the Hub (lowercase flux-2-klein-Nb-Q4_K_S.gguf and flux2-dev*).
- Add an explicit base_repo per curated entry so the backend never
falls back to the family default for the curated picker.
- Add the Apache 2.0 FLUX.2 klein base 4B entry so first-time users
have an ungated, no-token-required default.
- Hide the negative prompt field for FLUX.2 / FLUX.2 klein and show a
small explanatory note instead.
Tests
- Add 6 new backend tests: base_repo override, full-repo (no GGUF)
no-substitution, concurrent serialise race, signature-based kwarg
filter, negative_prompt strip on FLUX.2, negative_prompt preserved
on supporting pipelines. 33 tests passing.
Backend
- core/inference/diffusion.py: DiffusionBackend singleton that loads
diffusion GGUFs from Hugging Face via diffusers.GGUFQuantizationConfig
and runs them on the active CUDA / MPS / CPU device. Supports FLUX.2,
FLUX.2 klein, FLUX.1, Qwen-Image, Stable Diffusion 3, and SDXL.
- routes/inference.py: POST /api/inference/images/load,
POST /api/inference/images/generate, POST /api/inference/images/unload,
GET /api/inference/images/status mirroring the llama-server lifecycle.
- models/inference.py: DiffusionLoadRequest, DiffusionGenerateRequest,
DiffusionGenerateResponse pydantic schemas with prompt / step / size
validation up front so callers get clear 422s rather than VAE crashes.
- requirements/no-torch-runtime.txt: pin gguf alongside the existing
diffusers entry so GGUFQuantizationConfig works out of the box.
- tests/test_diffusion_backend.py + tests/test_diffusion_routes.py:
27 unit tests covering family detection, validation, lifecycle, and
the full FastAPI round trip with the backend stubbed. No torch /
diffusers / GPU required to run.
Frontend
- features/images/: standalone images-page.tsx with curated model picker
(FLUX.2 klein 4B / 9B, FLUX.2 dev, FLUX.1 dev), HF token field,
prompt + negative prompt, resolution presets, steps + guidance
sliders, seed input, and a result gallery that renders base64 PNGs
inline.
- app/routes/images.tsx: lazy /images route wired into router.tsx.
- components/app-sidebar.tsx: PaintBrush02Icon nav item between
Recipes and Export, hidden in chat-only mode.
* Studio: strip orphan tool_call XML from streamed visible content
The speculative-buffer state machine in
`studio/backend/core/inference/llama_cpp.py` can slice a tool_call XML
block between the silent DRAINING path and the user-visible
content_accum, depending on when in the model's emission the BUFFERING
-> STREAMING -> DRAINING transitions fire. Three leak shapes were
observed in a 2026-05-22 sweep of 900 Qwen3.5 / Qwen3.6 GGUF runs:
Pre-fix XML leak rate: 20/900 (2.22%), concentrated 6.7% on the
larger Q8 / MTP configs:
Qwen3.6-35B-A3B Q8_0 4/60 (6.7%)
Qwen3.6-35B-A3B-MTP Q4 4/60 (6.7%)
Qwen3.5-35B-A3B Q8_0 3/60 (5.0%)
Qwen3.6-27B Q8_0 3/60 (5.0%)
The existing `_TOOL_XML_RE` only matched well-formed
`<tool_call>...</tool_call>` and `<function=...></function>` pairs, so
unterminated openings (close was DRAINED) and orphan closes (opening
was DRAINED) survived the strip and reached the user.
Fix relaxes the regex to also strip:
1. Orphan opening up to end-of-string: `(?:</tool_call>|\Z)`
2. Orphan closing tag: bare `</tool_call>` / `</function>`
Verified on the full sweep: 20/900 -> 0/900 (100% of detected leaks
eliminated). 16 unit tests in `test_tool_xml_strip.py` pin all three
leak shapes plus the well-formed cases, plus parametrised checks on
the 5 actual real-world leak samples from the sweep data.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: strip tail-only </parameter> orphan + tighten regex
The 2026-05-22 gdpval sweep surfaced a 4th XML-leak shape not caught
by the earlier regex: a bare `</parameter>\n\n` at end-of-buffer (7
of 192 trials, all Qwen3.5-27B + a few Qwen3.6-27B). The model emits
the full `<tool_call><function=...><parameter=...>...content...
</parameter></function></tool_call>` envelope, the speculative buffer
DRAINS the opening tags as intended, but EOS (max_tokens cutoff)
truncates the outer `</function></tool_call>` close, leaving just
`</parameter>` as the visible tail.
We strip this ONLY when end-anchored (`\s*\Z`) so legitimate
mid-text uses (user code samples, documentation discussing the
Qwen tool-call XML shape) survive. Verified on the 192-trial
gdpval corpus: before=7, after=0.
While at it, fold the five top-level alternations into three by
sharing tag-name and prefix subgroups:
<tool_call>... + <function=\w+>... + --> <(?:tool_call|function=\w+)>...
</tool_call> | </function> --> </(?:tool_call|function)>
Semantically identical (verified by replay over the 192-trial
corpus + adversarial inputs, 0 diffs) and 1.34x faster on real
workloads. Backtracking-safety pinned by two new perf guards
(256KB '<' spam, 1000x orphan opens).
Tests: 16 -> 28 (6 new functional + 4 well-formed-vs-orphan +
2 perf guards).
* Tighten comments in XML-strip regex and tests
Code says what it does; comments were repeating it. Strip the verbose
explanations down to the WHY-only bits (engine quirk, tail-anchor
rationale, real-world source of each test sample). No code changes.
inference.py: 21 -> 12 lines around _TOOL_XML_RE
test_tool_xml_strip.py: 343 -> 259 lines (-84)
Tests: 28/28 still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* ci: broaden Linux llama.cpp runtime pattern to lib*.so*
#5741 patched the explicit Linux pattern list to add
``libllama-*-impl.so*`` after ggml-org/llama.cpp#23462 (between
b9279 and b9283) split each binary's entry code into a paired
``lib<binary>-impl.so`` shared library. Same class of upstream
repackaging will hit us again whenever a new shared lib is added.
Mirror what macOS already does and replace the per-lib list with a
single ``lib*.so*`` glob. ``copy_globs`` (line 3614) unions
patterns, so the per-variant ``libggml-cuda.so*`` / ``libggml-hip.so*``
entries were never filtering anything; the spec lives in
``runtime_payload_health_groups`` (line 5209) which keeps the
explicit minimum-required list per variant.
Dry-run against b9296-bin-ubuntu-x64.tar.gz: 40 files copied (all
ggml, llama, mtmd, impl variants + the two binaries we ship), 22
skipped (other CLIs, rpc-server, LICENSE). Functionally equal to
the post-#5741 set.
* cleanup: trim #5741 comments on the pydantic split
Comments added in #5741 explained the original bug in full each
time. They are mostly redundant with the commit message and the PR.
Trim them to one short paragraph per site.
No behavior change.
* ci: narrow Windows runtime pattern to llama-server.exe + llama-quantize.exe
Studio only invokes llama-server and llama-quantize. Mac and Linux
already filter to those two binaries; Windows was the odd one out
with ``*.exe`` copying every CLI upstream ships (llama-cli,
llama-bench, llama-mtmd-cli, ...).
Dry-run on b9296 (win cpu-x64, cpu-arm64, cuda-13.1, hip-radeon):
20 unused EXEs skipped per variant, all DLLs (incl. the new
llama-*-impl.dll family) still copied via ``*.dll``.
``existing_install_matches_choice`` already checks llama-server.exe
exists explicitly (line 5297), so the health gate is unchanged.
Bundles three independent CI regressions hitting the maintainer PR
backlog. Each one is verified end-to-end on a staging fork against
real Ubuntu / macOS / Windows GitHub-hosted runners before this
lands.
1. Windows --no-torch install: pydantic + pydantic-core drift to
incompatible versions under `uv pip install --no-deps -r
no-torch-runtime.txt` because pip resolves each independently
from latest. pydantic.VERSION 2.13.4 pins pydantic-core==2.46.4
but pydantic-core 2.47.0 was the freshest published wheel, so
`import pydantic` raised
`SystemError: pydantic-core 2.47.0 is incompatible with the
current pydantic version`. Resolve pydantic WITH deps in a
focused pip call (install.sh, install.ps1,
install_python_stack.py) before the --no-deps no-torch-runtime
pass so pip pins pydantic-core to the version pydantic declares.
pydantic's transitive deps (annotated-types, pydantic-core,
typing-extensions, typing-inspection) are torch-free. Drop the
redundant `Patch Studio venv with full typer / pydantic dep
trees` workaround from the four Windows smoke YAMLs.
Supersedes #5733 + #5734.
2. Linux Studio Update CI: upstream llama.cpp b9261+ split each
binary's entry code into a paired `libllama-<binary>-impl.so`
shared library. `llama-server` and `llama-quantize` NEEDED-link
against `libllama-server-impl.so` / `libllama-quantize-impl.so`
with RUNPATH `$ORIGIN`, so the prebuilt overlay must copy those
alongside the binaries. Without that, ldd reports them missing,
preflight rejects, the installer falls back to source build, and
studio-update-smoke annotates `setup.sh idempotency regressed`.
Add `libllama-*-impl.so*` to the Linux runtime patterns and lock
the pattern in test_rocm_support.TestRuntimePatterns.
3. Mac Studio UI Chat: change-password submit clicked while
disabled. The disable gate only checked new + confirm password
length, but Playwright's first click landed before the
current-password field's React state had committed, so the form
was simultaneously logically-invalid (current_password empty) and
the button was disabled. Tighten the gate to require
`currentPassword.length >= 8` and mirror the same check in the
submit handler so Enter / autofill cannot bypass.
Supersedes #5738.
* Studio: PDF / document attachments for Anthropic + OpenAI
Studio's local-GGUF chat already supports image attachments via the
`image_url` content part shape. PDFs and other documents had no
plumbing for the external-provider path: there was no normalised
content type the frontend could send that translated to Anthropic's
native `document` block or OpenAI's `input_file`.
Add a Studio-side `input_document` content part on assistant /
user messages with three shapes:
{type: "input_document",
file_data: "data:application/pdf;base64,<DATA>",
filename?: "name.pdf",
media_type?: "application/pdf"}
{type: "input_document",
file_url: "https://example.com/doc.pdf",
filename?: "doc.pdf"}
Translation:
- Anthropic Messages API: emits a `document` block with
`{source: {type:"base64", media_type, data}}` or
`{source: {type:"url", url}}`, plus an optional `title` from
`filename`. PDFs are extracted server-side by Anthropic per their
vision/document docs and counted toward input tokens.
- OpenAI Responses API: emits `{type:"input_file", file_data |
file_url, filename?}`. PDFs are extracted server-side.
Empty / unparseable `input_document` parts are silently dropped so
a malformed frontend payload can't blow up the request.
Tests:
- New `test_multimodal_document.py` with 6 cases pinning the
outbound body shape for base64 + URL inputs on both providers,
and the empty-part drop behavior on both.
- The Anthropic assertions strip the prompt-cache wrapper
(`cache_control:{type:ephemeral}` that the tail-message caching
layer adds) before comparing the document core fields, so this
test stays focused on the translation, not the caching layer.
Live verified end-to-end against both providers: a 363-byte
single-page "HELLO" PDF, base64-encoded, attached as a `document`
block to Opus 4.7 and as an `input_file` to gpt-5.5. Both models
correctly extracted the word "HELLO" from the PDF.
Follow-up (out of scope):
- Pydantic schema entry on ChatMessage.content for `input_document`
(today it rides through because ChatCompletionRequest uses
extra=allow). Will tighten when the frontend attach button lands.
- Frontend file-picker UX for non-image attachments on the external
provider path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: gate empty-content msg + skip empty data-URI payload
Gemini High + Codex P2 on PR #5689:
1. Anthropic translation appended an empty `anthropic_parts` array
when every part was dropped (e.g. user sent only an unparseable
input_document). Anthropic 400s on "messages.N.content: at least
one block is required". Skip the whole-message append when no
parts survived. The OpenAI Responses path already had the
equivalent guard, so this brings the two providers into parity.
2. `data:application/pdf;base64,` with no payload (or whitespace-only)
parses to an empty `source.data` string. Anthropic rejects that
with 400 as well. Skip the document block before constructing it.
Plus 2 new test cases pinning both behaviors:
- `test_anthropic_empty_only_document_drops_whole_message`: confirms
a turn whose only content is an unparseable input_document does
NOT make it onto the outbound `messages` array.
- `test_anthropic_empty_data_uri_payload_is_dropped`: confirms an
empty-payload data-URI is filtered out at translation time.
(Note re: gemini's other High note about adding `input_document` to
the Pydantic ContentPart union -- ChatCompletionRequest is configured
with `extra=allow` so the part rides through today. Tightening the
union belongs with the frontend attach-button PR that surfaces the
field; called out as follow-up in the PR description.)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: register input_document in ContentPart + builder
Reviewer caught that the translation code on the external_provider
side was unreachable from a real ChatCompletionRequest:
- ContentPart is a discriminated Union of (text, image_url) only, so
any `{"type": "input_document", ...}` part was rejected by Pydantic
at request parsing with a discriminator error before the helper
could see it.
- _build_external_messages in routes/inference.py only walked text
and image_url parts, so even with a permissive schema the document
parts would have been silently dropped instead of forwarded to
the per-provider translator.
Fixes:
- Add InputDocumentContentPart with optional file_data / file_url /
filename / media_type and Tag("input_document") on the Union.
- Extend _build_external_messages to pass input_document through as
a plain dict for vision-capable providers (so external_provider's
existing Anthropic `document` and OpenAI Responses `input_file`
mappers actually run) and strip them on non-vision providers.
Tests added: schema accepts input_document, builder passes it to
vision providers, builder strips it on non-vision providers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: validate file_data before preferring over file_url
Codex P2 caught that the OpenAI input_document translator treats any
truthy file_data as valid and never falls back to file_url. That
means a malformed `data:application/pdf;base64,` (empty payload) or
a whitespace-only data URI gets forwarded as `file_data=""` and
400s the whole turn, AND silently discards a perfectly recoverable
file_url on the same part.
Mirror the Anthropic-side guard onto the OpenAI Responses path:
treat any "data:" URI with no actual base64 payload as missing and
fall through to file_url. Standalone-empty data URIs (no fallback)
are dropped entirely instead of being sent to the wire.
Tests added: empty data URI + valid file_url -> file_url wins,
whitespace-only data URI + valid file_url -> file_url wins,
empty data URI without fallback -> part is dropped.
* Address review: Anthropic side also falls back to file_url on empty data URI
Codex P2 follow-up to my earlier fix: I added the empty-data-URI ->
file_url fallback to the OpenAI Responses translator but missed
the Anthropic translator, which still `continue`d on empty payloads
and discarded an otherwise valid file_url on the same part. Result:
when the frontend supplied both file_data (placeholder / broken)
AND a working file_url, Anthropic silently lost the attachment;
when the message contained only that part, the whole message could
be dropped before reaching the wire.
Mirrored the OpenAI guard: any "data:" URI with no actual base64
payload (`data:application/pdf;base64,` or whitespace-only) is
treated as missing, and the file_url branch takes over. The
all-parts-dropped guard further down already handles the
no-fallback case.
Tests added: empty data URI + valid file_url -> URL source on the
wire with the filename preserved; whitespace-only data URI + valid
file_url -> URL source on the wire.
* Address review: gate input_document passthrough to anthropic + openai
Codex P1: only `_stream_anthropic` and `_stream_openai_responses`
have explicit translation logic for input_document parts (the former
maps to {type:"document", source:...}, the latter to
{type:"input_file", file_data|file_url}). Every other provider
(gemini / mistral / kimi / openrouter / deepseek / qwen / custom)
goes through the generic /chat/completions passthrough that forwards
`messages` verbatim, so any input_document part on a non-vision
route on those providers would 400 with an unknown content_part
type.
Added `_INPUT_DOCUMENT_PROVIDERS = frozenset({"anthropic", "openai"})`
constant and gated the pass-through branch on `provider_type in
_INPUT_DOCUMENT_PROVIDERS`. Every other provider strips the part
(text content survives). Threaded provider_type through from
_proxy_to_external_provider's call site.
Tests updated: vision + provider in {anthropic, openai} still
forwards; six unmapped providers (gemini/mistral/kimi/openrouter/
deepseek/qwen) strip the part; missing provider_type strips
defensively. The existing non-vision drop test still passes.
* Fix stale web_fetch tool-version assertion after merging main
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: wire OpenAI Responses server-side context compaction
The OpenAI Responses API accepts a `context_management` field that
enables server-side compaction. When the rendered prompt crosses the
configured threshold, the API runs a server-side compaction step and
the request continues against the compacted prefix. No beta header
and no dated version pin are required, per the docs.
Changes:
- Add `compaction_threshold: Optional[int]` (ge=1_000, le=2_000_000)
to ChatCompletionRequest. Thread through `routes/inference.py` ->
`stream_chat_completion` -> `_stream_openai_responses`.
- In `_stream_openai_responses`, when threshold is set AND the base
URL points at cloud OpenAI (api.openai.com), attach
`context_management: [{type:"compaction", compact_threshold:N}]`
to the outbound body. Non-cloud bases (ollama, llama.cpp, "custom"
presets) silently drop the field so we don't 400 those servers.
- Add `test_openai_compaction.py` with 4 cases: cloud OpenAI sets
the field verbatim, low-threshold probe passes through (we don't
clamp on the OpenAI side because the API accepts whatever),
non-cloud base drops the field, omitted threshold leaves body
untouched.
Live verified against the real OpenAI API on gpt-5.5:
`context_management:[{type:"compaction", compact_threshold:200000}]`
returns 200 with no error.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: accept Azure OpenAI base URLs + raise compaction floor
Two reviewer follow-ups on the OpenAI compaction PR:
1. The `is_openai_cloud = "api.openai.com" in self.base_url` check
excluded Azure OpenAI Foundry, even though Azure exposes the
same /v1/responses extensions (context_management,
prompt_cache_retention, container shell). Users on Azure saw
their compaction toggle silently no-op. Broadened the check to
also match `*.openai.azure.com` and made it case-insensitive so
URLs copy-pasted from the Azure portal still resolve. Non-cloud
OpenAI-compatible servers (ollama / llama.cpp / vLLM / "custom"
preset) still fall outside the gate.
2. The schema floor on compaction_threshold was ge=1_000, which is
well below the upstream Responses API's effective minimum
(vercel/ai#12486, langchain-ai/langchain#35464 report
`compact_threshold is not enabled` 400s on Azure at 100k; cloud
uses 200k as the canonical example). Raised the floor to 10k
so obvious typos surface as a clean 422 from FastAPI rather than
an opaque upstream 400 the user has to debug from the SSE
stream.
Tests added: Azure base URL carries both context_management and
prompt_cache_retention; mixed-case Azure URLs match; schema rejects
9_999 and accepts 10_000.
* Address review: drop schema-level compaction floor (cross-provider regression)
Codex P2 follow-up on the previous floor bump: ge=10_000 was
enforced globally at the ChatCompletionRequest layer, but the field
is documented as a no-op on every non-cloud OpenAI base and every
non-OpenAI provider. With the global floor, an Anthropic / ollama
/ llama.cpp / custom request that happens to carry compaction_threshold
below 10k was rejected with 422 at request validation time instead
of being silently ignored as the description promised.
Reverted the schema floor to ge=1 (any positive int) and rewrote
the description to call out per-provider routing: OpenAI cloud's
effective floor is around 200k and surfaces upstream 400s below
that; _stream_anthropic clamps sub-50k values up. Per-provider
helpers stay the single source of truth on the floor.
Test updated to pin: zero is still rejected, but every positive
value (1, 5_000, 9_999, 10_000, 200_000) passes schema validation.
* Address CodeQL: hostname-anchored OpenAI cloud detection
CodeQL py/incomplete-url-substring-sanitization fired on
`".openai.azure.com" in _base`. An attacker who controls the
configured base_url could slip cloud-only request body fields
(prompt_cache_retention, context_management compaction, container
shell) to an arbitrary server with:
https://evil.com/api.openai.com/v1https://api.openai.com.attacker.com/v1https://attacker.com/.openai.azure.com/v1https://my-resource.openai.azure.com.attacker.com/openai/v1
Replaced the substring check with a `_is_openai_family_cloud`
helper that runs urllib.parse.urlparse on the URL and matches the
lowercased hostname exactly (`api.openai.com`) or via `endswith`
on the leading-dot suffix (`.openai.azure.com`). Both halves are
host-anchored so path / fake-subdomain bypasses fail.
Test added: every attacker-controlled bypass shape above must NOT
carry context_management OR prompt_cache_retention on the wire.
Existing Azure and openai.com tests still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: scope compaction_threshold description to OpenAI on this branch
Codex P2: the field description on this PR mentioned Anthropic
compaction behavior, but the Anthropic wiring lives on PR 5686
(separate branch). On feat/openai-compaction alone, _stream_anthropic
has no compaction_threshold parameter, so the field is silently
ignored for Anthropic requests and the doc claim was misleading.
Trimmed the description to OpenAI cloud + Azure Foundry only on
this branch. PR 5686 already re-adds the Anthropic clause via its
own change, so the rebase / merge order on main will land the
combined description naturally once both PRs ship.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: wire Anthropic server-side context compaction
Anthropic ships server-side context compaction as a beta
(`compact-2026-01-12`). When the rendered prompt crosses the
configured input-token threshold, Anthropic runs an extra LLM pass
that summarises older turns and the request continues against the
compacted prefix. The response carries the original top-level fields
plus a new `context_management` block (with `applied_edits`) and
`usage.iterations[]` accounting per pass.
Per the docs the feature is currently supported on Opus 4.6, Opus 4.7,
Sonnet 4.6, and Mythos preview. The minimum threshold is 50k tokens;
under-50k requests 400.
Changes:
- Add prefix gate + helper `_anthropic_supports_compaction` plus
constants `_ANTHROPIC_COMPACTION_PREFIXES`, `_ANTHROPIC_COMPACTION_BETA`,
`_ANTHROPIC_COMPACTION_TYPE`, `_ANTHROPIC_COMPACTION_MIN`.
- Add `compaction_threshold: Optional[int]` to ChatCompletionRequest
(50k ge bound, 2M le bound). Thread through `routes/inference.py`
-> `stream_chat_completion` -> `_stream_anthropic`.
- In `_stream_anthropic`, when threshold is set AND the model
accepts compaction, attach `context_management.edits[{type:
"compact_20260112", trigger:{type:"input_tokens", value:N}}]` to
the outbound body. Sub-50k values are clamped up to 50k to keep
the request well-formed.
- Refactor the anthropic-beta header builder to merge any combination
of `code-execution-2025-08-25` + `compact-2026-01-12` flags into
one header value. Unrelated betas added at the registry level still
pass through.
- Add `test_anthropic_compaction.py` with 16 cases: gate matrix
(every doc-listed model), correct body shape, threshold clamping,
beta header merge with code execution, silent no-op on unsupported
models, omitted-threshold pass-through.
Live verified end-to-end against the real Anthropic API:
`compact_20260112` accepted on Opus 4.7, response carries
`context_management.applied_edits` + `usage.iterations[]` as
documented. (The first WebFetch-summarised version of these docs
suggested `compact_20260120`; the actual API only accepts
`compact_20260112`, matching the beta-header date. Worth pinning
behind a test so a future doc update can't drift back.)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: drop ge=50_000 clamp + parse usage.iterations[]
Two reviewer follow-ups on the compaction PR:
1. Pydantic ge=50_000 on compaction_threshold was dead code.
FastAPI rejected sub-50k threshold values with a 422 before the
`max(int(...), _ANTHROPIC_COMPACTION_MIN)` clamp in
_stream_anthropic could ever fire. Relaxed the floor to ge=1 so
the in-helper clamp actually does its job; the schema comment
now explains why this is intentional. Added a regression test
that posts a value of 1 and 49_999 through the real request
schema.
2. Anthropic publishes per-iteration token counts in
`usage.iterations[]` whenever a fresh compaction has run, and
the top-level input_tokens / output_tokens cover only the
`message` iteration -- billing must add the compaction
iterations on top. Aggregate compaction iteration tokens into
`last_usage["compaction_input_tokens" / "compaction_output_tokens"]`
so the cost surface (PR 5690) can read them without re-walking
the array, and surface both figures in the closing stream
summary log. Added two tests: one that pins the aggregation on a
compacted turn and one that pins `None` when no fresh
iterations land (so re-applied compaction blocks don't double-bill).
Sourcing: https://platform.claude.com/docs/en/build-with-claude/compaction
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: round-trip Anthropic compaction blocks across turns
Codex P1: once context_management is enabled and Anthropic runs
server-side compaction mid-stream, the response carries a
`{type:"compaction", content:"<summary>"}` content block on the
assistant message. The translator only handled text_delta and
input_json_delta on content_block_delta, so the compaction block
was silently dropped. Worse, the request schema's ContentPart
discriminated Union didn't accept `type:"compaction"`, and
_build_external_messages didn't pass it through, so even a
hand-crafted assistant message carrying the block would 422 at
parse time. Net result: Anthropic re-compacted from scratch on
every subsequent turn, wasting input tokens and reasoning budget.
End-to-end backend wiring of the round-trip:
1. SSE translator. _stream_anthropic now tracks a `current_compaction`
state slot. content_block_start with type=="compaction" seeds it
(Anthropic may include the summary on the start event AND/OR
stream it via text_delta events on the same block index --
handle both). text_delta inside a compaction block routes into
the compaction buffer instead of the user-visible content
stream, since the summary is opaque internal state, not
assistant prose. content_block_stop emits a `compaction_block`
tool_event carrying the full summary so the chat-adapter can
persist it. compaction_blocks_seen is surfaced in the closing
summary log.
2. Pydantic schema. Added CompactionContentPart with Tag("compaction")
on the ContentPart Union so requests carrying the block parse
cleanly. Required `content` field with a docstring pointing at
the Anthropic docs.
3. Message builder. _build_external_messages forwards compaction
parts on both vision and non-vision paths; the per-provider
stream helper decides whether to forward to the wire (Anthropic
does; other providers ignore the part). When a non-vision route
ends up with a single text part, collapse back to a string
so providers that don't accept content arrays still get the
expected shape.
4. _stream_anthropic outbound translator. {type:"compaction"} parts
on an assistant message land on the wire verbatim. Empty/missing
`content` is skipped so a malformed stored block can't 400
Anthropic.
Tests added (5): stream emits compaction_block tool event with the
summary intact; user-visible content stream does NOT carry the
summary text; outbound body forwards compaction parts verbatim on
the next turn; Pydantic schema accepts the part; builder passes
it through on both vision and non-vision provider routes.
Frontend follow-up: the chat-adapter needs to persist the
compaction_block tool_event onto the stored assistant message so
turn N+1 includes it in payload.messages. Pinned in the PR
description.
Sourcing: https://platform.claude.com/docs/en/build-with-claude/compaction
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: gate compaction-part passthrough to Anthropic only
Codex P1: my previous round-trip change preserved {type:"compaction"}
parts on every provider route in _build_external_messages. That
meant a chat history with prior compaction state silently leaked
the Anthropic-specific block to OpenAI/DeepSeek/Mistral/Gemini/
Kimi/OpenRouter on a provider switch, where generic
/chat/completions passthrough hands the unknown content type to
the upstream API and 400s the whole turn.
Added a `provider_type` kwarg to _build_external_messages and
gated the compaction forwarder on `provider_type == "anthropic"`.
Every other value (including the legacy None for callers that
don't pass it yet) strips the part. The Anthropic stream helper
still maps it to a native `compaction` block on the wire.
Threaded provider_type through from _proxy_to_external_provider's
call site.
Tests updated: vision + provider="anthropic" still forwards; six
non-anthropic providers strip the part; missing provider_type
strips defensively; non-vision + anthropic still forwards; non-vision
+ non-anthropic collapses back to a text string.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: Persist chat history in backend storage
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address chat tombstone batching review
* fix: update desktop auth routes stub
* chat db settings storage
* chat db settings routes
* chat db settings client
* chat db settings store
* chat db settings wiring
* chat db history storage
* chat db settings migration
* chat db settings fallback
* chat db container metadata
* chat db legacy migration fixes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* chat ci auth background reads
* chat auth storage fixes
* chat migration final fixes
* chat export batch message lookup
* chat history review fixes
* chat prune sync fix
* chat settings hydration retry
* gate settings persistence
* Scope chat-history rows by subject; fix hijack, clear-confirm, hydrate race
Backend storage and routes:
- chat_threads / chat_messages / chat_settings carry a NOT NULL subject
column with composite PRIMARY KEY (id, subject). Two authenticated
identities can no longer see or wipe each other's data.
- Pre-existing rows on an existing studio.db migrate under sentinel
subject __legacy_unscoped__ via rename + rebuild + copy; single-user
installs see no behavior change.
- ON CONFLICT(id, subject) DO UPDATE ... WHERE chat_messages.thread_id =
excluded.thread_id refuses cross-thread re-parenting via upsert.
upsert_chat_message + sync_chat_messages now raise
ChatMessageThreadMismatch which the routes map to HTTP 409.
- replace_thread_messages rejects body messages whose threadId does not
match the URL thread (HTTP 400) instead of silently rewriting them.
- DELETE /api/chat requires ?confirm=true, returns row count, logs the
subject and count.
- upsert_chat_settings_merge does read + deep-merge + write inside a
single BEGIN IMMEDIATE so concurrent writers no longer drop each
other's updates. The route delegates to this helper.
- New POST /api/chat/messages:batch returns {thread_id -> messages[]}
for many threads in one HTTP call. Subject-scoped. Unknown ids return
empty lists instead of 404 so the sidebar/search caller can rebuild
atomically.
Frontend:
- chat-runtime-store: hydrate-failure catch sets settingsHydrated:true
so a transient backend blip no longer permanently disables
persistence. setParams bumps inferenceParamMutationVersions
unconditionally so a slow hydration response cannot clobber a
pre-hydrate user edit. saveSettingsPatch replaces the serial chain
with a debounced pendingPatch + deep merge; flush on beforeunload.
- chat-history-storage: clearStoredChats returns ClearStoredChatsResult
distinguishing backend / legacy / both outcomes.
listStoredChatThreadsWithMessages uses the batched fetch (one HTTP
call) instead of Promise.all per-thread; legacy Dexie fallback only
fires when the batch result is empty.
- chat-api: batchListChatMessages with graceful 404 / 405 fallback to
per-thread listChatMessages for older servers.
- chat-thread-tombstones: store {id, deletedAt} tuples with 90-day GC
and a 5000-entry cap so localStorage stays bounded. Back-compat reads
pre-fix plain strings. Adds removeChatThreadTombstones (rollback) and
clearAllChatThreadTombstones (post-legacy-purge clean-up).
- use-chat-sidebar-items: deleteChatItem tombstones synchronously
BEFORE the backend round-trip and rolls back on failure (restores
pre-PR optimistic UX). 300 ms trailing debounce on
CHAT_HISTORY_UPDATED_EVENT plus requestSeq guard so stream-time event
bursts produce at most one fetch per quiet window.
Tests:
- studio/backend/tests/pr5272_sim/ adds 64 regression tests covering
schema migration from pre-fix shape, subject scoping, cross-thread
hijack, bulk-replace mismatch, clear-confirm, concurrent settings,
unicode + 2MB content + SQL-injection-safe binding, chunking
boundary at 900 and 901 ids, batched endpoint (multi-subject + 1200
ids + per-thread order), and grep contracts for the frontend patches.
test_chat_history_storage.py updated to pass subject.
Verified locally on Linux + macOS + Windows GitHub Actions runners
(staging fork): 64 pass + 2 from the PR's own backend test on all
three OSes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop subject scoping and clear-confirm gate (Studio is single-user)
Per maintainer feedback: subject scoping, cross-thread message hijack
guard, and DELETE /api/chat ?confirm=true gate are unnecessary because
Studio is intentionally single-user (the client already shows a confirm
dialog before clear-all).
This commit reverts those backend changes and keeps only the
non-multi-user pieces from the earlier fix commit:
- studio_db.py: restored to pre-fix shape; adds upsert_chat_settings_merge
which does atomic read + deep-merge + write under BEGIN IMMEDIATE so
two concurrent slider drags cannot drop one another's updates.
- routes/chat_history.py: restored; put_settings now calls the atomic
merge instead of doing the read-merge-write across three separate
connections. Adds POST /api/chat/messages:batch to collapse the
sidebar/search rebuild from N round-trips to 1.
- frontend/api/chat-api.ts: align batchListChatMessages request and
response keys with the backend (threadIds / messagesByThreadId).
- tests/test_chat_history_storage.py: add atomic-merge concurrency test,
deep-merge nested-key test, and 901-id chunking-boundary test.
- Drop the pr5272_sim test directory (those tests covered the reverted
subject-scoping/hijack/confirm behavior).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix sidebar delete crash, keepalive on settings beforeunload flush, search rebuild race
Two correctness bugs and one perf race surfaced by a fresh code review of
the prior fix commit:
- chat-api.ts: notifyChatHistoryUpdated was declared as a non-exported
function, but use-chat-sidebar-items.ts imports it. The import would
fail tsc with TS2305 and at runtime the optimistic-delete and
delete-failure rollback paths would both throw.
- chat-runtime-store.ts + chat-settings-api.ts + chat-settings-storage.ts:
the beforeunload settings flush is now actually keepalive. Without it
the browser cancels the in-flight PUT on tab close, so the last slider
drag is silently dropped (which is exactly the case the
debounce+beforeunload combination was meant to protect against).
- use-chat-search-index.ts: rebuilds now coalesce with a 300ms trailing
debounce and discard out-of-order responses via a requestSeq guard.
Matches the sibling pattern in use-chat-sidebar-items.ts so two rapid
CHAT_HISTORY_UPDATED_EVENTs (run-start + run-end save during a turn)
cannot land with stale data winning.
- chat-thread-tombstones.ts: drop dead clearAllChatThreadTombstones with
no call sites; Dexie is never wiped so the function has no use.
* fix(studio): protect chat persistence writes
* fix(studio): align chat history clear semantics
* fix(studio): show partial chat clear feedback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(studio): preserve chat persistence fallbacks
* fix(studio): harden chat thread persistence checks
* Preserve chat message timestamps
* Gate chat stream on history save
* Make chat thread backfill best effort
* Avoid chat message 404 probe
* Tighten chat legacy fallbacks
* chat: server-side ledger so legacy Dexie import is recoverable
The boolean localStorage sentinel
(unsloth_chat_legacy_imported_to_studio_db) made importLegacyChatsIfNeeded
non-recoverable: deleting studio.db while the browser keeps the flag
silently hides every legacy Dexie thread from the sidebar (verified by
the 3-GPU validation probe; matches the third review comment on PR
#5272). Same trap fires for browser-profile sync to a fresh machine
and any other path that wipes studio.db while keeping IndexedDB.
Source of truth moves into studio.db itself via a new
chat_legacy_import_log table keyed by legacy thread id. The ledger
disappears together with studio.db, so the next launch re-runs the
import from whatever Dexie still holds. localStorage stays as a
per-session perf hint only.
Performance, all bounded by the three new fast-paths before any
backend work:
A) localStorage hint says "imported earlier in this session" -- 0
network, ~0 ms. Covers the warm sidebar mount.
B) indexedDB.databases() reports no "unsloth-chat" DB -- 0 network,
~1 ms. Covers every new user who never had the old browser-only
Studio (the common case after launch).
C) db.threads.count() + db.messages.count() are both 0 -- 0 network,
~5 ms. Covers returning users who migrated long ago and Dexie was
never repopulated.
Only when all three miss does the code talk to the backend
(GET /api/chat/import-ledger -> diff vs Dexie -> existing import path
-> POST /api/chat/import-ledger to record what was just imported).
Per-thread tracking is enough because Dexie is read-only after this
PR; a thread's message set does not grow.
Backend deployments that predate the import-ledger routes are
handled transparently: the client treats 404/405 as an empty ledger
and re-runs the (idempotent via UPSERT) import on next launch.
Changes:
- storage/studio_db.py: new chat_legacy_import_log table (WITHOUT
ROWID, PK on legacy_thread_id) + list_chat_legacy_import_log() +
record_chat_legacy_import_log() (idempotent batch UPSERT).
- routes/chat_history.py: GET + POST /api/chat/import-ledger with the
obvious request/response models.
- frontend api/chat-api.ts: listChatImportLedger() (returns a Set for
O(1) diff) + recordChatImportLedger(), both with 404/405 fallback.
- frontend utils/chat-history-storage.ts: importLegacyChatsIfNeeded
gains three fast-paths, ledger fetch on the slow path, and writes
the ledger after a successful import. The localStorage helper is
unchanged on the surface; it just stops being authoritative.
- tests: 5 new test_legacy_import_log_* cases (empty default, record
+ list round-trip, idempotency, input dedup, empty/null ignore).
All 9 pre-existing tests still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Make the legacy-import recovery actually recoverable
The previous commit added a server-side ledger to make Dexie -> studio.db
import recoverable after a studio.db wipe, but the localStorage perf hint
still short-circuited the import gate before the ledger was ever consulted.
After a wipe, the hint stayed "true" and the bulk re-import never ran -- the
ledger sat empty and only the per-thread lazy materialize-on-continue path
restored data.
Changes:
- Remove the localStorage short-circuit from importLegacyChatsIfNeeded so
the ledger is checked on every fresh tab. legacyChatImportPromise keeps
the per-session cache; the hint now only matters for the listing paths.
- Batch the slow path: one db.messages.where().anyOf().toArray() and one
batchListChatMessages() instead of 2N round-trips. At 1k threads this
drops a multi-second blocking import to a single request pair.
- recordChatImportLedger returns {accepted, inserted, supported}. The
localStorage hint is only flipped when supported is true, so old
backends (404 / 405 / 501) no longer permanently poison recovery.
- Ledger backfill: threads already present in chat_threads but missing
from the ledger now get added too, so old-FE-then-new-FE deployments
don't redo the diff every launch.
- Backend response field renamed recorded -> {accepted, inserted}.
accepted is the deduped non-empty input count; inserted is the rows
actually new (via INSERT ... RETURNING). Bounded by Field(max_length=
10_000) on the request payload.
- Storage helpers renamed: chat_legacy_import_log -> chat_legacy_imports,
record_* -> upsert_* to match the existing noun/verb conventions.
- DEXIE_DB_NAME exported from db.ts; duplicate constant in
chat-history-storage.ts removed.
- 3 new route-level tests for /api/chat/import-ledger covering the
round-trip, the (accepted, inserted) split, and the 10k payload cap.
All 18 chat-history tests pass.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: shine1i <wasimysdev@gmail.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: wire Anthropic web_fetch server-side tool
Studio's Anthropic passthrough only forwarded web_search and
code_execution when enabled_tools was set. Asking Claude through Studio
to fetch a URL produced no fetch (the tool was not in the outbound
tools array), so users had to fall back to web_search even when they
already had the exact URL they wanted.
This change opts in web_fetch_20250910 when enabled_tools contains
"web_fetch". The new tool entry is appended alongside any existing
web_search / code_execution entries:
{"type": "web_fetch_20250910", "name": "web_fetch", "max_uses": 5}
No anthropic-beta header is required (web_fetch is GA); the existing
code-execution-2025-08-25 flag continues to merge cleanly when both
tools are enabled in the same turn.
SSE translation mirrors the web_search path. A `server_tool_use` block
with name="web_fetch" emits a `tool_start` _toolEvent carrying the
URL the model asked to fetch; the matching `web_fetch_tool_result`
block emits a `tool_end` _toolEvent whose result string follows the
Title / URL / Snippet shape parseSourcesFromResult on the frontend
already expects, so the source pill renders identically. Error blocks
(`web_fetch_tool_error`) are surfaced as "Error: <error_code>" matching
the code_execution error path.
The final "Anthropic stream complete" log line picks up web_fetch_
requested / web_fetch_invocations / web_fetch_urls so support reports
of "the model did not fetch anything" can be triaged from the log.
Verified end to end against claude-haiku-4-5 with
`enabled_tools=["web_fetch"]`: the model emitted tool_start with
url=https://example.com and tool_end with the page Title + URL +
Snippet, plus the assistant message correctly read back "Example
Domain" as the title.
Tests:
- 5 new unit tests in test_anthropic_web_fetch.py covering tool
registration, the combined web_search + web_fetch + code_execution
request body, the pill-off case, and SSE translation for both
success and error paths.
- All 242 existing Anthropic + OpenAI provider tests still pass.
The enabled_tools field description in models/inference.py is updated
so OpenAPI consumers see the new option.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* web_fetch: title fallback to URL, log parse failures, drop dead checks
Three review nits on the previous commit:
1. `_format_web_fetch_result` left `title` empty when Anthropic omitted
`document.title`. The frontend `parseSourcesFromResult` only emits
a source pill when both `Title:` and `URL:` lines are present, so
fetches against pages without an HTML title tag silently lost
their citation in the UI. Fall back to `title = title or url`,
matching the web_search formatter.
2. The broad `except Exception` around `json.loads(buffer)` for the
web_fetch input swallowed the failure with no trace. Log at debug
so a malformed partial_json buffer can be triaged from the server
log without changing behavior.
3. `inner` was already sanitised to a dict at the matching
content_block_start and `_format_web_fetch_result` always returns
a non-empty string (defaulting to "(fetch complete)"), so the
`isinstance(inner, dict) else {}` guard and the
`result_text or "(fetch complete)"` fallback at the emit site
were dead code. Removed.
Added a test exercising the titleless path so the fallback stays
covered.
* chat-adapter: emit source pills for web_fetch tool calls
`parseSourcesFromResult` was only wired up for tool calls where
`toolName === "web_search"`, so the Title / URL / Snippet block the
backend formatter emits for `web_fetch_tool_result` never reached the
source-pill renderer. Users saw the raw tool result in the tool card
but the dedicated source-pill row at the message tail stayed empty.
Both web_search and web_fetch ship the same text shape today, so the
fix is to broaden the gate.
* Address review: wire web_fetch from Search pill + fix pause_turn truncation
Two reviewer follow-ups on the Anthropic web_fetch PR:
1. The backend tool wiring landed but the frontend chat-adapter
never put `web_fetch` in `enabled_tools`, so toggling the Search
pill only ever attached `web_search` -- web_fetch was unreachable
from the UI. Added providerSupportsBuiltinWebFetch() (Anthropic
today) and paired the entry with the existing Search pill, since
the canonical workflow is "search returns URLs, fetch reads
them" and there is no separate UI toggle yet.
2. `pause_turn` from Anthropic's stop_reason vocabulary fell through
the finish_reason map's "stop" default, which the OpenAI-format
client renders as end-of-message and truncates the answer. Per
the docs pause_turn means "Claude paused a long server-tool
turn (web_search / web_fetch) and will resume". Mapped to None
and skipped the chunk emission so the SSE stream still ends with
[DONE] on message_stop but no terminal finish_reason lands on
the client. While there: added explicit mappings for `tool_use`
(-> tool_calls) and `refusal` (-> content_filter) which were
also falling through to "stop".
Tests added: pause_turn emits no finish_reason, end_turn still
emits "stop", refusal maps to "content_filter".
Sourcing: https://platform.claude.com/docs/en/api/messages#response-stop-reason
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: per-session cost calculator + /api/providers/pricing endpoint
Neither the Anthropic Messages API nor the OpenAI Responses API
reports a `cost` field on the response. Both expose detailed token
counts (input, output, cache hits, server-tool invocations); pricing
multipliers live in the provider docs. The frontend's "cost so far"
display was impossible without scraping the server log.
Land the math + a snapshot endpoint so the cost calculator can run
client-side from the existing usage chunk plumbing. The actual UI
hookup belongs in a frontend follow-up (and is gated on PR #5670's
usage-chunk emission landing so the frontend sees the usage block
in the first place).
Changes:
- New `core/inference/pricing.py` with:
- Per-MTok base pricing tables for every active Anthropic and
gpt-5.x family member. Dated snapshots inherit the canonical-id
price via prefix match so future snapshots cost the same as the
canonical id until pricing changes.
- Shared multipliers for Anthropic cache writes (5m: 1.25x, 1h: 2x)
and reads (0.1x); OpenAI cache reads (0.1x); Anthropic server
tool surcharges ($10 / 1k web_search, $0.05 / hour code_exec
beyond the 50-hour daily free tier).
- `calculate_cost(provider, model, usage)` returns a per-turn USD
breakdown plus billable token counts, with priced=False for
unknown models so the UI can still render token counts.
- `pricing_snapshot()` returns the whole table for the frontend
so it doesn't re-implement the multipliers.
- New `GET /api/providers/pricing` returning the snapshot, scoped
behind the existing auth dependency.
- New `backend/tests/test_pricing.py` with 12 cases pinning the
math against documented values: base input/output multiplication,
5m / 1h / read multipliers, default-to-5m fallback when the
breakdown is absent, web_search per-1k pricing, code_execution
per-hour pricing, dated-snapshot fallback, OpenAI cache-read
discount accounting (cached tokens subtracted from full-price
bucket and re-billed at 0.1x), unknown model graceful-degrade,
and the snapshot endpoint shape.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: verified OpenAI pricing + fix billable input double-count
Address the cost-calculator review:
- OpenAI prices were 2-6x under the actual published rates.
Cross-checked the live developers.openai.com/api/docs/pricing page
and replaced every entry. gpt-5.5 is 5/30, gpt-5.5-pro is 30/180,
gpt-5.4 is 2.5/15, gpt-5.4-mini 0.75/4.5, gpt-5.4-nano 0.20/1.25,
gpt-5.3-codex 1.75/14. Added chat-latest alias to the canonical
chat-snapshot rate. Dropped o3 / o4 / gpt-4.5 rows that are no
longer listed on the page; calculator returns priced=False instead
of silently billing at zero.
- billable_input_tokens was double-counting cached tokens for
OpenAI. Anthropic excludes cache_* buckets from input_tokens so
we add them; OpenAI folds cache_read_input_tokens into
input_tokens already, so the tooltip read 1.8M for a 1.0M bill.
Branched the math by provider and added a regression test.
Sourcing notes in the module docstring updated.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: canonical 4.5 ids, long-context tier, OpenAI tool fees
Three Codex P1 follow-ups on the cost calculator:
1. Canonical Anthropic 4.5 ids missing from ANTHROPIC_PRICING.
claude-opus-4-5 / claude-sonnet-4-5 / claude-haiku-4-5 (no date
suffix) are the ids used by backend defaults
(PROVIDER_REGISTRY['anthropic'].default_models), but the table
only had the dated forms. _lookup's prefix fallback doesn't help
because the canonical id is SHORTER than the dated key, so
str.startswith goes the wrong way and the calculator returned
priced=False + zero cost. Added the canonical aliases for
opus-4-5, sonnet-4-5, haiku-4-5, and opus-4-1.
2. OpenAI long-context tier. gpt-5.5 and gpt-5.4 cross over at
272k input tokens to a 2x input / 1.5x output rate (gpt-5.5:
$5/$30 -> $10/$45; gpt-5.4: $2.50/$15 -> $5/$22.50). Turns past
the threshold were systematically undercounted at headline
rates. Added long_context_threshold / long_context_input_per_mtok /
long_context_output_per_mtok columns and a tier-selection step
in calculate_cost; model_priced gains a "(long-context >272000)"
suffix when the higher tier applies so the tooltip can show
which rate was used. gpt-5.5-pro / gpt-5.4-pro / mini / nano /
codex have no published long-context tier today, so they keep a
single rate.
3. OpenAI server-tool surcharges. web_search is $10/1000 calls and
the hosted shell container is $0.03 per 20-minute session on the
default 1g tier (~$0.09/hr). server_tools_usd was previously
stuck at 0.0 for OpenAI even when web_search and shell tools
fired, so sessions with tool use understated cost. Added
OPENAI_WEB_SEARCH_USD_PER_1K and OPENAI_CONTAINER_USD_PER_HOUR
constants plus a parallel of the Anthropic surcharge block that
reads counts from usage["openai_tool_use"]. The SSE translator
wires the counts in a follow-up commit; the calculator is now
ready for them. pricing_snapshot also exposes both constants so
the frontend tooltip can render the per-call rate.
Existing tests updated to stay in the short-context tier where they
were testing base rates; new tests pin canonical 4.5 lookups,
long-context crossover on gpt-5.5/gpt-5.4, the absence of crossover
on mini/nano/codex, and OpenAI tool surcharges (web_search,
container hours, combined total).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: wire OpenAI image_generation tool
OpenAI's Responses API exposes server-side image generation as a
tool entry (`{type: "image_generation"}`); the result comes back as
an `image_generation_call` output item with the base64 image on
`result`, the actual prompt used on `revised_prompt`, plus `size`,
`quality`, `output_format`, `background`. The model decides when to
call the tool based on the user's request; rendering uses one of
the gpt-image-* backbones server-side.
Available on every gpt-5.x family member plus gpt-4.1, gpt-4o, o3,
o4-mini per the docs.
Changes:
- Append `{type:"image_generation"}` to the Responses request tools
array when `enabled_tools` carries `image_generation` AND the base
URL points at cloud OpenAI. Non-cloud bases (ollama, llama.cpp,
"custom" presets that collapse to provider="openai") silently drop
the tool to avoid 400s.
- Mirror the same logic in `_build_body` (the post-expiry retry
builder) so retries carry the same tool set as the original
attempt.
- Handle `image_generation_call` items in
`response.output_item.done`: emit `tool_start` with
`arguments:{kind:"image", prompt:<revised_prompt>}` and `tool_end`
with `image_b64`, `image_mime`, `size`, `quality`, `background`
so the chat adapter can render an inline preview. Image bytes go
on the tool_end chunk; no extra fields on the chat-completions
envelope so the OpenAI SDK shape stays clean.
- Add `import time` (used for synthesised tool_call_id fallback).
- Add `test_openai_image_generation.py` with 5 cases: tool entry on
cloud OpenAI, combined with web_search + code_execution
(verifies all three coexist), non-cloud drop, omitted pill leaves
body untouched, output item translation produces the expected
tool_start + tool_end chunks.
Live verified end-to-end: `gpt-5.4-mini` with `image_generation`
tool returned an `image_generation_call` carrying ~1MB of base64
PNG plus the gpt-image backbone's revised prompt.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use time.time_ns() for synthesised image_generation tool_call_id
Gemini medium on PR #5688: `int(time.time() * 1000)` has 1ms
resolution; two image generations resolving in the same millisecond
would collide on the synthesised id. Bump to nanoseconds.
(In practice the upstream `image_generation_call` item always carries
its own `id`; the synthesised fallback only fires when OpenAI omits
it -- rare, but cheap to harden.)
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: support Anthropic 1h cache TTL via prompt_cache_ttl field
Anthropic exposes two ephemeral cache pools per request: the default
5-minute pool, and a 1-hour pool selected by attaching `ttl:"1h"` to
the `cache_control` marker. 1h writes are billed at 2x base input vs
1.25x for 5m, but reads stay at 0.1x for both, so a single extra read
landing more than 5 minutes after the write pays off the premium.
Studio hardcoded the 5m pool via `cache_control: {type:"ephemeral"}`
on both breakpoints. For chats with multi-minute idle gaps (people
juggling tabs, long-running tool calls between turns), the cache
expires before the next turn and every read becomes a cache_creation,
not a cache_read -- exactly the case where the 1h pool wins.
Changes:
- Add `prompt_cache_ttl: Optional[Literal["5m", "1h"]]` to
ChatCompletionRequest. Default (None) preserves today's 5m behavior.
- Thread through `routes/inference.py` ->
`stream_chat_completion` -> `_stream_anthropic`.
- Build a shared `cache_marker` dict in `_stream_anthropic`; attach
`ttl` only when the request asks for one of the two valid values.
Unknown TTL strings are silently dropped to avoid sending malformed
markers (the upstream API would 400).
- Apply the same marker to both existing breakpoints (system block at
line 1175 and the latest-message tail at line 1198 / 1213) so the
pool selection is consistent across the whole prefix.
- Add `test_anthropic_cache_ttl.py` with 11 parametrized cases
pinning the outbound body shape: omitted -> default marker;
explicit `5m`/`1h` -> ttl field set; unknown values dropped;
caching off -> no markers at all.
Verified upstream that `cache_control: {type:"ephemeral", ttl:"1h"}`
is accepted by the Anthropic API today; no beta header required.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Relax prompt_cache_ttl to Optional[str] (Codex P1)
Declaring `prompt_cache_ttl` as `Optional[Literal["5m", "1h"]]` made
FastAPI/Pydantic 422 the request before _stream_anthropic could even
see the field. The whole point of the downstream drop-unknown-values
behaviour was to keep a stale frontend from crashing the request;
the strict Literal at the request layer defeated that.
Loosen the schema to Optional[str]; the existing in-helper guard
already restricts forwarded values to {"5m", "1h"} (everything else
is silently dropped). Test suite stays unchanged -- the bogus-value
cases in test_anthropic_cache_ttl.py already pass arbitrary strings
through and assert they are dropped before the wire.
* Address review: confirm extended-cache-ttl beta header is GA
Reviewer asked whether the 1h cache TTL still requires the
`extended-cache-ttl-2025-04-11` anthropic-beta header. Investigated:
- Live-tested api.anthropic.com on claude-opus-4-7 (2026-05-22)
with cache_control={type:"ephemeral", ttl:"1h"} and NO beta
header. Got status 200 and ephemeral_1h_input_tokens populated
on the create turn, plus cache_read_input_tokens populated on
the reuse turn.
- Cross-checked the current prompt-caching docs: no mention of
any beta header on the 1h TTL path.
Conclusion: the gate has been promoted to GA. The code already
does not send the beta header (the cache_marker dict only carries
`type`/`ttl`), so no wire change is needed. Pinned the contract
with two regression tests that assert the header is NOT on the
outbound request, and added a docstring note explaining the
investigation outcome so a future reader does not re-add it.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: per-model Anthropic server-side tool versions
Anthropic ships date-pinned tool versions per model family. Studio
currently hard-codes `web_search_20250305`, `web_fetch_20250910`, and
`code_execution_20250825` for every model, which means Opus 4.6/4.7,
Sonnet 4.6 and the Opus/Sonnet 4.5 family never get the newer
`_20260209` / `_20260120` variants. Those newer variants add dynamic
filtering (Claude writes code to rank/filter web results before they
enter context) and REPL state persistence + programmatic tool calling
inside the sandbox, which is what the user-facing pills are supposed
to expose.
Hardcoding the legacy versions also breaks if a future model family
drops the legacy types: the request 400s instead of falling back.
Changes:
- Add `_anthropic_web_search_version`, `_anthropic_web_fetch_version`,
`_anthropic_code_execution_version` helpers that pick the newest
variant the model accepts and fall back to the GA versions for
everything else.
- Add `_ANTHROPIC_CODE_EXECUTION_BETA` constant since the beta header
(`code-execution-2025-08-25`) is shared across both code-execution
date variants per the upstream docs.
- Wire the helpers into `_stream_anthropic` so the outbound body
carries the right pinned version per request.
- Add parametrized dispatch tests in
`test_anthropic_tool_versions.py` covering Opus 4.7/4.6/4.5,
Sonnet 4.6/4.5, Haiku 4.5, Opus 4.1/4.0, Sonnet 4.0, 3.5 Sonnet,
plus streaming integration tests that verify the outbound body
uses the right versions on Opus 4.7 (new web_search + new
code_execution), Haiku 4.5 (legacy both), and Sonnet 4.5 (legacy
web_search + new code_execution).
- Update existing `test_anthropic_code_execution.py` cases that
pinned the old version on Opus 4.7 to expect the new ones.
Verified end-to-end against the live Anthropic API: Opus 4.7 with
both pills enabled accepts the newer-pinned tools without a 400, and
Haiku 4.5 still works on the legacy fallback path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Daniel Han-Chen