Four valid review comments from gemini-code-assist[bot] on #5759:
1. core/rag/bm25.py:_load — wrap bm25s.BM25.load + json.loads with
specific exception handlers (FileNotFoundError, OSError,
JSONDecodeError, ValueError) and log a warning instead of
propagating a 500. Corrupt/partial bm25 dirs now degrade to
empty-search rather than crashing the request.
2. core/rag/tool.py was importing _resolve_scope_embedder from
routes/rag.py — a layering violation (core depending on
routes). Move the resolver into a new core/rag/scope.py module
along with the chat-settings key constants; routes/rag.py
now re-imports it under the same name. Same behaviour, no
cycle, one source of truth for the resolution logic.
3. core/rag/bm25.py:rebuild_index — call delete_scope before saving
the new index so stale files from a previous build (or a
bm25s naming change) never coexist with current files. The
library's save() doesn't unlink files it doesn't write.
4. routes/rag.py:_save_upload was running f.write() synchronously
inside an async def. Switch to anyio.open_file() so each chunk
write runs in a worker thread instead of blocking the event
loop on multi-MB uploads. Cleanup unlink happens after the
async-with closes the handle so Windows is happy.
Skipped one (vector_store.py:133 'hasattr query_points' redundancy)
— that comment was on the pre-rewrite Qdrant code; the file is
now sqlite-vec backed and the hasattr check is gone.
core/rag/db.py, vector_store.py, tool.py, bm25.py, and reranker.py
all run only in the FastAPI parent process. Switch their loggers
from Python stdlib to studio's structlog get_logger so their output
shows up in the same JSON stream as the rest of the backend (the
request_completed / RAG search lines).
embeddings.py and ingestion.py stay on stdlib because they execute
inside the mp.spawn ingestion subprocess, which doesn't inherit the
parent's structlog configuration.
asg017/sqlite-vec is Apache-2.0 and OSI-approved. Replaces
qdrant-client (~30 MB) with a small SQLite extension loaded into a
dedicated rag.db file. Single file holds RAG vectors; bm25s indexes
and chat-side studio.db are unaffected.
- New core/rag/db.py owns the rag.db connection and sqlite-vec load.
Extension load runs once at first open. Process-wide singleton
protected by a lock; check_same_thread=False + WAL handles the
FastAPI thread pool.
- core/rag/vector_store.py keeps the same public API
(ensure_collection / upsert_chunks / search / collection_exists /
delete_scope / delete_document) so callers in routes/rag.py,
core/rag/ingestion.py, core/rag/tool.py, and core/rag/retrieval.py
don't change. ensure_collection is now a no-op; collection_exists
returns True iff the scope has at least one indexed vector.
- search uses sqlite-vec's vec_distance_cosine and converts distance
to similarity in [0, 1] so the per-scope min_score threshold
semantics stay identical.
- Mixed-dim scopes coexist behind WHERE scope = ? — the per-scope
embedder resolver guarantees one embedder per scope.
- requirements/rag.txt swaps qdrant-client for sqlite-vec.
- utils/paths/storage_roots.py drops rag_vectordb_root() (the old
qdrant directory); rag.db lives directly under rag_root().
- Rewritten tests/python/test_rag_vector_store.py for the new
semantics (collection_exists tracks populated scopes; new tests
for filtered search and upsert conflict resolution).
Python build requirement: connection.enable_load_extension(True)
must be available. install.sh creates the venv via uv-managed
python-build-standalone, which is compiled with
--enable-loadable-sqlite-extensions, so this works on standard
installs. core/rag/db.py raises an actionable error on the rare
custom-interpreter case.
The query was always going through the default text embedder
(bge-small, 384-d) regardless of how the scope was ingested. A
multimodal thread indexed by Qwen3-VL (2048-d) crashed at cosine
similarity with 'shapes (104,2048) and (384,) not aligned'.
Resolve the scope's embedder at search time:
- kb_<id> -> rag_knowledge_bases.embedding_model column
- thread_<id> -> thread settings (with fallback to defaults +
RAG_EMBEDDER_MATRIX matrix lookup)
Pass it through retrieve_hybrid/retrieve_dense to embeddings.encode
so the query lands in the same vector space as the docs. Both the
/api/rag/search route and the search_knowledge_base tool use the
same resolver.
Qwen3-VL-Embedding-2B's modules.json references
sentence_transformers.base.modules, introduced after 5.2.0.
Unlike BGE-VL, Qwen3-VL has no fragile custom forward subclass to
break against the newer ST internals, so the bump should land
cleanly for the multimodal path.
Built on Qwen3 (not CLIP), so the 77-token text cap that bit
BGE-VL-base is gone — long chunks embed losslessly. Loads via
vanilla SentenceTransformer with trust_remote_code, no custom
adapter needed. 2048-d shared text+image space.
Adds qwen-vl-utils>=0.0.14 to rag.txt for image preprocessing,
required by the Qwen3-VL embedder family.
BGE-VL inherits CLIP's 77-token text positional embedding table —
longer chunks crash inside the text model with a shape mismatch.
Pre-tokenize with truncation=True, max_length=77 and call
get_text_features directly so the high-level encode() (which does
not truncate) is bypassed. Log when truncation happens — text
chunks beyond the cap are silently cut, so multimodal mode is
lossy on the text channel. Image channel is unaffected.
BGE-VL's sentence-transformers shim (bge_vl_clip_transformer.py)
is coupled to specific ST internals and broke across both the 5.2
and 5.3 pin attempts. The canonical load path documented at
bge-model.com is transformers.AutoModel with trust_remote_code +
model.set_processor() + model.encode(text=/images=).
- Wrap that path in _BGEVLAdapter exposing the slice of
SentenceTransformer API the ingester uses (encode for text or PIL
images, get_sentence_embedding_dimension, best-effort tokenize).
- Restore BAAI/BGE-VL-base in RAG_EMBEDDER_MATRIX.
- Revert sentence_transformers pin to 5.2.0 — no longer relevant
since the multimodal path no longer touches ST.
BGE-VL-base ships a custom Transformer subclass that's tightly
coupled to a specific sentence-transformers internal API — it
imports a module path missing in older ST and overrides forward()
expecting a return shape that changed in newer ST. We can't pin ST
to BGE-VL's exact version without risking the training/inference
paths that use the same library, so swap the multimodal default to
the canonical clip-ViT-B-32 which sentence-transformers wraps
natively (no trust_remote_code, no shim, ST-version-independent).
Same 512-d shared text+image space.
BGE-VL's custom modeling code imports
sentence_transformers.base.modules.transformer, which doesn't exist
in the 5.2.0 pin. Bump to >=5.3.0,<7 so the new Module package is
available.
BGE-VL (multimodal) and nomic-embed-text-v1.5 (late chunking) both
ship custom modeling code in their model repos; sentence-transformers
refuses to import the referenced module (e.g. bge_vl_clip_transformer)
without trust_remote_code=True. Safe to enable because the embedder
matrix is config-pinned — users don't supply arbitrary names.
client.search() was removed/renamed to query_points() in
qdrant-client 1.10+. Use the new API when available and fall back
to search() for older pinned versions.
- chat-adapter pre-fetches retrieval on every turn when RAG is on,
regardless of provider. Users no longer have to phrase queries as
'the document I attached' for retrieval to fire. Local tool models
still get search_knowledge_base registered as a refinement path.
- New per-thread ragMinScore slider (Min relevance, 0..1) gates
retrieved hits by dense cosine similarity. Hits below the floor
(and BM25-only hits with no dense signal) are dropped server-side
so unrelated docs don't get injected when the user's query is
off-topic from what's indexed.
- Backend logs at search start (scope, top_k, min_score, query
preview), after retrieval (retrieved vs met_threshold counts),
and on return (final hit count) for both /api/rag/search and the
search_knowledge_base tool path.
- System-prompt nudge prepended when pre-fetch returns hits so the
model knows to cite [1], [2] rather than paraphrase silently.
no-torch-runtime.txt is only consumed in NO_TORCH (Intel Mac
GGUF-only) mode, so qdrant-client / bm25s / pymupdf etc. were never
installed in the normal install path. Split them into rag.txt and
add a step to install_python_stack.py that installs it after studio
deps, skipped only when NO_TORCH is set.
When ragToolEnabled is on, the in-thread composer's + button now
opens a doc-only picker and uploads selected files to the per-thread
RAG pipeline (matching SharedComposer). Pending chips show
Uploading -> Indexing -> Ready status; Send is blocked while any
doc is in-flight. Previously these files were base64'd inline as
native attachments, which bypassed RAG entirely.
The Phase 4 RAG button only existed in shared-composer; the
assistant-ui in-thread composer renders its own pill set, so RAG
was missing once a thread had messages.
Fresh [] literals from useRagStore selectors triggered Zustand's
Object.is snapshot check on every render, causing React #185 on the
chat page when documentsByScope[key] was unpopulated.
Two useEffects added in Phase 2C and Phase 4 followed the anti-pattern
of calling a state setter inside the effect and listing the setter's
output in the dep array. Both could re-fire indefinitely when the
selected slice changed shape on each render — which the chat page hit
on first load.
chat-settings-sheet.tsx (thread settings loader)
- Before: `useEffect(load, [..., threadSettings, ...])` with
`if (!threadSettings) load()` inside. After load, the Zustand
selector returned a freshly-constructed slice, dep changed,
effect re-ran. If anything in between caused threadSettings to
briefly flicker undefined (e.g. a race during initial hydration
or a fast subsequent thread switch), the load fired again and the
cycle repeated.
- After: ref-guarded by activeThreadId — `threadSettingsLoadedRef`
tracks which threadId has been loaded; the effect deps shrink to
`[ragSource.kind, activeThreadId, loadThreadSettings]`, all
stable per-thread, removing the feedback loop.
ingestion-toast-stack.tsx (terminal-job auto-dismiss)
- Before: `useEffect(..., [jobs, dismissedJobs])` with
`setDismissedJobs(prev => new Set(prev).add(jobId))` inside the
scheduled setTimeout. Each setter creates a new Set reference;
the dep change re-triggers the effect, which clears and
reschedules timers. Under fast SSE event arrival or a strict-mode
double-mount, the scheduler runs faster than its cleanup and
React caps the depth.
- After: dismissedJobs is read via a ref (kept in sync at the top
of the component); the effect only depends on `[jobs]`. A
`scheduledJobsRef` prevents duplicate timer scheduling for the
same job across multiple effect runs, and the setDismissedJobs
updater no-ops when the job is already dismissed.
No behavior change for the happy path — toasts still auto-dismiss
after DISMISS_DELAY_MS; thread settings still load on first sight
of a thread.
Two errors surfaced by the frontend build (`tsc -b`) after the Phase 4
commit c74fc13eb landed:
src/features/chat/chat-settings-sheet.tsx(449,9): TS2451 — cannot
redeclare block-scoped 'activeThreadId'.
src/features/rag/stores/rag-store.ts(326,9): TS2774 — condition will
always return true since this function is always defined.
Fixes
- chat-settings-sheet.tsx: an `activeThreadId` declaration already
existed near the code-exec section (line 636). Phase 2B's RAG
retrieval-section block introduced a second declaration at line 449.
The earlier one is needed for the Retrieval block; drop the later
redeclaration — downstream code still resolves it via lexical scope.
- rag-store.ts subscribeJob: `get().jobUnsubscribers[jobId]` indexes a
`Record<string, () => void>`. Without `noUncheckedIndexedAccess` TS
infers the result as the function type (never undefined), so
`if (existing)` is always-truthy. Replace with `if (jobId in
get().jobUnsubscribers) return;` — same semantics, satisfies TS.
Promotes RAG to a first-class composer toggle alongside Think / Web
Search / Code, with tool-use semantics on local models that support
tools and a pre-fetch fallback on external providers. The model
decides when to call `search_knowledge_base` on local inference; on
external providers retrieval still fires before each message (the
existing pre-fetch path), gated on the same button.
Backend
- core/rag/tool.py (new): search_knowledge_base handler + JSON-schema
tool spec. Resolves scope (kb_id wins over thread_id) from the
request's rag_scope, runs retrieve_hybrid + optional rerank, then
hydrates filename / page_number / text from sqlite and formats as
numbered Markdown citations ('[1] file.pdf (page 5): ...') for the
LLM to cite. Empty scope returns a user-facing hint; empty results
return a clear no-match message instead of an empty string.
- core/inference/tools.py: SEARCH_KNOWLEDGE_BASE_TOOL added to
ALL_TOOLS (lazy import keeps tools.py importable on inference
paths that never touch RAG). execute_tool() gains a tool_context
parameter that carries per-request extras the LLM doesn't see
(currently just rag_scope). The new 'search_knowledge_base' branch
dispatches to the handler with scope unpacked from tool_context.
- core/inference/llama_cpp.py + safetensors_agentic.py +
orchestrator.py: thread tool_context through generate_chat_completion_
with_tools / run_safetensors_tool_loop / execute_tool. Both local
backends (GGUF llama-server and safetensors agentic) carry the same
context object.
- models/inference.py: ChatCompletionRequest gains optional
rag_scope: dict ({kb_id?, thread_id?, enable_rerank?, default_top_k?,
reranker_model?}). Ignored unless 'search_knowledge_base' is in
enabled_tools.
- routes/inference.py: both the GGUF and safetensors call sites for
generate_chat_completion_with_tools forward payload.rag_scope into
tool_context.
Frontend
- chat-runtime-store.ts: global ragToolEnabled boolean + setter +
CHAT_RAG_TOOL_ENABLED_KEY localStorage, mirroring toolsEnabled /
codeToolsEnabled. Settings-hydration migration auto-flips
ragToolEnabled=true for pre-Phase-4 users who already had ragSource
set, so existing RAG users don't silently lose retrieval on upgrade.
- shared-composer.tsx: new 'RAG' pill button after Images (uses
lucide BookOpenIcon, composer-pill-btn style, data-active toggle).
Disabled when no model is loaded. Toggling on from ragSource='off'
auto-flips source to 'thread' so the sidebar lands ready-to-go.
- chat-adapter.ts:
* The existing pre-fetch block is now gated on ragToolEnabled AND
only fires when the tool path isn't viable (external provider OR
local model without tool-use support). Tool-capable local models
skip pre-fetch and let the LLM decide.
* The local-model body assembly adds 'search_knowledge_base' to
enabled_tools and packs ragSource + enableRerank + ragTopK into a
rag_scope object the backend tool handler consumes.
- chat-settings-sheet.tsx: entire Retrieval CollapsibleSection is
wrapped in {ragToolEnabled && ...} so it hides when the button is
off — the button is now the single on/off control. The 'Off'
option is removed from the Source dropdown (the button handles
that). Default open when shown so settings are one click away.
Tests
- test_rag_tool_handler.py: handler covers empty query, missing
scope, kb_id > thread_id precedence, thread-only path, citation
formatting (numbered + page numbers + unknown source); tool spec
shape (function/name/required); execute_tool dispatch with and
without tool_context; ALL_TOOLS includes the new spec without
dropping the existing ones.
Verification scope
- Local GGUF with tools: toggle button on, upload doc, ask about
doc content → assistant emits a search_knowledge_base tool call
card (rendered by the existing ToolFallback component since no
custom UI exists yet — that's a v2 nice-to-have).
- External provider (Anthropic / OpenAI / etc.): same button, same
UX, but uses the pre-fetch path under the hood.
- Migration: pre-existing ragSource != off → button initializes ON
so retrieval keeps working.
@pytest.mark.server integration coverage that drives the ingestion
subprocess in-process and asserts the full multimodal pipeline
produces image + caption chunks with proper kind / image_path /
pair_group metadata. Default pytest runs skip; explicitly:
pytest -m server tests/python/test_rag_multimodal_integration.py
Generates a small PDF (text + PNG figure + caption paragraph) via
pymupdf, points UNSLOTH_STUDIO_HOME at tmp_path, resets the embedder
singleton, and calls _subprocess_worker with mode='multimodal'.
Asserts:
- at least one chunk of each kind (text / image / caption) is emitted
- image chunks carry a real on-disk image_path under tmp_path
- image + caption chunks share a pair_group
A second test confirms BGE-VL produces text and image vectors of the
same dimension — sanity-check for the shared-space assumption that
the multimodal retrieval path relies on.
Downloads BGE-VL-base (~600 MB) on first run, so the test is gated
behind the existing server marker rather than the default suite.
Threads can now opt into late chunking or multimodal mode independently
of the KBs they reference. Per-thread settings persist in chat_settings
under thread:<id>:rag and fall back to the app-level defaults when the
thread hasn't set anything explicitly.
Backend (routes/rag.py)
- ThreadRagSettings / UpdateThreadRagSettingsRequest Pydantic models.
- GET/PUT /api/rag/threads/{thread_id}/settings backed by
chat_settings (upsert_chat_settings_merge). Same (multimodal, late)
constraint enforcement as the create + defaults endpoints.
- POST /api/rag/threads/{thread_id}/reingest now accepts the same
body shape — if any field is set, the new settings are persisted
via set_thread_rag_settings BEFORE the reingest, so subsequent
uploads pick up the change too.
- upload_thread_document reads the per-thread settings and passes
them through to _start_ingestion, replacing the previous hard-coded
('standard', 'text', RAG_EMBEDDING_MODEL) defaults.
Frontend
- rag-api.ts: ThreadRagSettings type + getThreadRagSettings /
setThreadRagSettings wrappers. reingestThreadDocuments now accepts
optional UpdateThreadRagSettingsRequest opts.
- rag-store.ts: threadSettings map keyed by threadId, plus
loadThreadSettings / updateThreadSettings actions. reingestThread
refreshes the local settings copy when opts were supplied.
- chat-settings-sheet.tsx Retrieval section: when source = thread,
shows side-by-side Mode + Chunking selects above the documents
list. Selecting a different value:
- persists immediately if the thread has no docs
- prompts "Re-index N documents?" if docs exist; on Yes calls
reingestThread with the new opts, on No reverts the select
The (multimodal, late) constraint is enforced via per-option
disabled + tooltip, matching the KB create dialog.
Power users can now set their preferred chunking strategy / mode /
embedder once in Settings → Knowledge Bases and have new KBs use those
values by default, instead of toggling on every create.
Backend
- routes/rag.py:
- GET /api/rag/defaults returns the stored RagDefaults (or sensible
fallbacks when nothing is set: standard / text / null embedder).
- PUT /api/rag/defaults is PATCH-style — only fields present in the
body overwrite. The (multimodal, late) constraint is enforced
here too, so users can't poison the defaults with a combination
the create path would reject.
- Persistence reuses the existing chat_settings store via
upsert_chat_settings_merge; the values live under a single
rag.defaults key as a nested JSON dict.
Frontend
- rag-api.ts: getRagDefaults / setRagDefaults wrappers + RagDefaults
+ UpdateRagDefaultsRequest types.
- rag-store.ts: defaults state, loadDefaults / updateDefaults
actions. loadDefaults swallows errors so a missing endpoint just
leaves defaults null.
- rag-defaults-section.tsx (new): self-contained mode + strategy +
embedding-model controls, persists on change. Used in the Settings
KB tab below the ThreadIndexList section.
- knowledge-bases-tab.tsx: mounts RagDefaultsSection below thread
indexes with a separator.
- kb-create-dialog.tsx: loads defaults on open and prefills the form
with them (falls back to hard-coded standard / text when defaults
haven't loaded yet). reset() returns to the latest defaults rather
than the hard-coded ones.
Adds a floating progress-card stack mounted at the app root that
watches useRagStore.jobs and renders one card per in-flight ingestion
job, regardless of which page the user is on. Wraps the existing
IngestionProgress component so the progress UI stays consistent with
the per-doc chips in the KB detail panel and chat sidebar.
- Terminal cards (complete/error) linger for 4s then auto-dismiss.
- A manual dismiss button is always available.
- Reduced-motion preference is respected (no slide animation).
- Positioned bottom-right, z-50, pointer-events-none container so
clicks pass through to the page underneath.
Mounted in app/routes/__root.tsx next to the existing SettingsDialog
so it's visible across every authenticated route. Closes the last
deferred item from Phase 2.
Closes the upgrade-path gap from Phase 3: a KB or thread whose chunks
were ingested under one strategy can now be rebuilt under a different
one without losing the uploaded files.
Backend
- routes/rag.py:
- POST /api/rag/knowledge-bases/{kb_id}/reingest takes optional
chunking_strategy / mode / embedding_model in the body. Validates
the (multimodal, late) constraint via _validate_mode_combo, updates
the rag_knowledge_bases row, wipes scope artifacts (sqlite chunks
via cascade, Qdrant collection, bm25), and re-INSERTs a fresh
rag_documents row + ingestion job per stored file. Returns the new
job IDs so callers can stream progress via the existing SSE.
- POST /api/rag/threads/{thread_id}/reingest is the simpler thread
variant — no body, rebuilds with current defaults.
- Shared _reingest_scope helper strips the UUID upload prefix when
re-naming docs so users see the original filenames again.
Frontend
- rag-api.ts: reingestKnowledgeBase(kbId, opts) and
reingestThreadDocuments(threadId) wrappers + ReingestResponse type.
- rag-store.ts: reingestKB / reingestThread actions refresh the KB +
doc lists and subscribe to every returned job so the existing
IngestionProgress chips render without further wiring.
- kb-reconfigure-dialog.tsx (new): mirrors KBCreateDialog but
pre-fills with the KB's current strategy / mode / embedder, enforces
the same (multimodal + late) constraint with disabled options, and
confirms before submitting. Submit label flips between "Re-index"
(no settings change) and "Reconfigure & re-index".
- kb-detail-panel.tsx: header gains the chunking + mode summary and
a "Reconfigure…" button that opens the dialog. Button is disabled
when the KB has no documents.
- chat-settings-sheet.tsx Retrieval section: "Re-index" button beside
the existing "Clear thread index" when the thread has documents.
Tests
- test_rag_reingest.py: ReingestKBRequest accepts optional fields,
rejects unknown enum values via Pydantic, and the shared mode-combo
guard still bites on the reingest path.
When a KB has mode = 'multimodal', ingestion extracts images alongside
text and embeds both into a shared 512-d vector space via BGE-VL-base.
Image hits become first-class search results — useful for slides,
reports, and diagrams where text-only retrieval loses ~30-50% of the
content.
Backend
- embeddings.py: new encode_images(image_bytes_list) — opens bytes via
PIL and routes to the SentenceTransformer (BGE-VL accepts PIL images
in the same encode call as text).
- ingestion.py: _subprocess_worker gains document_id arg and a new
_stream_image_chunks() helper. For multimodal KBs the standard text
chunking runs first, then images are saved to
rag_uploads_root() / 'images' / <document_id> / img-NNNN.<ext> and
embedded; for each image with an adjacent caption, both an
'image'-kind chunk (vector = encoded image) and a 'caption'-kind
chunk (vector = encoded caption text) are streamed back with a
shared pair_group field.
- ingestion.py parent: _insert_chunks_and_collect_for_bm25 now reads
kind / image_path / pair_group from the subprocess message,
populates the new rag_chunks columns, and runs a second pass that
sets linked_chunk_id for each image ↔ caption pair. BM25 indexes
text + caption chunks only — image chunks have no tokenisable body.
- retrieval.py: Hit gains a `kind` field plumbed through bm25, dense,
RRF, and rerank paths.
- reranker.py: image-kind hits skip CrossEncoder rerank (text-only
model) but are appended back in their original relative position
rather than dropped.
- routes/rag.py: new GET /api/rag/images/{document_id}/{filename}
static-file route with realpath containment check. SearchHit gains
`kind` and `image_url` fields so the chat UI can render image
thumbnails alongside text hits. KB-doc upload threads kind/mode
through to ingestion.
Frontend
- rag-api.ts: SearchHit gains optional `kind` and `image_url`.
- kb-create-dialog.tsx: new Mode select (Text / Multimodal) alongside
the existing Chunking strategy select. The forbidden
(multimodal + late) combo is enforced in the UI — each side
disables the conflicting option on the other side with a tooltip
explaining why. Embedding-model placeholder cycles through the
three valid defaults (bge-small / nomic / BGE-VL).
- kb-list.tsx + chat-settings-sheet.tsx: 🖼️ MM badge alongside the
⚡ Late one so multimodal KBs are obvious at a glance.
Tests
- test_rag_multimodal.py: parser returns images when want_images=True
and skips them when False; _validate_mode_combo rejects the
forbidden (multimodal, late) pair with 400; RAG_EMBEDDER_MATRIX
contains the three valid combos and excludes the forbidden one;
image URL construction shape is verified. A server-marked test
loads BGE-VL-base end-to-end and confirms image + text vectors
share the same dimension.
Phase 3 of the plan is now feature-complete on the backend; the
remaining items (re-ingest UX for changing strategy on existing KBs)
are tracked under "Backfill UX" and can land separately.
When a KB has chunking_strategy = 'late', ingestion takes a separate
code path that embeds the full document in a single forward pass and
mean-pools token embeddings per chunk span. Each chunk vector carries
full-document context via the encoder's bidirectional attention —
Jina's published technique, ~+6.5 nDCG@10 on long docs.
Backend
- chunking.py: new chunk_pages_with_spans() that joins all pages into a
single full_doc, runs the existing recursive splitter, and returns
per-chunk (char_start, char_end) offsets. Page-number metadata is
recovered by overlap with the original page ranges so PDF citations
still work. Existing chunk_pages() unchanged.
- embeddings.py: new late_chunk_encode(doc_text, char_spans). Tokenizes
the doc with return_offsets_mapping, runs the underlying transformer
to get per-token last_hidden_state, then mean-pools per chunk span.
When the doc exceeds the embedder's context, falls back to windowed
late chunking with a 512-token overlap so cross-window context is
partially preserved.
- ingestion.py _subprocess_worker: branches on chunking_strategy.
'late' path: chunk_pages_with_spans -> late_chunk_encode -> one big
chunks_batch message. 'standard' path unchanged. Both reuse the same
parent-side pump.
- ingestion.enqueue_ingestion: new chunking_strategy + mode kwargs;
defaults to 'standard' / 'text' for legacy callers. embedder model
resolved via resolve_embedder() from the (mode, strategy) matrix.
- routes/rag.py: KB-doc upload reads chunking_strategy + mode from the
KB row (defensive .get for pre-Phase-3 schemas) and threads them
through _start_ingestion.
Frontend
- kb-create-dialog.tsx: new "Chunking strategy" select with Standard /
Late options. Embedding-model placeholder switches to nomic when
Late is picked. createKB request now carries chunking_strategy.
- kb-list.tsx + chat-settings-sheet.tsx: small "⚡ Late" badge next to
late-chunking KB names in the settings KB list and the chat sidebar
dropdown so users see the mode at a glance.
Tests
- test_rag_late_chunking.py: pure-python tests for chunk_pages_with_spans
(chunks index back into full_doc; page numbers inherited by overlap;
pages joined with blank line). A server-marked test loads
all-MiniLM-L6-v2 to exercise late_chunk_encode end-to-end.
No multimodal yet; that's Phase 3B-multimodal (next PR).
Phase 3 lays two orthogonal per-KB knobs in the data and API layers so
follow-up commits (Phase 3B-late, Phase 3B-multimodal) only need to add
their code path and UI selector, not schema or types.
Schema (studio/backend/storage/studio_db.py)
- rag_knowledge_bases gains chunking_strategy ('standard'|'late', default
'standard') and mode ('text'|'multimodal', default 'text'). Both are
immutable after KB creation — changing either invalidates existing
chunks because they were ingested through a specific pipeline.
- rag_chunks gains kind ('text'|'image'|'caption', default 'text'),
image_path (NULLABLE), linked_chunk_id (NULLABLE) — used by Phase
3B-multimodal to pair image chunks with their captions.
- Idempotent ALTER TABLE additions for existing installs (mirrors the
chat_threads display_name / *_code_exec_container_id pattern earlier
in the file).
API (studio/backend/routes/rag.py)
- ChunkingStrategy + KBMode Literal aliases.
- CreateKBRequest accepts both fields with backward-compat defaults.
- KBResponse exposes both.
- _validate_mode_combo rejects (multimodal, late) with 400 — no public
open-weight embedder supports both at once. Surface the constraint
early rather than failing silently during ingestion.
Config (studio/backend/utils/rag/config.py)
- RAG_EMBEDDER_MATRIX dict keyed by (mode, strategy) → embedder name.
- resolve_embedder() helper falls back to RAG_EMBEDDING_MODEL for legacy
KBs that pre-date the columns.
- (multimodal, late) intentionally absent.
Frontend (studio/frontend/src/features/rag/)
- api/rag-api.ts: ChunkingStrategy + KBMode types; KnowledgeBase
interface and createKnowledgeBase request type updated.
- stores/rag-store.ts: createKB signature uses the shared request type.
No user-visible UI changes yet — the only currently-usable combination
is (text, standard), so adding one-option selectors would be UX noise.
Phase 3B-late and Phase 3B-multimodal each add the relevant selector
option as part of shipping the code path.
Replace bare-pypdf/python-docx/BeautifulSoup extraction with Markdown-
preserving parsers so the chunker can split on real heading boundaries
instead of running paragraphs together.
Parsers
- pdf.py: pymupdf + pymupdf4llm.to_markdown() per page; pypdf kept as
fallback when pymupdf can't open the file.
- docx.py: mammoth.convert_to_html() + markdownify, with an explicit
style_map so Title/Heading 1..6 become h1..h6 in the output.
- html.py: BeautifulSoup pre-scrub (drop script/style) then markdownify
so <h*>, <table>, <ul> convert faithfully.
- text.py: signature update only; TXT/MD pass through unchanged.
- parsers/__init__.py: new ParsedImage + ParseResult dataclass; parse()
signature is now parse(path, *, want_images=False) -> ParseResult.
ParseResult is iterable over .pages for backward compat.
Chunker
- chunking.py: prepend Markdown heading separators ("\n# " .. "\n#### ")
to the priority list so heading-aware splits happen for free once the
parsers emit Markdown.
Ingestion
- ingestion.py: single call site updated to consume ParseResult.pages.
Deps (no-torch-runtime.txt)
+ pymupdf>=1.24, pymupdf4llm>=0.0.17, mammoth>=1.7, markdownify>=0.13
- pypdf kept as a fallback path.
Tests
- test_rag_parsers.py asserts Markdown headings survive PDF/DOCX/HTML
extraction; also exercises ParseResult iteration backward-compat.
- test_rag_chunking.py: new case verifying chunks start at Markdown
heading boundaries when the input is Markdown.
Foundation for Phase 3B-late (heading-aware spans for late chunking)
and Phase 3B-multimodal (want_images=True enables image extraction in
the same parser layer). No schema or opt-in flags in this commit.
Bundles three independent CI regressions hitting the maintainer PR
backlog. Each one is verified end-to-end on a staging fork against
real Ubuntu / macOS / Windows GitHub-hosted runners before this
lands.
1. Windows --no-torch install: pydantic + pydantic-core drift to
incompatible versions under `uv pip install --no-deps -r
no-torch-runtime.txt` because pip resolves each independently
from latest. pydantic.VERSION 2.13.4 pins pydantic-core==2.46.4
but pydantic-core 2.47.0 was the freshest published wheel, so
`import pydantic` raised
`SystemError: pydantic-core 2.47.0 is incompatible with the
current pydantic version`. Resolve pydantic WITH deps in a
focused pip call (install.sh, install.ps1,
install_python_stack.py) before the --no-deps no-torch-runtime
pass so pip pins pydantic-core to the version pydantic declares.
pydantic's transitive deps (annotated-types, pydantic-core,
typing-extensions, typing-inspection) are torch-free. Drop the
redundant `Patch Studio venv with full typer / pydantic dep
trees` workaround from the four Windows smoke YAMLs.
Supersedes #5733 + #5734.
2. Linux Studio Update CI: upstream llama.cpp b9261+ split each
binary's entry code into a paired `libllama-<binary>-impl.so`
shared library. `llama-server` and `llama-quantize` NEEDED-link
against `libllama-server-impl.so` / `libllama-quantize-impl.so`
with RUNPATH `$ORIGIN`, so the prebuilt overlay must copy those
alongside the binaries. Without that, ldd reports them missing,
preflight rejects, the installer falls back to source build, and
studio-update-smoke annotates `setup.sh idempotency regressed`.
Add `libllama-*-impl.so*` to the Linux runtime patterns and lock
the pattern in test_rocm_support.TestRuntimePatterns.
3. Mac Studio UI Chat: change-password submit clicked while
disabled. The disable gate only checked new + confirm password
length, but Playwright's first click landed before the
current-password field's React state had committed, so the form
was simultaneously logically-invalid (current_password empty) and
the button was disabled. Tighten the gate to require
`currentPassword.length >= 8` and mirror the same check in the
submit handler so Enter / autofill cannot bypass.
Supersedes #5738.
PyPI release 2026.5.6 is now live; update install.sh and install.ps1 to
pin against the new minimum so fresh installs pick up the latest wheel.
Co-authored-by: Michael Han <michaelhan2050@gmail.com>
* fix(gpt-oss): prefer flex attention over sdpa
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix(gpt-oss): use eager config for unsupported backends
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The pill wired the request end of the loop but the response was lost
on the client: the backend emits a `tool_end` _toolEvent carrying the
base64 PNG on `image_b64` / `image_mime`, but the chat-adapter only
read the `result` string and the generic ToolFallback printed the
prompt as JSON args with an empty Result block -- the "I see no
image" symptom in the chat.
- chat-adapter: when the closing `tool_end` is for `image_generation`,
repackage `image_b64` + `image_mime` (+ size/quality/background)
into a structured result object instead of dropping them.
- New `ImageGenerationToolUI` reads that result and renders the image
inline via `<img src="data:image/...;base64,...">` with the prompt
as a caption. Falls back to a spinner while the request is still
running.
- Register the component under `image_generation` in thread.tsx's
tools.by_name map so it preempts ToolFallback for this tool only.
#5685 wired the backend to honor `prompt_cache_ttl` on the request,
but there was no UI to actually pick it -- every Studio chat ended up
on Anthropic's default 5 minute pool. This adds a Cache TTL selector
to the chat settings sheet's Provider section, visible only when the
provider supports the choice (Anthropic today) and Prompt caching is
on.
- New `promptCacheTtl?: "5m" | "1h"` on `ExternalProviderConfig`.
Normalizer drops the field on providers that don't support the
choice so localStorage stays clean across provider swaps.
- `supportsProviderPromptCacheTtl` + `isPromptCacheTtl` helpers so
the picker, normalizer, and adapter all agree on which values are
valid.
- Settings sheet renders a small Select (5 minutes / 1 hour) right
under the Prompt caching switch when the toggle is on; flipping
it persists on the provider config like the other per-provider
knobs.
- chat-adapter passes `prompt_cache_ttl` on outbound requests when
the value is valid; omitted otherwise so the backend keeps
inheriting Anthropic's 5m default.
The backend already wires OpenAI's Responses-API image_generation
server tool: when `enabled_tools` carries "image_generation" on an
OpenAI cloud request, _stream_openai_responses appends
`{type: "image_generation"}` to the request's tools array and emits
`image_generation_call` output items back to the assistant stream
(see backend/core/inference/external_provider.py and
backend/tests/test_openai_image_generation.py for the round-trip).
This wires the frontend half so a user can actually opt into it from
the composer next to the Search and Code pills, instead of the tool
sitting dormant.
- `providerSupportsBuiltinImageGeneration` gates on OpenAI cloud
(`api.openai.com`) + a Responses-API model prefix (gpt-5.x, o3).
Mirror of the backend's `is_openai_cloud` guard so the pill is hidden
on custom OpenAI-compat backends (ollama / llama.cpp / vLLM) that
report `provider_type="openai"` but would 400 on the tool.
- New `imageToolsEnabled` flag in chat-runtime-store, persisted under
`unsloth_chat_image_tools_enabled` and reset on model change in
chat-page exactly like `codeToolsEnabled`.
- `chat-adapter` appends "image_generation" to `enabled_tools` and
flips `enable_tools: true` when the pill is on, so the existing
backend dispatch picks it up.
- Composer renders an Images pill (lucide `ImageIcon`) immediately
after the Code pill, only when the active model advertises the
capability. The in-thread composer (assistant-ui/thread.tsx) gets
the matching `ImagesToggle` for parity.
The first pass only wired the localStorage mirror into `setCheckpoint`,
but the main chat-page picker actually selects an external model by
calling `setParams({ ...store.params, checkpoint: value })`. That path
never hit `setCheckpoint`, so the persisted slot stayed empty and a
refresh fell back to whatever `/api/inference/status.active_model`
returned -- the previously loaded local model (Qwen3.5 etc) or null
("Select model") when nothing was loaded locally.
Mirror the persistence in `setParams` whenever the checkpoint changes
so every entry point converges on the same behavior. `setCheckpoint`
still does it directly so the load path (compare, GGUF auto-load,
gemma fallback in chat-adapter) keeps working.
* Add Anthropic prompt guards for disabled tools
* fix: merge Anthropic tool guard into structured system prompts
* fix: scope Anthropic disabled-tool guard wording
* chore: adjust claude guard prompt
* chore: add openai to list of prompt guarded providers
* Studio: include web_fetch in the per-turn disabled-tool guard
Add webFetchEnabledForThisTurn alongside webSearchEnabledForThisTurn
and codeExecEnabledForThisTurn. Use it in the enabled_tools payload
so web_fetch follows the Search pill the same way web_search does,
and mention "web fetch" in the disabled-tool guard prose on providers
that ship the tool (Anthropic today; other providers stay inert via
providerSupportsBuiltinWebFetch).
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Selecting a connected external provider (Anthropic, OpenAI, Google, etc.)
and refreshing the page reverted the picker back to no selection. Root
cause is that `PersistedInferenceParams` in `chat-settings-api.ts`
excludes `checkpoint` from the server-side settings payload by design.
Local model selections survive refresh because the backend re-derives
them from `/api/inference/status.active_model`, but external selections
have no backend mirror, so they were lost.
Fix: persist `external::*` checkpoints to a small dedicated
`localStorage` key (`unsloth_chat_last_external_checkpoint`) and hydrate
from it on store init. Local checkpoints continue to come from the
backend status as before; only external ids are mirrored client-side.
`setCheckpoint` writes the key when an external id is selected and
clears it when switching back to a local id, and `clearCheckpoint`
clears it so the picker does not snap back after an explicit reset.
Deleting a connection in one browser left the same connection stuck in
every other browser/tab. The user could not delete or edit it from there
because the local state never caught up with the server, and clicks
either no-op'd or threw on a missing-row backend response.
Two pieces caused the bug:
1. `ChatProvidersSettings` ran its backend sync once on mount and then
silently kept localStorage providers whenever `listProviderConfigs`
returned an empty array, on the assumption that an empty server
response had to be a transient glitch. That assumption is wrong when
another browser removed the last connection. With the guard gone,
trust any successful API response, including an empty list. A focus /
visibilitychange listener now triggers a silent re-sync so the dialog
does not need to be closed and reopened to pick up remote deletes.
2. `deleteProviderConfig` threw on HTTP 404, so once Browser A deleted a
connection, Browser B's "Delete" click failed and the local row stuck
around. Treat 404 as success: the server's job is already done and
the local cache only needs to be pruned.
* Studio: PDF / document attachments for Anthropic + OpenAI
Studio's local-GGUF chat already supports image attachments via the
`image_url` content part shape. PDFs and other documents had no
plumbing for the external-provider path: there was no normalised
content type the frontend could send that translated to Anthropic's
native `document` block or OpenAI's `input_file`.
Add a Studio-side `input_document` content part on assistant /
user messages with three shapes:
{type: "input_document",
file_data: "data:application/pdf;base64,<DATA>",
filename?: "name.pdf",
media_type?: "application/pdf"}
{type: "input_document",
file_url: "https://example.com/doc.pdf",
filename?: "doc.pdf"}
Translation:
- Anthropic Messages API: emits a `document` block with
`{source: {type:"base64", media_type, data}}` or
`{source: {type:"url", url}}`, plus an optional `title` from
`filename`. PDFs are extracted server-side by Anthropic per their
vision/document docs and counted toward input tokens.
- OpenAI Responses API: emits `{type:"input_file", file_data |
file_url, filename?}`. PDFs are extracted server-side.
Empty / unparseable `input_document` parts are silently dropped so
a malformed frontend payload can't blow up the request.
Tests:
- New `test_multimodal_document.py` with 6 cases pinning the
outbound body shape for base64 + URL inputs on both providers,
and the empty-part drop behavior on both.
- The Anthropic assertions strip the prompt-cache wrapper
(`cache_control:{type:ephemeral}` that the tail-message caching
layer adds) before comparing the document core fields, so this
test stays focused on the translation, not the caching layer.
Live verified end-to-end against both providers: a 363-byte
single-page "HELLO" PDF, base64-encoded, attached as a `document`
block to Opus 4.7 and as an `input_file` to gpt-5.5. Both models
correctly extracted the word "HELLO" from the PDF.
Follow-up (out of scope):
- Pydantic schema entry on ChatMessage.content for `input_document`
(today it rides through because ChatCompletionRequest uses
extra=allow). Will tighten when the frontend attach button lands.
- Frontend file-picker UX for non-image attachments on the external
provider path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: gate empty-content msg + skip empty data-URI payload
Gemini High + Codex P2 on PR #5689:
1. Anthropic translation appended an empty `anthropic_parts` array
when every part was dropped (e.g. user sent only an unparseable
input_document). Anthropic 400s on "messages.N.content: at least
one block is required". Skip the whole-message append when no
parts survived. The OpenAI Responses path already had the
equivalent guard, so this brings the two providers into parity.
2. `data:application/pdf;base64,` with no payload (or whitespace-only)
parses to an empty `source.data` string. Anthropic rejects that
with 400 as well. Skip the document block before constructing it.
Plus 2 new test cases pinning both behaviors:
- `test_anthropic_empty_only_document_drops_whole_message`: confirms
a turn whose only content is an unparseable input_document does
NOT make it onto the outbound `messages` array.
- `test_anthropic_empty_data_uri_payload_is_dropped`: confirms an
empty-payload data-URI is filtered out at translation time.
(Note re: gemini's other High note about adding `input_document` to
the Pydantic ContentPart union -- ChatCompletionRequest is configured
with `extra=allow` so the part rides through today. Tightening the
union belongs with the frontend attach-button PR that surfaces the
field; called out as follow-up in the PR description.)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: register input_document in ContentPart + builder
Reviewer caught that the translation code on the external_provider
side was unreachable from a real ChatCompletionRequest:
- ContentPart is a discriminated Union of (text, image_url) only, so
any `{"type": "input_document", ...}` part was rejected by Pydantic
at request parsing with a discriminator error before the helper
could see it.
- _build_external_messages in routes/inference.py only walked text
and image_url parts, so even with a permissive schema the document
parts would have been silently dropped instead of forwarded to
the per-provider translator.
Fixes:
- Add InputDocumentContentPart with optional file_data / file_url /
filename / media_type and Tag("input_document") on the Union.
- Extend _build_external_messages to pass input_document through as
a plain dict for vision-capable providers (so external_provider's
existing Anthropic `document` and OpenAI Responses `input_file`
mappers actually run) and strip them on non-vision providers.
Tests added: schema accepts input_document, builder passes it to
vision providers, builder strips it on non-vision providers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: validate file_data before preferring over file_url
Codex P2 caught that the OpenAI input_document translator treats any
truthy file_data as valid and never falls back to file_url. That
means a malformed `data:application/pdf;base64,` (empty payload) or
a whitespace-only data URI gets forwarded as `file_data=""` and
400s the whole turn, AND silently discards a perfectly recoverable
file_url on the same part.
Mirror the Anthropic-side guard onto the OpenAI Responses path:
treat any "data:" URI with no actual base64 payload as missing and
fall through to file_url. Standalone-empty data URIs (no fallback)
are dropped entirely instead of being sent to the wire.
Tests added: empty data URI + valid file_url -> file_url wins,
whitespace-only data URI + valid file_url -> file_url wins,
empty data URI without fallback -> part is dropped.
* Address review: Anthropic side also falls back to file_url on empty data URI
Codex P2 follow-up to my earlier fix: I added the empty-data-URI ->
file_url fallback to the OpenAI Responses translator but missed
the Anthropic translator, which still `continue`d on empty payloads
and discarded an otherwise valid file_url on the same part. Result:
when the frontend supplied both file_data (placeholder / broken)
AND a working file_url, Anthropic silently lost the attachment;
when the message contained only that part, the whole message could
be dropped before reaching the wire.
Mirrored the OpenAI guard: any "data:" URI with no actual base64
payload (`data:application/pdf;base64,` or whitespace-only) is
treated as missing, and the file_url branch takes over. The
all-parts-dropped guard further down already handles the
no-fallback case.
Tests added: empty data URI + valid file_url -> URL source on the
wire with the filename preserved; whitespace-only data URI + valid
file_url -> URL source on the wire.
* Address review: gate input_document passthrough to anthropic + openai
Codex P1: only `_stream_anthropic` and `_stream_openai_responses`
have explicit translation logic for input_document parts (the former
maps to {type:"document", source:...}, the latter to
{type:"input_file", file_data|file_url}). Every other provider
(gemini / mistral / kimi / openrouter / deepseek / qwen / custom)
goes through the generic /chat/completions passthrough that forwards
`messages` verbatim, so any input_document part on a non-vision
route on those providers would 400 with an unknown content_part
type.
Added `_INPUT_DOCUMENT_PROVIDERS = frozenset({"anthropic", "openai"})`
constant and gated the pass-through branch on `provider_type in
_INPUT_DOCUMENT_PROVIDERS`. Every other provider strips the part
(text content survives). Threaded provider_type through from
_proxy_to_external_provider's call site.
Tests updated: vision + provider in {anthropic, openai} still
forwards; six unmapped providers (gemini/mistral/kimi/openrouter/
deepseek/qwen) strip the part; missing provider_type strips
defensively. The existing non-vision drop test still passes.
* Fix stale web_fetch tool-version assertion after merging main
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: wire OpenAI Responses server-side context compaction
The OpenAI Responses API accepts a `context_management` field that
enables server-side compaction. When the rendered prompt crosses the
configured threshold, the API runs a server-side compaction step and
the request continues against the compacted prefix. No beta header
and no dated version pin are required, per the docs.
Changes:
- Add `compaction_threshold: Optional[int]` (ge=1_000, le=2_000_000)
to ChatCompletionRequest. Thread through `routes/inference.py` ->
`stream_chat_completion` -> `_stream_openai_responses`.
- In `_stream_openai_responses`, when threshold is set AND the base
URL points at cloud OpenAI (api.openai.com), attach
`context_management: [{type:"compaction", compact_threshold:N}]`
to the outbound body. Non-cloud bases (ollama, llama.cpp, "custom"
presets) silently drop the field so we don't 400 those servers.
- Add `test_openai_compaction.py` with 4 cases: cloud OpenAI sets
the field verbatim, low-threshold probe passes through (we don't
clamp on the OpenAI side because the API accepts whatever),
non-cloud base drops the field, omitted threshold leaves body
untouched.
Live verified against the real OpenAI API on gpt-5.5:
`context_management:[{type:"compaction", compact_threshold:200000}]`
returns 200 with no error.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: accept Azure OpenAI base URLs + raise compaction floor
Two reviewer follow-ups on the OpenAI compaction PR:
1. The `is_openai_cloud = "api.openai.com" in self.base_url` check
excluded Azure OpenAI Foundry, even though Azure exposes the
same /v1/responses extensions (context_management,
prompt_cache_retention, container shell). Users on Azure saw
their compaction toggle silently no-op. Broadened the check to
also match `*.openai.azure.com` and made it case-insensitive so
URLs copy-pasted from the Azure portal still resolve. Non-cloud
OpenAI-compatible servers (ollama / llama.cpp / vLLM / "custom"
preset) still fall outside the gate.
2. The schema floor on compaction_threshold was ge=1_000, which is
well below the upstream Responses API's effective minimum
(vercel/ai#12486, langchain-ai/langchain#35464 report
`compact_threshold is not enabled` 400s on Azure at 100k; cloud
uses 200k as the canonical example). Raised the floor to 10k
so obvious typos surface as a clean 422 from FastAPI rather than
an opaque upstream 400 the user has to debug from the SSE
stream.
Tests added: Azure base URL carries both context_management and
prompt_cache_retention; mixed-case Azure URLs match; schema rejects
9_999 and accepts 10_000.
* Address review: drop schema-level compaction floor (cross-provider regression)
Codex P2 follow-up on the previous floor bump: ge=10_000 was
enforced globally at the ChatCompletionRequest layer, but the field
is documented as a no-op on every non-cloud OpenAI base and every
non-OpenAI provider. With the global floor, an Anthropic / ollama
/ llama.cpp / custom request that happens to carry compaction_threshold
below 10k was rejected with 422 at request validation time instead
of being silently ignored as the description promised.
Reverted the schema floor to ge=1 (any positive int) and rewrote
the description to call out per-provider routing: OpenAI cloud's
effective floor is around 200k and surfaces upstream 400s below
that; _stream_anthropic clamps sub-50k values up. Per-provider
helpers stay the single source of truth on the floor.
Test updated to pin: zero is still rejected, but every positive
value (1, 5_000, 9_999, 10_000, 200_000) passes schema validation.
* Address CodeQL: hostname-anchored OpenAI cloud detection
CodeQL py/incomplete-url-substring-sanitization fired on
`".openai.azure.com" in _base`. An attacker who controls the
configured base_url could slip cloud-only request body fields
(prompt_cache_retention, context_management compaction, container
shell) to an arbitrary server with:
https://evil.com/api.openai.com/v1https://api.openai.com.attacker.com/v1https://attacker.com/.openai.azure.com/v1https://my-resource.openai.azure.com.attacker.com/openai/v1
Replaced the substring check with a `_is_openai_family_cloud`
helper that runs urllib.parse.urlparse on the URL and matches the
lowercased hostname exactly (`api.openai.com`) or via `endswith`
on the leading-dot suffix (`.openai.azure.com`). Both halves are
host-anchored so path / fake-subdomain bypasses fail.
Test added: every attacker-controlled bypass shape above must NOT
carry context_management OR prompt_cache_retention on the wire.
Existing Azure and openai.com tests still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: scope compaction_threshold description to OpenAI on this branch
Codex P2: the field description on this PR mentioned Anthropic
compaction behavior, but the Anthropic wiring lives on PR 5686
(separate branch). On feat/openai-compaction alone, _stream_anthropic
has no compaction_threshold parameter, so the field is silently
ignored for Anthropic requests and the doc claim was misleading.
Trimmed the description to OpenAI cloud + Azure Foundry only on
this branch. PR 5686 already re-adds the Anthropic clause via its
own change, so the rebase / merge order on main will land the
combined description naturally once both PRs ship.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: wire Anthropic server-side context compaction
Anthropic ships server-side context compaction as a beta
(`compact-2026-01-12`). When the rendered prompt crosses the
configured input-token threshold, Anthropic runs an extra LLM pass
that summarises older turns and the request continues against the
compacted prefix. The response carries the original top-level fields
plus a new `context_management` block (with `applied_edits`) and
`usage.iterations[]` accounting per pass.
Per the docs the feature is currently supported on Opus 4.6, Opus 4.7,
Sonnet 4.6, and Mythos preview. The minimum threshold is 50k tokens;
under-50k requests 400.
Changes:
- Add prefix gate + helper `_anthropic_supports_compaction` plus
constants `_ANTHROPIC_COMPACTION_PREFIXES`, `_ANTHROPIC_COMPACTION_BETA`,
`_ANTHROPIC_COMPACTION_TYPE`, `_ANTHROPIC_COMPACTION_MIN`.
- Add `compaction_threshold: Optional[int]` to ChatCompletionRequest
(50k ge bound, 2M le bound). Thread through `routes/inference.py`
-> `stream_chat_completion` -> `_stream_anthropic`.
- In `_stream_anthropic`, when threshold is set AND the model
accepts compaction, attach `context_management.edits[{type:
"compact_20260112", trigger:{type:"input_tokens", value:N}}]` to
the outbound body. Sub-50k values are clamped up to 50k to keep
the request well-formed.
- Refactor the anthropic-beta header builder to merge any combination
of `code-execution-2025-08-25` + `compact-2026-01-12` flags into
one header value. Unrelated betas added at the registry level still
pass through.
- Add `test_anthropic_compaction.py` with 16 cases: gate matrix
(every doc-listed model), correct body shape, threshold clamping,
beta header merge with code execution, silent no-op on unsupported
models, omitted-threshold pass-through.
Live verified end-to-end against the real Anthropic API:
`compact_20260112` accepted on Opus 4.7, response carries
`context_management.applied_edits` + `usage.iterations[]` as
documented. (The first WebFetch-summarised version of these docs
suggested `compact_20260120`; the actual API only accepts
`compact_20260112`, matching the beta-header date. Worth pinning
behind a test so a future doc update can't drift back.)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: drop ge=50_000 clamp + parse usage.iterations[]
Two reviewer follow-ups on the compaction PR:
1. Pydantic ge=50_000 on compaction_threshold was dead code.
FastAPI rejected sub-50k threshold values with a 422 before the
`max(int(...), _ANTHROPIC_COMPACTION_MIN)` clamp in
_stream_anthropic could ever fire. Relaxed the floor to ge=1 so
the in-helper clamp actually does its job; the schema comment
now explains why this is intentional. Added a regression test
that posts a value of 1 and 49_999 through the real request
schema.
2. Anthropic publishes per-iteration token counts in
`usage.iterations[]` whenever a fresh compaction has run, and
the top-level input_tokens / output_tokens cover only the
`message` iteration -- billing must add the compaction
iterations on top. Aggregate compaction iteration tokens into
`last_usage["compaction_input_tokens" / "compaction_output_tokens"]`
so the cost surface (PR 5690) can read them without re-walking
the array, and surface both figures in the closing stream
summary log. Added two tests: one that pins the aggregation on a
compacted turn and one that pins `None` when no fresh
iterations land (so re-applied compaction blocks don't double-bill).
Sourcing: https://platform.claude.com/docs/en/build-with-claude/compaction
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: round-trip Anthropic compaction blocks across turns
Codex P1: once context_management is enabled and Anthropic runs
server-side compaction mid-stream, the response carries a
`{type:"compaction", content:"<summary>"}` content block on the
assistant message. The translator only handled text_delta and
input_json_delta on content_block_delta, so the compaction block
was silently dropped. Worse, the request schema's ContentPart
discriminated Union didn't accept `type:"compaction"`, and
_build_external_messages didn't pass it through, so even a
hand-crafted assistant message carrying the block would 422 at
parse time. Net result: Anthropic re-compacted from scratch on
every subsequent turn, wasting input tokens and reasoning budget.
End-to-end backend wiring of the round-trip:
1. SSE translator. _stream_anthropic now tracks a `current_compaction`
state slot. content_block_start with type=="compaction" seeds it
(Anthropic may include the summary on the start event AND/OR
stream it via text_delta events on the same block index --
handle both). text_delta inside a compaction block routes into
the compaction buffer instead of the user-visible content
stream, since the summary is opaque internal state, not
assistant prose. content_block_stop emits a `compaction_block`
tool_event carrying the full summary so the chat-adapter can
persist it. compaction_blocks_seen is surfaced in the closing
summary log.
2. Pydantic schema. Added CompactionContentPart with Tag("compaction")
on the ContentPart Union so requests carrying the block parse
cleanly. Required `content` field with a docstring pointing at
the Anthropic docs.
3. Message builder. _build_external_messages forwards compaction
parts on both vision and non-vision paths; the per-provider
stream helper decides whether to forward to the wire (Anthropic
does; other providers ignore the part). When a non-vision route
ends up with a single text part, collapse back to a string
so providers that don't accept content arrays still get the
expected shape.
4. _stream_anthropic outbound translator. {type:"compaction"} parts
on an assistant message land on the wire verbatim. Empty/missing
`content` is skipped so a malformed stored block can't 400
Anthropic.
Tests added (5): stream emits compaction_block tool event with the
summary intact; user-visible content stream does NOT carry the
summary text; outbound body forwards compaction parts verbatim on
the next turn; Pydantic schema accepts the part; builder passes
it through on both vision and non-vision provider routes.
Frontend follow-up: the chat-adapter needs to persist the
compaction_block tool_event onto the stored assistant message so
turn N+1 includes it in payload.messages. Pinned in the PR
description.
Sourcing: https://platform.claude.com/docs/en/build-with-claude/compaction
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: gate compaction-part passthrough to Anthropic only
Codex P1: my previous round-trip change preserved {type:"compaction"}
parts on every provider route in _build_external_messages. That
meant a chat history with prior compaction state silently leaked
the Anthropic-specific block to OpenAI/DeepSeek/Mistral/Gemini/
Kimi/OpenRouter on a provider switch, where generic
/chat/completions passthrough hands the unknown content type to
the upstream API and 400s the whole turn.
Added a `provider_type` kwarg to _build_external_messages and
gated the compaction forwarder on `provider_type == "anthropic"`.
Every other value (including the legacy None for callers that
don't pass it yet) strips the part. The Anthropic stream helper
still maps it to a native `compaction` block on the wire.
Threaded provider_type through from _proxy_to_external_provider's
call site.
Tests updated: vision + provider="anthropic" still forwards; six
non-anthropic providers strip the part; missing provider_type
strips defensively; non-vision + anthropic still forwards; non-vision
+ non-anthropic collapses back to a text string.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
