Dense vectors don't preserve numbers (BGE-small treats 'Figure 1' and
'Figure 10' as nearly identical), so a query like 'what does Figure 1
show' got out-ranked by chunks describing other figures that share more
vocabulary with the question — even after the figure-boundary chunker
ensured Figure 1's chunk started with the literal caption.
Detect 'Figure N' / 'Table N' (numbered, decimal, appendix-style)
references in the query, look up chunks that start with those captions
directly, and feed the result as a third RRF source. RRF gives them
rank-0 in the third ranking and the fused score lifts them above the
dense-vocabulary noise. No-ops when the query has no figure ref.
Dense embedders mean-pool over a whole chunk, so a 'Figure 1:' caption
buried at the end of a 500-token body chunk gets washed out by the
surrounding theory text and never surfaces for queries about that
figure. Pre-split each page's markdown at the start of every
Figure/Table caption line so the caption anchors its own chunk, which
gives both BM25 and the dense vector a focused, figure-dominated
target. Handles numbered, decimal, and appendix-style labels
(Figure 1, Figure 1.2, Figure B.1, Table 4, Fig./Tab. abbreviations).
Reasoning models (gemma-4, qwen3-thinking) burn the entire max_tokens
budget on <thinking> output and return empty visible content, so the
captioner produced zero captions for every image. Pass
chat_template_kwargs={enable_thinking: false} per-request to skip the
reasoning phase, and bump max_tokens 120 -> 200 as headroom.
Both modules used stdlib logging.getLogger which is not bridged to the
project's structlog config, so every probe / captioner log was silently
dropped. Switch to loggers.get_logger and convert %-format calls to
structlog kwargs so the captioning path becomes observable.
Four valid review comments from gemini-code-assist[bot] on #5759:
1. core/rag/bm25.py:_load — wrap bm25s.BM25.load + json.loads with
specific exception handlers (FileNotFoundError, OSError,
JSONDecodeError, ValueError) and log a warning instead of
propagating a 500. Corrupt/partial bm25 dirs now degrade to
empty-search rather than crashing the request.
2. core/rag/tool.py was importing _resolve_scope_embedder from
routes/rag.py — a layering violation (core depending on
routes). Move the resolver into a new core/rag/scope.py module
along with the chat-settings key constants; routes/rag.py
now re-imports it under the same name. Same behaviour, no
cycle, one source of truth for the resolution logic.
3. core/rag/bm25.py:rebuild_index — call delete_scope before saving
the new index so stale files from a previous build (or a
bm25s naming change) never coexist with current files. The
library's save() doesn't unlink files it doesn't write.
4. routes/rag.py:_save_upload was running f.write() synchronously
inside an async def. Switch to anyio.open_file() so each chunk
write runs in a worker thread instead of blocking the event
loop on multi-MB uploads. Cleanup unlink happens after the
async-with closes the handle so Windows is happy.
Skipped one (vector_store.py:133 'hasattr query_points' redundancy)
— that comment was on the pre-rewrite Qdrant code; the file is
now sqlite-vec backed and the hasattr check is gone.
core/rag/db.py, vector_store.py, tool.py, bm25.py, and reranker.py
all run only in the FastAPI parent process. Switch their loggers
from Python stdlib to studio's structlog get_logger so their output
shows up in the same JSON stream as the rest of the backend (the
request_completed / RAG search lines).
embeddings.py and ingestion.py stay on stdlib because they execute
inside the mp.spawn ingestion subprocess, which doesn't inherit the
parent's structlog configuration.
asg017/sqlite-vec is Apache-2.0 and OSI-approved. Replaces
qdrant-client (~30 MB) with a small SQLite extension loaded into a
dedicated rag.db file. Single file holds RAG vectors; bm25s indexes
and chat-side studio.db are unaffected.
- New core/rag/db.py owns the rag.db connection and sqlite-vec load.
Extension load runs once at first open. Process-wide singleton
protected by a lock; check_same_thread=False + WAL handles the
FastAPI thread pool.
- core/rag/vector_store.py keeps the same public API
(ensure_collection / upsert_chunks / search / collection_exists /
delete_scope / delete_document) so callers in routes/rag.py,
core/rag/ingestion.py, core/rag/tool.py, and core/rag/retrieval.py
don't change. ensure_collection is now a no-op; collection_exists
returns True iff the scope has at least one indexed vector.
- search uses sqlite-vec's vec_distance_cosine and converts distance
to similarity in [0, 1] so the per-scope min_score threshold
semantics stay identical.
- Mixed-dim scopes coexist behind WHERE scope = ? — the per-scope
embedder resolver guarantees one embedder per scope.
- requirements/rag.txt swaps qdrant-client for sqlite-vec.
- utils/paths/storage_roots.py drops rag_vectordb_root() (the old
qdrant directory); rag.db lives directly under rag_root().
- Rewritten tests/python/test_rag_vector_store.py for the new
semantics (collection_exists tracks populated scopes; new tests
for filtered search and upsert conflict resolution).
Python build requirement: connection.enable_load_extension(True)
must be available. install.sh creates the venv via uv-managed
python-build-standalone, which is compiled with
--enable-loadable-sqlite-extensions, so this works on standard
installs. core/rag/db.py raises an actionable error on the rare
custom-interpreter case.
The query was always going through the default text embedder
(bge-small, 384-d) regardless of how the scope was ingested. A
multimodal thread indexed by Qwen3-VL (2048-d) crashed at cosine
similarity with 'shapes (104,2048) and (384,) not aligned'.
Resolve the scope's embedder at search time:
- kb_<id> -> rag_knowledge_bases.embedding_model column
- thread_<id> -> thread settings (with fallback to defaults +
RAG_EMBEDDER_MATRIX matrix lookup)
Pass it through retrieve_hybrid/retrieve_dense to embeddings.encode
so the query lands in the same vector space as the docs. Both the
/api/rag/search route and the search_knowledge_base tool use the
same resolver.
Qwen3-VL-Embedding-2B's modules.json references
sentence_transformers.base.modules, introduced after 5.2.0.
Unlike BGE-VL, Qwen3-VL has no fragile custom forward subclass to
break against the newer ST internals, so the bump should land
cleanly for the multimodal path.
Built on Qwen3 (not CLIP), so the 77-token text cap that bit
BGE-VL-base is gone — long chunks embed losslessly. Loads via
vanilla SentenceTransformer with trust_remote_code, no custom
adapter needed. 2048-d shared text+image space.
Adds qwen-vl-utils>=0.0.14 to rag.txt for image preprocessing,
required by the Qwen3-VL embedder family.
BGE-VL inherits CLIP's 77-token text positional embedding table —
longer chunks crash inside the text model with a shape mismatch.
Pre-tokenize with truncation=True, max_length=77 and call
get_text_features directly so the high-level encode() (which does
not truncate) is bypassed. Log when truncation happens — text
chunks beyond the cap are silently cut, so multimodal mode is
lossy on the text channel. Image channel is unaffected.
BGE-VL's sentence-transformers shim (bge_vl_clip_transformer.py)
is coupled to specific ST internals and broke across both the 5.2
and 5.3 pin attempts. The canonical load path documented at
bge-model.com is transformers.AutoModel with trust_remote_code +
model.set_processor() + model.encode(text=/images=).
- Wrap that path in _BGEVLAdapter exposing the slice of
SentenceTransformer API the ingester uses (encode for text or PIL
images, get_sentence_embedding_dimension, best-effort tokenize).
- Restore BAAI/BGE-VL-base in RAG_EMBEDDER_MATRIX.
- Revert sentence_transformers pin to 5.2.0 — no longer relevant
since the multimodal path no longer touches ST.
BGE-VL-base ships a custom Transformer subclass that's tightly
coupled to a specific sentence-transformers internal API — it
imports a module path missing in older ST and overrides forward()
expecting a return shape that changed in newer ST. We can't pin ST
to BGE-VL's exact version without risking the training/inference
paths that use the same library, so swap the multimodal default to
the canonical clip-ViT-B-32 which sentence-transformers wraps
natively (no trust_remote_code, no shim, ST-version-independent).
Same 512-d shared text+image space.
BGE-VL's custom modeling code imports
sentence_transformers.base.modules.transformer, which doesn't exist
in the 5.2.0 pin. Bump to >=5.3.0,<7 so the new Module package is
available.
BGE-VL (multimodal) and nomic-embed-text-v1.5 (late chunking) both
ship custom modeling code in their model repos; sentence-transformers
refuses to import the referenced module (e.g. bge_vl_clip_transformer)
without trust_remote_code=True. Safe to enable because the embedder
matrix is config-pinned — users don't supply arbitrary names.
client.search() was removed/renamed to query_points() in
qdrant-client 1.10+. Use the new API when available and fall back
to search() for older pinned versions.
- chat-adapter pre-fetches retrieval on every turn when RAG is on,
regardless of provider. Users no longer have to phrase queries as
'the document I attached' for retrieval to fire. Local tool models
still get search_knowledge_base registered as a refinement path.
- New per-thread ragMinScore slider (Min relevance, 0..1) gates
retrieved hits by dense cosine similarity. Hits below the floor
(and BM25-only hits with no dense signal) are dropped server-side
so unrelated docs don't get injected when the user's query is
off-topic from what's indexed.
- Backend logs at search start (scope, top_k, min_score, query
preview), after retrieval (retrieved vs met_threshold counts),
and on return (final hit count) for both /api/rag/search and the
search_knowledge_base tool path.
- System-prompt nudge prepended when pre-fetch returns hits so the
model knows to cite [1], [2] rather than paraphrase silently.
no-torch-runtime.txt is only consumed in NO_TORCH (Intel Mac
GGUF-only) mode, so qdrant-client / bm25s / pymupdf etc. were never
installed in the normal install path. Split them into rag.txt and
add a step to install_python_stack.py that installs it after studio
deps, skipped only when NO_TORCH is set.
Promotes RAG to a first-class composer toggle alongside Think / Web
Search / Code, with tool-use semantics on local models that support
tools and a pre-fetch fallback on external providers. The model
decides when to call `search_knowledge_base` on local inference; on
external providers retrieval still fires before each message (the
existing pre-fetch path), gated on the same button.
Backend
- core/rag/tool.py (new): search_knowledge_base handler + JSON-schema
tool spec. Resolves scope (kb_id wins over thread_id) from the
request's rag_scope, runs retrieve_hybrid + optional rerank, then
hydrates filename / page_number / text from sqlite and formats as
numbered Markdown citations ('[1] file.pdf (page 5): ...') for the
LLM to cite. Empty scope returns a user-facing hint; empty results
return a clear no-match message instead of an empty string.
- core/inference/tools.py: SEARCH_KNOWLEDGE_BASE_TOOL added to
ALL_TOOLS (lazy import keeps tools.py importable on inference
paths that never touch RAG). execute_tool() gains a tool_context
parameter that carries per-request extras the LLM doesn't see
(currently just rag_scope). The new 'search_knowledge_base' branch
dispatches to the handler with scope unpacked from tool_context.
- core/inference/llama_cpp.py + safetensors_agentic.py +
orchestrator.py: thread tool_context through generate_chat_completion_
with_tools / run_safetensors_tool_loop / execute_tool. Both local
backends (GGUF llama-server and safetensors agentic) carry the same
context object.
- models/inference.py: ChatCompletionRequest gains optional
rag_scope: dict ({kb_id?, thread_id?, enable_rerank?, default_top_k?,
reranker_model?}). Ignored unless 'search_knowledge_base' is in
enabled_tools.
- routes/inference.py: both the GGUF and safetensors call sites for
generate_chat_completion_with_tools forward payload.rag_scope into
tool_context.
Frontend
- chat-runtime-store.ts: global ragToolEnabled boolean + setter +
CHAT_RAG_TOOL_ENABLED_KEY localStorage, mirroring toolsEnabled /
codeToolsEnabled. Settings-hydration migration auto-flips
ragToolEnabled=true for pre-Phase-4 users who already had ragSource
set, so existing RAG users don't silently lose retrieval on upgrade.
- shared-composer.tsx: new 'RAG' pill button after Images (uses
lucide BookOpenIcon, composer-pill-btn style, data-active toggle).
Disabled when no model is loaded. Toggling on from ragSource='off'
auto-flips source to 'thread' so the sidebar lands ready-to-go.
- chat-adapter.ts:
* The existing pre-fetch block is now gated on ragToolEnabled AND
only fires when the tool path isn't viable (external provider OR
local model without tool-use support). Tool-capable local models
skip pre-fetch and let the LLM decide.
* The local-model body assembly adds 'search_knowledge_base' to
enabled_tools and packs ragSource + enableRerank + ragTopK into a
rag_scope object the backend tool handler consumes.
- chat-settings-sheet.tsx: entire Retrieval CollapsibleSection is
wrapped in {ragToolEnabled && ...} so it hides when the button is
off — the button is now the single on/off control. The 'Off'
option is removed from the Source dropdown (the button handles
that). Default open when shown so settings are one click away.
Tests
- test_rag_tool_handler.py: handler covers empty query, missing
scope, kb_id > thread_id precedence, thread-only path, citation
formatting (numbered + page numbers + unknown source); tool spec
shape (function/name/required); execute_tool dispatch with and
without tool_context; ALL_TOOLS includes the new spec without
dropping the existing ones.
Verification scope
- Local GGUF with tools: toggle button on, upload doc, ask about
doc content → assistant emits a search_knowledge_base tool call
card (rendered by the existing ToolFallback component since no
custom UI exists yet — that's a v2 nice-to-have).
- External provider (Anthropic / OpenAI / etc.): same button, same
UX, but uses the pre-fetch path under the hood.
- Migration: pre-existing ragSource != off → button initializes ON
so retrieval keeps working.
Threads can now opt into late chunking or multimodal mode independently
of the KBs they reference. Per-thread settings persist in chat_settings
under thread:<id>:rag and fall back to the app-level defaults when the
thread hasn't set anything explicitly.
Backend (routes/rag.py)
- ThreadRagSettings / UpdateThreadRagSettingsRequest Pydantic models.
- GET/PUT /api/rag/threads/{thread_id}/settings backed by
chat_settings (upsert_chat_settings_merge). Same (multimodal, late)
constraint enforcement as the create + defaults endpoints.
- POST /api/rag/threads/{thread_id}/reingest now accepts the same
body shape — if any field is set, the new settings are persisted
via set_thread_rag_settings BEFORE the reingest, so subsequent
uploads pick up the change too.
- upload_thread_document reads the per-thread settings and passes
them through to _start_ingestion, replacing the previous hard-coded
('standard', 'text', RAG_EMBEDDING_MODEL) defaults.
Frontend
- rag-api.ts: ThreadRagSettings type + getThreadRagSettings /
setThreadRagSettings wrappers. reingestThreadDocuments now accepts
optional UpdateThreadRagSettingsRequest opts.
- rag-store.ts: threadSettings map keyed by threadId, plus
loadThreadSettings / updateThreadSettings actions. reingestThread
refreshes the local settings copy when opts were supplied.
- chat-settings-sheet.tsx Retrieval section: when source = thread,
shows side-by-side Mode + Chunking selects above the documents
list. Selecting a different value:
- persists immediately if the thread has no docs
- prompts "Re-index N documents?" if docs exist; on Yes calls
reingestThread with the new opts, on No reverts the select
The (multimodal, late) constraint is enforced via per-option
disabled + tooltip, matching the KB create dialog.
Power users can now set their preferred chunking strategy / mode /
embedder once in Settings → Knowledge Bases and have new KBs use those
values by default, instead of toggling on every create.
Backend
- routes/rag.py:
- GET /api/rag/defaults returns the stored RagDefaults (or sensible
fallbacks when nothing is set: standard / text / null embedder).
- PUT /api/rag/defaults is PATCH-style — only fields present in the
body overwrite. The (multimodal, late) constraint is enforced
here too, so users can't poison the defaults with a combination
the create path would reject.
- Persistence reuses the existing chat_settings store via
upsert_chat_settings_merge; the values live under a single
rag.defaults key as a nested JSON dict.
Frontend
- rag-api.ts: getRagDefaults / setRagDefaults wrappers + RagDefaults
+ UpdateRagDefaultsRequest types.
- rag-store.ts: defaults state, loadDefaults / updateDefaults
actions. loadDefaults swallows errors so a missing endpoint just
leaves defaults null.
- rag-defaults-section.tsx (new): self-contained mode + strategy +
embedding-model controls, persists on change. Used in the Settings
KB tab below the ThreadIndexList section.
- knowledge-bases-tab.tsx: mounts RagDefaultsSection below thread
indexes with a separator.
- kb-create-dialog.tsx: loads defaults on open and prefills the form
with them (falls back to hard-coded standard / text when defaults
haven't loaded yet). reset() returns to the latest defaults rather
than the hard-coded ones.
Closes the upgrade-path gap from Phase 3: a KB or thread whose chunks
were ingested under one strategy can now be rebuilt under a different
one without losing the uploaded files.
Backend
- routes/rag.py:
- POST /api/rag/knowledge-bases/{kb_id}/reingest takes optional
chunking_strategy / mode / embedding_model in the body. Validates
the (multimodal, late) constraint via _validate_mode_combo, updates
the rag_knowledge_bases row, wipes scope artifacts (sqlite chunks
via cascade, Qdrant collection, bm25), and re-INSERTs a fresh
rag_documents row + ingestion job per stored file. Returns the new
job IDs so callers can stream progress via the existing SSE.
- POST /api/rag/threads/{thread_id}/reingest is the simpler thread
variant — no body, rebuilds with current defaults.
- Shared _reingest_scope helper strips the UUID upload prefix when
re-naming docs so users see the original filenames again.
Frontend
- rag-api.ts: reingestKnowledgeBase(kbId, opts) and
reingestThreadDocuments(threadId) wrappers + ReingestResponse type.
- rag-store.ts: reingestKB / reingestThread actions refresh the KB +
doc lists and subscribe to every returned job so the existing
IngestionProgress chips render without further wiring.
- kb-reconfigure-dialog.tsx (new): mirrors KBCreateDialog but
pre-fills with the KB's current strategy / mode / embedder, enforces
the same (multimodal + late) constraint with disabled options, and
confirms before submitting. Submit label flips between "Re-index"
(no settings change) and "Reconfigure & re-index".
- kb-detail-panel.tsx: header gains the chunking + mode summary and
a "Reconfigure…" button that opens the dialog. Button is disabled
when the KB has no documents.
- chat-settings-sheet.tsx Retrieval section: "Re-index" button beside
the existing "Clear thread index" when the thread has documents.
Tests
- test_rag_reingest.py: ReingestKBRequest accepts optional fields,
rejects unknown enum values via Pydantic, and the shared mode-combo
guard still bites on the reingest path.
When a KB has mode = 'multimodal', ingestion extracts images alongside
text and embeds both into a shared 512-d vector space via BGE-VL-base.
Image hits become first-class search results — useful for slides,
reports, and diagrams where text-only retrieval loses ~30-50% of the
content.
Backend
- embeddings.py: new encode_images(image_bytes_list) — opens bytes via
PIL and routes to the SentenceTransformer (BGE-VL accepts PIL images
in the same encode call as text).
- ingestion.py: _subprocess_worker gains document_id arg and a new
_stream_image_chunks() helper. For multimodal KBs the standard text
chunking runs first, then images are saved to
rag_uploads_root() / 'images' / <document_id> / img-NNNN.<ext> and
embedded; for each image with an adjacent caption, both an
'image'-kind chunk (vector = encoded image) and a 'caption'-kind
chunk (vector = encoded caption text) are streamed back with a
shared pair_group field.
- ingestion.py parent: _insert_chunks_and_collect_for_bm25 now reads
kind / image_path / pair_group from the subprocess message,
populates the new rag_chunks columns, and runs a second pass that
sets linked_chunk_id for each image ↔ caption pair. BM25 indexes
text + caption chunks only — image chunks have no tokenisable body.
- retrieval.py: Hit gains a `kind` field plumbed through bm25, dense,
RRF, and rerank paths.
- reranker.py: image-kind hits skip CrossEncoder rerank (text-only
model) but are appended back in their original relative position
rather than dropped.
- routes/rag.py: new GET /api/rag/images/{document_id}/{filename}
static-file route with realpath containment check. SearchHit gains
`kind` and `image_url` fields so the chat UI can render image
thumbnails alongside text hits. KB-doc upload threads kind/mode
through to ingestion.
Frontend
- rag-api.ts: SearchHit gains optional `kind` and `image_url`.
- kb-create-dialog.tsx: new Mode select (Text / Multimodal) alongside
the existing Chunking strategy select. The forbidden
(multimodal + late) combo is enforced in the UI — each side
disables the conflicting option on the other side with a tooltip
explaining why. Embedding-model placeholder cycles through the
three valid defaults (bge-small / nomic / BGE-VL).
- kb-list.tsx + chat-settings-sheet.tsx: 🖼️ MM badge alongside the
⚡ Late one so multimodal KBs are obvious at a glance.
Tests
- test_rag_multimodal.py: parser returns images when want_images=True
and skips them when False; _validate_mode_combo rejects the
forbidden (multimodal, late) pair with 400; RAG_EMBEDDER_MATRIX
contains the three valid combos and excludes the forbidden one;
image URL construction shape is verified. A server-marked test
loads BGE-VL-base end-to-end and confirms image + text vectors
share the same dimension.
Phase 3 of the plan is now feature-complete on the backend; the
remaining items (re-ingest UX for changing strategy on existing KBs)
are tracked under "Backfill UX" and can land separately.
When a KB has chunking_strategy = 'late', ingestion takes a separate
code path that embeds the full document in a single forward pass and
mean-pools token embeddings per chunk span. Each chunk vector carries
full-document context via the encoder's bidirectional attention —
Jina's published technique, ~+6.5 nDCG@10 on long docs.
Backend
- chunking.py: new chunk_pages_with_spans() that joins all pages into a
single full_doc, runs the existing recursive splitter, and returns
per-chunk (char_start, char_end) offsets. Page-number metadata is
recovered by overlap with the original page ranges so PDF citations
still work. Existing chunk_pages() unchanged.
- embeddings.py: new late_chunk_encode(doc_text, char_spans). Tokenizes
the doc with return_offsets_mapping, runs the underlying transformer
to get per-token last_hidden_state, then mean-pools per chunk span.
When the doc exceeds the embedder's context, falls back to windowed
late chunking with a 512-token overlap so cross-window context is
partially preserved.
- ingestion.py _subprocess_worker: branches on chunking_strategy.
'late' path: chunk_pages_with_spans -> late_chunk_encode -> one big
chunks_batch message. 'standard' path unchanged. Both reuse the same
parent-side pump.
- ingestion.enqueue_ingestion: new chunking_strategy + mode kwargs;
defaults to 'standard' / 'text' for legacy callers. embedder model
resolved via resolve_embedder() from the (mode, strategy) matrix.
- routes/rag.py: KB-doc upload reads chunking_strategy + mode from the
KB row (defensive .get for pre-Phase-3 schemas) and threads them
through _start_ingestion.
Frontend
- kb-create-dialog.tsx: new "Chunking strategy" select with Standard /
Late options. Embedding-model placeholder switches to nomic when
Late is picked. createKB request now carries chunking_strategy.
- kb-list.tsx + chat-settings-sheet.tsx: small "⚡ Late" badge next to
late-chunking KB names in the settings KB list and the chat sidebar
dropdown so users see the mode at a glance.
Tests
- test_rag_late_chunking.py: pure-python tests for chunk_pages_with_spans
(chunks index back into full_doc; page numbers inherited by overlap;
pages joined with blank line). A server-marked test loads
all-MiniLM-L6-v2 to exercise late_chunk_encode end-to-end.
No multimodal yet; that's Phase 3B-multimodal (next PR).
Phase 3 lays two orthogonal per-KB knobs in the data and API layers so
follow-up commits (Phase 3B-late, Phase 3B-multimodal) only need to add
their code path and UI selector, not schema or types.
Schema (studio/backend/storage/studio_db.py)
- rag_knowledge_bases gains chunking_strategy ('standard'|'late', default
'standard') and mode ('text'|'multimodal', default 'text'). Both are
immutable after KB creation — changing either invalidates existing
chunks because they were ingested through a specific pipeline.
- rag_chunks gains kind ('text'|'image'|'caption', default 'text'),
image_path (NULLABLE), linked_chunk_id (NULLABLE) — used by Phase
3B-multimodal to pair image chunks with their captions.
- Idempotent ALTER TABLE additions for existing installs (mirrors the
chat_threads display_name / *_code_exec_container_id pattern earlier
in the file).
API (studio/backend/routes/rag.py)
- ChunkingStrategy + KBMode Literal aliases.
- CreateKBRequest accepts both fields with backward-compat defaults.
- KBResponse exposes both.
- _validate_mode_combo rejects (multimodal, late) with 400 — no public
open-weight embedder supports both at once. Surface the constraint
early rather than failing silently during ingestion.
Config (studio/backend/utils/rag/config.py)
- RAG_EMBEDDER_MATRIX dict keyed by (mode, strategy) → embedder name.
- resolve_embedder() helper falls back to RAG_EMBEDDING_MODEL for legacy
KBs that pre-date the columns.
- (multimodal, late) intentionally absent.
Frontend (studio/frontend/src/features/rag/)
- api/rag-api.ts: ChunkingStrategy + KBMode types; KnowledgeBase
interface and createKnowledgeBase request type updated.
- stores/rag-store.ts: createKB signature uses the shared request type.
No user-visible UI changes yet — the only currently-usable combination
is (text, standard), so adding one-option selectors would be UX noise.
Phase 3B-late and Phase 3B-multimodal each add the relevant selector
option as part of shipping the code path.
Replace bare-pypdf/python-docx/BeautifulSoup extraction with Markdown-
preserving parsers so the chunker can split on real heading boundaries
instead of running paragraphs together.
Parsers
- pdf.py: pymupdf + pymupdf4llm.to_markdown() per page; pypdf kept as
fallback when pymupdf can't open the file.
- docx.py: mammoth.convert_to_html() + markdownify, with an explicit
style_map so Title/Heading 1..6 become h1..h6 in the output.
- html.py: BeautifulSoup pre-scrub (drop script/style) then markdownify
so <h*>, <table>, <ul> convert faithfully.
- text.py: signature update only; TXT/MD pass through unchanged.
- parsers/__init__.py: new ParsedImage + ParseResult dataclass; parse()
signature is now parse(path, *, want_images=False) -> ParseResult.
ParseResult is iterable over .pages for backward compat.
Chunker
- chunking.py: prepend Markdown heading separators ("\n# " .. "\n#### ")
to the priority list so heading-aware splits happen for free once the
parsers emit Markdown.
Ingestion
- ingestion.py: single call site updated to consume ParseResult.pages.
Deps (no-torch-runtime.txt)
+ pymupdf>=1.24, pymupdf4llm>=0.0.17, mammoth>=1.7, markdownify>=0.13
- pypdf kept as a fallback path.
Tests
- test_rag_parsers.py asserts Markdown headings survive PDF/DOCX/HTML
extraction; also exercises ParseResult iteration backward-compat.
- test_rag_chunking.py: new case verifying chunks start at Markdown
heading boundaries when the input is Markdown.
Foundation for Phase 3B-late (heading-aware spans for late chunking)
and Phase 3B-multimodal (want_images=True enables image extraction in
the same parser layer). No schema or opt-in flags in this commit.
Bundles three independent CI regressions hitting the maintainer PR
backlog. Each one is verified end-to-end on a staging fork against
real Ubuntu / macOS / Windows GitHub-hosted runners before this
lands.
1. Windows --no-torch install: pydantic + pydantic-core drift to
incompatible versions under `uv pip install --no-deps -r
no-torch-runtime.txt` because pip resolves each independently
from latest. pydantic.VERSION 2.13.4 pins pydantic-core==2.46.4
but pydantic-core 2.47.0 was the freshest published wheel, so
`import pydantic` raised
`SystemError: pydantic-core 2.47.0 is incompatible with the
current pydantic version`. Resolve pydantic WITH deps in a
focused pip call (install.sh, install.ps1,
install_python_stack.py) before the --no-deps no-torch-runtime
pass so pip pins pydantic-core to the version pydantic declares.
pydantic's transitive deps (annotated-types, pydantic-core,
typing-extensions, typing-inspection) are torch-free. Drop the
redundant `Patch Studio venv with full typer / pydantic dep
trees` workaround from the four Windows smoke YAMLs.
Supersedes #5733 + #5734.
2. Linux Studio Update CI: upstream llama.cpp b9261+ split each
binary's entry code into a paired `libllama-<binary>-impl.so`
shared library. `llama-server` and `llama-quantize` NEEDED-link
against `libllama-server-impl.so` / `libllama-quantize-impl.so`
with RUNPATH `$ORIGIN`, so the prebuilt overlay must copy those
alongside the binaries. Without that, ldd reports them missing,
preflight rejects, the installer falls back to source build, and
studio-update-smoke annotates `setup.sh idempotency regressed`.
Add `libllama-*-impl.so*` to the Linux runtime patterns and lock
the pattern in test_rocm_support.TestRuntimePatterns.
3. Mac Studio UI Chat: change-password submit clicked while
disabled. The disable gate only checked new + confirm password
length, but Playwright's first click landed before the
current-password field's React state had committed, so the form
was simultaneously logically-invalid (current_password empty) and
the button was disabled. Tighten the gate to require
`currentPassword.length >= 8` and mirror the same check in the
submit handler so Enter / autofill cannot bypass.
Supersedes #5738.
* Studio: PDF / document attachments for Anthropic + OpenAI
Studio's local-GGUF chat already supports image attachments via the
`image_url` content part shape. PDFs and other documents had no
plumbing for the external-provider path: there was no normalised
content type the frontend could send that translated to Anthropic's
native `document` block or OpenAI's `input_file`.
Add a Studio-side `input_document` content part on assistant /
user messages with three shapes:
{type: "input_document",
file_data: "data:application/pdf;base64,<DATA>",
filename?: "name.pdf",
media_type?: "application/pdf"}
{type: "input_document",
file_url: "https://example.com/doc.pdf",
filename?: "doc.pdf"}
Translation:
- Anthropic Messages API: emits a `document` block with
`{source: {type:"base64", media_type, data}}` or
`{source: {type:"url", url}}`, plus an optional `title` from
`filename`. PDFs are extracted server-side by Anthropic per their
vision/document docs and counted toward input tokens.
- OpenAI Responses API: emits `{type:"input_file", file_data |
file_url, filename?}`. PDFs are extracted server-side.
Empty / unparseable `input_document` parts are silently dropped so
a malformed frontend payload can't blow up the request.
Tests:
- New `test_multimodal_document.py` with 6 cases pinning the
outbound body shape for base64 + URL inputs on both providers,
and the empty-part drop behavior on both.
- The Anthropic assertions strip the prompt-cache wrapper
(`cache_control:{type:ephemeral}` that the tail-message caching
layer adds) before comparing the document core fields, so this
test stays focused on the translation, not the caching layer.
Live verified end-to-end against both providers: a 363-byte
single-page "HELLO" PDF, base64-encoded, attached as a `document`
block to Opus 4.7 and as an `input_file` to gpt-5.5. Both models
correctly extracted the word "HELLO" from the PDF.
Follow-up (out of scope):
- Pydantic schema entry on ChatMessage.content for `input_document`
(today it rides through because ChatCompletionRequest uses
extra=allow). Will tighten when the frontend attach button lands.
- Frontend file-picker UX for non-image attachments on the external
provider path.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: gate empty-content msg + skip empty data-URI payload
Gemini High + Codex P2 on PR #5689:
1. Anthropic translation appended an empty `anthropic_parts` array
when every part was dropped (e.g. user sent only an unparseable
input_document). Anthropic 400s on "messages.N.content: at least
one block is required". Skip the whole-message append when no
parts survived. The OpenAI Responses path already had the
equivalent guard, so this brings the two providers into parity.
2. `data:application/pdf;base64,` with no payload (or whitespace-only)
parses to an empty `source.data` string. Anthropic rejects that
with 400 as well. Skip the document block before constructing it.
Plus 2 new test cases pinning both behaviors:
- `test_anthropic_empty_only_document_drops_whole_message`: confirms
a turn whose only content is an unparseable input_document does
NOT make it onto the outbound `messages` array.
- `test_anthropic_empty_data_uri_payload_is_dropped`: confirms an
empty-payload data-URI is filtered out at translation time.
(Note re: gemini's other High note about adding `input_document` to
the Pydantic ContentPart union -- ChatCompletionRequest is configured
with `extra=allow` so the part rides through today. Tightening the
union belongs with the frontend attach-button PR that surfaces the
field; called out as follow-up in the PR description.)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: register input_document in ContentPart + builder
Reviewer caught that the translation code on the external_provider
side was unreachable from a real ChatCompletionRequest:
- ContentPart is a discriminated Union of (text, image_url) only, so
any `{"type": "input_document", ...}` part was rejected by Pydantic
at request parsing with a discriminator error before the helper
could see it.
- _build_external_messages in routes/inference.py only walked text
and image_url parts, so even with a permissive schema the document
parts would have been silently dropped instead of forwarded to
the per-provider translator.
Fixes:
- Add InputDocumentContentPart with optional file_data / file_url /
filename / media_type and Tag("input_document") on the Union.
- Extend _build_external_messages to pass input_document through as
a plain dict for vision-capable providers (so external_provider's
existing Anthropic `document` and OpenAI Responses `input_file`
mappers actually run) and strip them on non-vision providers.
Tests added: schema accepts input_document, builder passes it to
vision providers, builder strips it on non-vision providers.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: validate file_data before preferring over file_url
Codex P2 caught that the OpenAI input_document translator treats any
truthy file_data as valid and never falls back to file_url. That
means a malformed `data:application/pdf;base64,` (empty payload) or
a whitespace-only data URI gets forwarded as `file_data=""` and
400s the whole turn, AND silently discards a perfectly recoverable
file_url on the same part.
Mirror the Anthropic-side guard onto the OpenAI Responses path:
treat any "data:" URI with no actual base64 payload as missing and
fall through to file_url. Standalone-empty data URIs (no fallback)
are dropped entirely instead of being sent to the wire.
Tests added: empty data URI + valid file_url -> file_url wins,
whitespace-only data URI + valid file_url -> file_url wins,
empty data URI without fallback -> part is dropped.
* Address review: Anthropic side also falls back to file_url on empty data URI
Codex P2 follow-up to my earlier fix: I added the empty-data-URI ->
file_url fallback to the OpenAI Responses translator but missed
the Anthropic translator, which still `continue`d on empty payloads
and discarded an otherwise valid file_url on the same part. Result:
when the frontend supplied both file_data (placeholder / broken)
AND a working file_url, Anthropic silently lost the attachment;
when the message contained only that part, the whole message could
be dropped before reaching the wire.
Mirrored the OpenAI guard: any "data:" URI with no actual base64
payload (`data:application/pdf;base64,` or whitespace-only) is
treated as missing, and the file_url branch takes over. The
all-parts-dropped guard further down already handles the
no-fallback case.
Tests added: empty data URI + valid file_url -> URL source on the
wire with the filename preserved; whitespace-only data URI + valid
file_url -> URL source on the wire.
* Address review: gate input_document passthrough to anthropic + openai
Codex P1: only `_stream_anthropic` and `_stream_openai_responses`
have explicit translation logic for input_document parts (the former
maps to {type:"document", source:...}, the latter to
{type:"input_file", file_data|file_url}). Every other provider
(gemini / mistral / kimi / openrouter / deepseek / qwen / custom)
goes through the generic /chat/completions passthrough that forwards
`messages` verbatim, so any input_document part on a non-vision
route on those providers would 400 with an unknown content_part
type.
Added `_INPUT_DOCUMENT_PROVIDERS = frozenset({"anthropic", "openai"})`
constant and gated the pass-through branch on `provider_type in
_INPUT_DOCUMENT_PROVIDERS`. Every other provider strips the part
(text content survives). Threaded provider_type through from
_proxy_to_external_provider's call site.
Tests updated: vision + provider in {anthropic, openai} still
forwards; six unmapped providers (gemini/mistral/kimi/openrouter/
deepseek/qwen) strip the part; missing provider_type strips
defensively. The existing non-vision drop test still passes.
* Fix stale web_fetch tool-version assertion after merging main
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: wire OpenAI Responses server-side context compaction
The OpenAI Responses API accepts a `context_management` field that
enables server-side compaction. When the rendered prompt crosses the
configured threshold, the API runs a server-side compaction step and
the request continues against the compacted prefix. No beta header
and no dated version pin are required, per the docs.
Changes:
- Add `compaction_threshold: Optional[int]` (ge=1_000, le=2_000_000)
to ChatCompletionRequest. Thread through `routes/inference.py` ->
`stream_chat_completion` -> `_stream_openai_responses`.
- In `_stream_openai_responses`, when threshold is set AND the base
URL points at cloud OpenAI (api.openai.com), attach
`context_management: [{type:"compaction", compact_threshold:N}]`
to the outbound body. Non-cloud bases (ollama, llama.cpp, "custom"
presets) silently drop the field so we don't 400 those servers.
- Add `test_openai_compaction.py` with 4 cases: cloud OpenAI sets
the field verbatim, low-threshold probe passes through (we don't
clamp on the OpenAI side because the API accepts whatever),
non-cloud base drops the field, omitted threshold leaves body
untouched.
Live verified against the real OpenAI API on gpt-5.5:
`context_management:[{type:"compaction", compact_threshold:200000}]`
returns 200 with no error.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: accept Azure OpenAI base URLs + raise compaction floor
Two reviewer follow-ups on the OpenAI compaction PR:
1. The `is_openai_cloud = "api.openai.com" in self.base_url` check
excluded Azure OpenAI Foundry, even though Azure exposes the
same /v1/responses extensions (context_management,
prompt_cache_retention, container shell). Users on Azure saw
their compaction toggle silently no-op. Broadened the check to
also match `*.openai.azure.com` and made it case-insensitive so
URLs copy-pasted from the Azure portal still resolve. Non-cloud
OpenAI-compatible servers (ollama / llama.cpp / vLLM / "custom"
preset) still fall outside the gate.
2. The schema floor on compaction_threshold was ge=1_000, which is
well below the upstream Responses API's effective minimum
(vercel/ai#12486, langchain-ai/langchain#35464 report
`compact_threshold is not enabled` 400s on Azure at 100k; cloud
uses 200k as the canonical example). Raised the floor to 10k
so obvious typos surface as a clean 422 from FastAPI rather than
an opaque upstream 400 the user has to debug from the SSE
stream.
Tests added: Azure base URL carries both context_management and
prompt_cache_retention; mixed-case Azure URLs match; schema rejects
9_999 and accepts 10_000.
* Address review: drop schema-level compaction floor (cross-provider regression)
Codex P2 follow-up on the previous floor bump: ge=10_000 was
enforced globally at the ChatCompletionRequest layer, but the field
is documented as a no-op on every non-cloud OpenAI base and every
non-OpenAI provider. With the global floor, an Anthropic / ollama
/ llama.cpp / custom request that happens to carry compaction_threshold
below 10k was rejected with 422 at request validation time instead
of being silently ignored as the description promised.
Reverted the schema floor to ge=1 (any positive int) and rewrote
the description to call out per-provider routing: OpenAI cloud's
effective floor is around 200k and surfaces upstream 400s below
that; _stream_anthropic clamps sub-50k values up. Per-provider
helpers stay the single source of truth on the floor.
Test updated to pin: zero is still rejected, but every positive
value (1, 5_000, 9_999, 10_000, 200_000) passes schema validation.
* Address CodeQL: hostname-anchored OpenAI cloud detection
CodeQL py/incomplete-url-substring-sanitization fired on
`".openai.azure.com" in _base`. An attacker who controls the
configured base_url could slip cloud-only request body fields
(prompt_cache_retention, context_management compaction, container
shell) to an arbitrary server with:
https://evil.com/api.openai.com/v1https://api.openai.com.attacker.com/v1https://attacker.com/.openai.azure.com/v1https://my-resource.openai.azure.com.attacker.com/openai/v1
Replaced the substring check with a `_is_openai_family_cloud`
helper that runs urllib.parse.urlparse on the URL and matches the
lowercased hostname exactly (`api.openai.com`) or via `endswith`
on the leading-dot suffix (`.openai.azure.com`). Both halves are
host-anchored so path / fake-subdomain bypasses fail.
Test added: every attacker-controlled bypass shape above must NOT
carry context_management OR prompt_cache_retention on the wire.
Existing Azure and openai.com tests still pass.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: scope compaction_threshold description to OpenAI on this branch
Codex P2: the field description on this PR mentioned Anthropic
compaction behavior, but the Anthropic wiring lives on PR 5686
(separate branch). On feat/openai-compaction alone, _stream_anthropic
has no compaction_threshold parameter, so the field is silently
ignored for Anthropic requests and the doc claim was misleading.
Trimmed the description to OpenAI cloud + Azure Foundry only on
this branch. PR 5686 already re-adds the Anthropic clause via its
own change, so the rebase / merge order on main will land the
combined description naturally once both PRs ship.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
