Auto-downloading a 1.1 GB cross-encoder at every studio start is
wrong for users who never use rerank — reranker is opt-in by design.
Move the precache from a startup daemon thread to an explicit
POST /api/rag/reranker/precache endpoint, and have the chat settings
sheet call it the moment the 'Use reranker' switch is flipped on.
- Backend: drop the startup _precache_reranker thread; add the
/api/rag/reranker/precache route that calls precache_reranker().
- Frontend: new precacheRagReranker() in rag-api, wired into the
Switch's onCheckedChange so the download runs synchronously
with a loading toast. On success: 'Reranker ready'. On failure:
error toast + auto-flip the switch back off so the next query
doesn't trigger another long hang.
First toggle-on pays the 1.1 GB download once; subsequent toggles
hit the HF cache and return ~instantly.
The reranker model (BAAI/bge-reranker-base by default, ~1.1 GB) was
never precached, so the first user-facing rerank call paid the full
download cost — which on slow connections looked like a hang and got
retried by upstream timeouts. The deprecation warning that surfaced
during the hang was actually from sentence-transformers internals
firing while the download was still in flight.
Mirror the precache_helper_gguf pattern: add precache_reranker() that
calls snapshot_download in a daemon thread at FastAPI startup. The
first opt-in rerank now finds the weights already on disk and only
pays the in-process model load.
Also tighten the loader:
- explicit device selection (cuda when torch.cuda.is_available,
else cpu) so we don't rely on sentence-transformers auto-detect
behaviour that has historically picked cpu under odd
CUDA_VISIBLE_DEVICES configs;
- structlog-shaped logs with elapsed_seconds around load + predict
so a real runtime hang is visible in /tmp/studio.log with
'RAG reranker predict starting' / 'RAG reranker predict done'.
The RAG ingestion toast stack was pinned to bottom-right and rendered
its title and progress text with gap-0.5, so the stage label and
percentage rendered close enough to look concatenated (e.g.
'indexing5%') on narrow widths. The default Sonner toaster also
defaulted to bottom-right, so other notifications were inconsistent
with the new layout.
- Move the ingestion toast stack to top-right.
- Set Sonner's default position to top-right.
- Bump vertical gap between title and progress (gap-1.5).
- Add horizontal gap-3 between stage label and percentage, plus
shrink-0 + tabular-nums on the percent so it never collides with
the (truncatable) stage label.
If RAG is toggled on but the active scope (thread or KB) has zero
indexed documents, exposing search_knowledge_base to the model just
wastes a tool-call turn — the model calls the tool, gets back 'no
matching chunks', and has to re-plan. The system prompt nudge that
instructs the model to call the tool before answering is similarly
counterproductive.
Frontend: before computing ragToolPathTaken in chat-adapter, fetch the
document list for the current scope (KB or thread) and require docs to
exist. The flag now gates both the system prompt injection and the
enabled_tools list. Defensive fallback: if the docs lookup fails the
flag stays true so we don't silently swallow RAG.
Backend: add _drop_rag_tool_if_scope_empty in routes/inference.py that
counts rag_documents for the request's rag_scope and strips
search_knowledge_base from the tool list when 0. Applied at both
chat-completion tool-filter sites so the protection works regardless
of which streaming path serves the request.
Scores were only ever useful for debugging; surfacing them in chunk
cards (score X.XXX · dense Y.YYY) and citation hovers made the UI
noisy without giving the user anything actionable. Drop them in three
places:
- Backend search_knowledge_base no longer emits score / dense_score
attributes on the <chunk> tags fed to the LLM; the tool description
is updated to match.
- Chunk-card metadata in the assistant-ui tool result strips score /
dense lines.
- Source-badge hover tooltips drop the 'score N' meta line.
Also remove the 'Min relevance' slider from the chat settings sheet.
The backend min_score field stays plumbed (default 0 = no filter) so
the threshold can be re-exposed later or driven programmatically.
Captions were appended at the bottom of the page text, so the chunk
containing 'Figure 1: Asymmetries ...' got chunked separately from
'**Figure**: Flowchart with ...' on the same page. Retrieval surfaced
the caption-text chunk but the VLM description landed in a different
chunk, leaving the LLM without the visual content right next to the
figure label.
Splice each VLM caption right after the matching 'Figure N:' (or
'Table N:') line as '**Figure N description**: ...', so:
- The figure-boundary chunker now keeps both the original in-PDF
caption AND the VLM description in the same chunk (which starts
with 'Figure N:').
- Multi-figure pages get per-figure attribution — the prefix
'Figure N description' lets the LLM tell two figures on the same
page apart, even though the bbox renderer still emits one image
per page today (multi-figure clustering is a follow-up).
- When the page text has no figure lines (DOCX/HTML/TXT or rare
PDF layouts) the old end-of-page appendix is kept as a fallback.
page.get_images() only returns raster blobs embedded in the PDF's
resource dictionary, so vector schematics like Figure 1 — drawn purely
with paths/lines — were never extracted, and the VLM only ever saw
incidental embedded photos that happened to live near figures.
Replace the xref-based extraction with bbox rendering: union the
bounding rects of all vector drawings and raster image_info entries on
each page, expand a few points, and render the region with
get_pixmap(clip=bbox, matrix=2x). The captioner now receives the
actual figure — schematic arrows, box labels, legend text, and any
inset photos — and produces a caption that describes the figure as a
whole, not just one embedded sub-image.
Also sharpen the captioner prompt: explicitly tell the VLM the image
is a single figure cropped from a PDF page, and not to describe page
chrome or body paragraphs.
Dense vectors don't preserve numbers (BGE-small treats 'Figure 1' and
'Figure 10' as nearly identical), so a query like 'what does Figure 1
show' got out-ranked by chunks describing other figures that share more
vocabulary with the question — even after the figure-boundary chunker
ensured Figure 1's chunk started with the literal caption.
Detect 'Figure N' / 'Table N' (numbered, decimal, appendix-style)
references in the query, look up chunks that start with those captions
directly, and feed the result as a third RRF source. RRF gives them
rank-0 in the third ranking and the fused score lifts them above the
dense-vocabulary noise. No-ops when the query has no figure ref.
Dense embedders mean-pool over a whole chunk, so a 'Figure 1:' caption
buried at the end of a 500-token body chunk gets washed out by the
surrounding theory text and never surfaces for queries about that
figure. Pre-split each page's markdown at the start of every
Figure/Table caption line so the caption anchors its own chunk, which
gives both BM25 and the dense vector a focused, figure-dominated
target. Handles numbered, decimal, and appendix-style labels
(Figure 1, Figure 1.2, Figure B.1, Table 4, Fig./Tab. abbreviations).
Reasoning models (gemma-4, qwen3-thinking) burn the entire max_tokens
budget on <thinking> output and return empty visible content, so the
captioner produced zero captions for every image. Pass
chat_template_kwargs={enable_thinking: false} per-request to skip the
reasoning phase, and bump max_tokens 120 -> 200 as headroom.
Both modules used stdlib logging.getLogger which is not bridged to the
project's structlog config, so every probe / captioner log was silently
dropped. Switch to loggers.get_logger and convert %-format calls to
structlog kwargs so the captioning path becomes observable.
Four valid review comments from gemini-code-assist[bot] on #5759:
1. core/rag/bm25.py:_load — wrap bm25s.BM25.load + json.loads with
specific exception handlers (FileNotFoundError, OSError,
JSONDecodeError, ValueError) and log a warning instead of
propagating a 500. Corrupt/partial bm25 dirs now degrade to
empty-search rather than crashing the request.
2. core/rag/tool.py was importing _resolve_scope_embedder from
routes/rag.py — a layering violation (core depending on
routes). Move the resolver into a new core/rag/scope.py module
along with the chat-settings key constants; routes/rag.py
now re-imports it under the same name. Same behaviour, no
cycle, one source of truth for the resolution logic.
3. core/rag/bm25.py:rebuild_index — call delete_scope before saving
the new index so stale files from a previous build (or a
bm25s naming change) never coexist with current files. The
library's save() doesn't unlink files it doesn't write.
4. routes/rag.py:_save_upload was running f.write() synchronously
inside an async def. Switch to anyio.open_file() so each chunk
write runs in a worker thread instead of blocking the event
loop on multi-MB uploads. Cleanup unlink happens after the
async-with closes the handle so Windows is happy.
Skipped one (vector_store.py:133 'hasattr query_points' redundancy)
— that comment was on the pre-rewrite Qdrant code; the file is
now sqlite-vec backed and the hasattr check is gone.
core/rag/db.py, vector_store.py, tool.py, bm25.py, and reranker.py
all run only in the FastAPI parent process. Switch their loggers
from Python stdlib to studio's structlog get_logger so their output
shows up in the same JSON stream as the rest of the backend (the
request_completed / RAG search lines).
embeddings.py and ingestion.py stay on stdlib because they execute
inside the mp.spawn ingestion subprocess, which doesn't inherit the
parent's structlog configuration.
asg017/sqlite-vec is Apache-2.0 and OSI-approved. Replaces
qdrant-client (~30 MB) with a small SQLite extension loaded into a
dedicated rag.db file. Single file holds RAG vectors; bm25s indexes
and chat-side studio.db are unaffected.
- New core/rag/db.py owns the rag.db connection and sqlite-vec load.
Extension load runs once at first open. Process-wide singleton
protected by a lock; check_same_thread=False + WAL handles the
FastAPI thread pool.
- core/rag/vector_store.py keeps the same public API
(ensure_collection / upsert_chunks / search / collection_exists /
delete_scope / delete_document) so callers in routes/rag.py,
core/rag/ingestion.py, core/rag/tool.py, and core/rag/retrieval.py
don't change. ensure_collection is now a no-op; collection_exists
returns True iff the scope has at least one indexed vector.
- search uses sqlite-vec's vec_distance_cosine and converts distance
to similarity in [0, 1] so the per-scope min_score threshold
semantics stay identical.
- Mixed-dim scopes coexist behind WHERE scope = ? — the per-scope
embedder resolver guarantees one embedder per scope.
- requirements/rag.txt swaps qdrant-client for sqlite-vec.
- utils/paths/storage_roots.py drops rag_vectordb_root() (the old
qdrant directory); rag.db lives directly under rag_root().
- Rewritten tests/python/test_rag_vector_store.py for the new
semantics (collection_exists tracks populated scopes; new tests
for filtered search and upsert conflict resolution).
Python build requirement: connection.enable_load_extension(True)
must be available. install.sh creates the venv via uv-managed
python-build-standalone, which is compiled with
--enable-loadable-sqlite-extensions, so this works on standard
installs. core/rag/db.py raises an actionable error on the rare
custom-interpreter case.
The query was always going through the default text embedder
(bge-small, 384-d) regardless of how the scope was ingested. A
multimodal thread indexed by Qwen3-VL (2048-d) crashed at cosine
similarity with 'shapes (104,2048) and (384,) not aligned'.
Resolve the scope's embedder at search time:
- kb_<id> -> rag_knowledge_bases.embedding_model column
- thread_<id> -> thread settings (with fallback to defaults +
RAG_EMBEDDER_MATRIX matrix lookup)
Pass it through retrieve_hybrid/retrieve_dense to embeddings.encode
so the query lands in the same vector space as the docs. Both the
/api/rag/search route and the search_knowledge_base tool use the
same resolver.
Qwen3-VL-Embedding-2B's modules.json references
sentence_transformers.base.modules, introduced after 5.2.0.
Unlike BGE-VL, Qwen3-VL has no fragile custom forward subclass to
break against the newer ST internals, so the bump should land
cleanly for the multimodal path.
Built on Qwen3 (not CLIP), so the 77-token text cap that bit
BGE-VL-base is gone — long chunks embed losslessly. Loads via
vanilla SentenceTransformer with trust_remote_code, no custom
adapter needed. 2048-d shared text+image space.
Adds qwen-vl-utils>=0.0.14 to rag.txt for image preprocessing,
required by the Qwen3-VL embedder family.
BGE-VL inherits CLIP's 77-token text positional embedding table —
longer chunks crash inside the text model with a shape mismatch.
Pre-tokenize with truncation=True, max_length=77 and call
get_text_features directly so the high-level encode() (which does
not truncate) is bypassed. Log when truncation happens — text
chunks beyond the cap are silently cut, so multimodal mode is
lossy on the text channel. Image channel is unaffected.
BGE-VL's sentence-transformers shim (bge_vl_clip_transformer.py)
is coupled to specific ST internals and broke across both the 5.2
and 5.3 pin attempts. The canonical load path documented at
bge-model.com is transformers.AutoModel with trust_remote_code +
model.set_processor() + model.encode(text=/images=).
- Wrap that path in _BGEVLAdapter exposing the slice of
SentenceTransformer API the ingester uses (encode for text or PIL
images, get_sentence_embedding_dimension, best-effort tokenize).
- Restore BAAI/BGE-VL-base in RAG_EMBEDDER_MATRIX.
- Revert sentence_transformers pin to 5.2.0 — no longer relevant
since the multimodal path no longer touches ST.
BGE-VL-base ships a custom Transformer subclass that's tightly
coupled to a specific sentence-transformers internal API — it
imports a module path missing in older ST and overrides forward()
expecting a return shape that changed in newer ST. We can't pin ST
to BGE-VL's exact version without risking the training/inference
paths that use the same library, so swap the multimodal default to
the canonical clip-ViT-B-32 which sentence-transformers wraps
natively (no trust_remote_code, no shim, ST-version-independent).
Same 512-d shared text+image space.
BGE-VL's custom modeling code imports
sentence_transformers.base.modules.transformer, which doesn't exist
in the 5.2.0 pin. Bump to >=5.3.0,<7 so the new Module package is
available.
BGE-VL (multimodal) and nomic-embed-text-v1.5 (late chunking) both
ship custom modeling code in their model repos; sentence-transformers
refuses to import the referenced module (e.g. bge_vl_clip_transformer)
without trust_remote_code=True. Safe to enable because the embedder
matrix is config-pinned — users don't supply arbitrary names.
