assistant-ui mints a fresh __LOCALID_* draft id on every page load, so
RAG docs uploaded under the previous draft id were orphaned after a
refresh — the doc panel queries useThreadDocuments(activeThreadId) and
the new id had nothing.
Persist activeThreadId in localStorage and, on the first settled render
after load, have ActiveThreadSync ask aui to switchToThread(persisted)
when it differs from the freshly-minted draft. Because the draft was
already persisted to the backend by initialize()/ensureThreadRecord
when its first doc was uploaded, the adapter's fetch() resolves it and
aui adopts it as mainThreadId. That keeps aui's mainThreadId and our
activeThreadId unified, so the earlier divergence (uploads under the
persisted id vs chat-completion reading aui's fresh id) can't recur —
unlike the reverted localStorage-only attempt, the chat-adapter's
unstable_threadId now equals the persisted id after the switch.
A one-shot ref ensures we only re-adopt on initial load; user-driven
new-chat / thread switches still flow through normally. If the
persisted draft was never initialized (no doc/message, not in the
backend), switchToThread rejects and we fall back to the fresh draft.
The two previous commits (41e43b6c7 and 3d3a00c2b) persisted
activeThreadId in localStorage so RAG docs would survive a page
reload. That broke fresh chats: stale localStorage values from a
prior session pinned activeThreadId to an old draft id, but the
chat-completion path reads aui's current mainThreadId via
unstable_threadId. The two diverged — uploads went under the stale
persisted id, the chat-completion turn looked up docs under the new
aui id, and nothing matched.
Revert the persistence + ActiveThreadSync guard. We're back to the
pre-fix behaviour where uploads-in-the-same-session work, and a
proper fix for the reload case (promote drafts to real chat_threads
rows on first doc upload so the id never changes) will land next.
ActiveThreadSync was reacting to aui's mainThreadId === null on mount
(aui hasn't booted yet) by calling setActiveThreadId(null), which
wiped the just-restored persisted draft id from localStorage and
emptied the doc panel for the user's thread. The previous fix only
covered the 'aui minted a different LOCALID' branch; it missed the
'mainThreadId is null while aui boots' branch.
Treat a null mainThreadId as a no-op for the sync. Explicit clears
(new chat, sidebar delete) keep going through setActiveThreadId(null)
directly, so this guard doesn't trap stale state — it just gives the
persisted draft id a chance to survive until aui finishes booting.
When the user uploads a doc to a brand-new chat (a draft thread with
an assistant-ui __LOCALID_* id), the backend stores rag_documents
rows scoped to that id. On page reload assistant-ui mints a fresh
__LOCALID_* for the new mainThreadId, so useThreadDocuments asks the
backend for docs under the NEW id and gets nothing, even though the
original rows are still on disk under the OLD id.
- chat-runtime-store: persist activeThreadId in localStorage via
a new CHAT_ACTIVE_THREAD_KEY, restore on init, save on every
setActiveThreadId call (including clears, which write '').
- ActiveThreadSync: when aui's freshly-minted mainThreadId is a
__LOCALID_* and we already have a persisted __LOCALID_* draft,
keep ours instead of overwriting. This only affects RAG/doc
lookup; aui's chat history for the new draft starts empty
either way, so there's no regression for users who don't have
attached docs.
User-initiated thread switches (new chat, switching to a saved
thread, deleting the current thread) all go through setActiveThreadId
with the new id (or null), so they correctly replace/clear the
persisted value.
RAG is opt-in but the chat store had a hydration-time migration that
silently flipped ragToolEnabled to true whenever a persisted ragSource
was anything other than 'off'. Plus there was no logic to reset the
pill across model loads, so the pill stayed on across sessions even
after the user explicitly disabled and re-enabled it.
- Remove the migration block in chat-runtime-store.hydrate — the
embedder warmup still runs when ragToolEnabled is genuinely
persisted true.
- In use-chat-model-runtime's load-success handler, call
setRagToolEnabled(false) (via the setter, so localStorage stays
in sync) immediately after the loaded-state setState batch. Every
fresh model load now starts with the pill off and the user must
toggle it explicitly.
get_reranker() acquires the module-level _lock and then, on first load,
calls unload() to clear any stale state before _load() instantiates the
CrossEncoder. unload() acquires the same _lock — but threading.Lock is
non-reentrant, so the second acquisition by the holding thread blocked
forever. Symptom: rerank=True hung the search_knowledge_base tool with
no further log output past 'rerank entered'.
Switch to threading.RLock so the same thread can re-enter without
blocking. unload()'s independent callers still work the same way; the
only behaviour change is that re-entrant acquisition from one thread
now succeeds.
When the reranker hung on rerank=True there were zero log lines after
'retrieved=N (no threshold)', which made it impossible to tell whether
the hang was in _load (CrossEncoder construction), in get_reranker's
lock acquisition, or in predict. Structlog routing may also be the
culprit since we never saw the 'Loading RAG reranker' info line.
Add unconditional stderr prints at each milestone — entered, device
resolved, before CrossEncoder, after CrossEncoder, rerank entered,
predict starting, predict done. These bypass any logger config and
show up directly in /tmp/studio.log next to the rest of the captured
stdout/stderr. Leaving structlog logger.info calls in place too so
the structured stream still gets the same data when routing works.
Auto-downloading a 1.1 GB cross-encoder at every studio start is
wrong for users who never use rerank — reranker is opt-in by design.
Move the precache from a startup daemon thread to an explicit
POST /api/rag/reranker/precache endpoint, and have the chat settings
sheet call it the moment the 'Use reranker' switch is flipped on.
- Backend: drop the startup _precache_reranker thread; add the
/api/rag/reranker/precache route that calls precache_reranker().
- Frontend: new precacheRagReranker() in rag-api, wired into the
Switch's onCheckedChange so the download runs synchronously
with a loading toast. On success: 'Reranker ready'. On failure:
error toast + auto-flip the switch back off so the next query
doesn't trigger another long hang.
First toggle-on pays the 1.1 GB download once; subsequent toggles
hit the HF cache and return ~instantly.
The reranker model (BAAI/bge-reranker-base by default, ~1.1 GB) was
never precached, so the first user-facing rerank call paid the full
download cost — which on slow connections looked like a hang and got
retried by upstream timeouts. The deprecation warning that surfaced
during the hang was actually from sentence-transformers internals
firing while the download was still in flight.
Mirror the precache_helper_gguf pattern: add precache_reranker() that
calls snapshot_download in a daemon thread at FastAPI startup. The
first opt-in rerank now finds the weights already on disk and only
pays the in-process model load.
Also tighten the loader:
- explicit device selection (cuda when torch.cuda.is_available,
else cpu) so we don't rely on sentence-transformers auto-detect
behaviour that has historically picked cpu under odd
CUDA_VISIBLE_DEVICES configs;
- structlog-shaped logs with elapsed_seconds around load + predict
so a real runtime hang is visible in /tmp/studio.log with
'RAG reranker predict starting' / 'RAG reranker predict done'.
The RAG ingestion toast stack was pinned to bottom-right and rendered
its title and progress text with gap-0.5, so the stage label and
percentage rendered close enough to look concatenated (e.g.
'indexing5%') on narrow widths. The default Sonner toaster also
defaulted to bottom-right, so other notifications were inconsistent
with the new layout.
- Move the ingestion toast stack to top-right.
- Set Sonner's default position to top-right.
- Bump vertical gap between title and progress (gap-1.5).
- Add horizontal gap-3 between stage label and percentage, plus
shrink-0 + tabular-nums on the percent so it never collides with
the (truncatable) stage label.
If RAG is toggled on but the active scope (thread or KB) has zero
indexed documents, exposing search_knowledge_base to the model just
wastes a tool-call turn — the model calls the tool, gets back 'no
matching chunks', and has to re-plan. The system prompt nudge that
instructs the model to call the tool before answering is similarly
counterproductive.
Frontend: before computing ragToolPathTaken in chat-adapter, fetch the
document list for the current scope (KB or thread) and require docs to
exist. The flag now gates both the system prompt injection and the
enabled_tools list. Defensive fallback: if the docs lookup fails the
flag stays true so we don't silently swallow RAG.
Backend: add _drop_rag_tool_if_scope_empty in routes/inference.py that
counts rag_documents for the request's rag_scope and strips
search_knowledge_base from the tool list when 0. Applied at both
chat-completion tool-filter sites so the protection works regardless
of which streaming path serves the request.
Scores were only ever useful for debugging; surfacing them in chunk
cards (score X.XXX · dense Y.YYY) and citation hovers made the UI
noisy without giving the user anything actionable. Drop them in three
places:
- Backend search_knowledge_base no longer emits score / dense_score
attributes on the <chunk> tags fed to the LLM; the tool description
is updated to match.
- Chunk-card metadata in the assistant-ui tool result strips score /
dense lines.
- Source-badge hover tooltips drop the 'score N' meta line.
Also remove the 'Min relevance' slider from the chat settings sheet.
The backend min_score field stays plumbed (default 0 = no filter) so
the threshold can be re-exposed later or driven programmatically.
Captions were appended at the bottom of the page text, so the chunk
containing 'Figure 1: Asymmetries ...' got chunked separately from
'**Figure**: Flowchart with ...' on the same page. Retrieval surfaced
the caption-text chunk but the VLM description landed in a different
chunk, leaving the LLM without the visual content right next to the
figure label.
Splice each VLM caption right after the matching 'Figure N:' (or
'Table N:') line as '**Figure N description**: ...', so:
- The figure-boundary chunker now keeps both the original in-PDF
caption AND the VLM description in the same chunk (which starts
with 'Figure N:').
- Multi-figure pages get per-figure attribution — the prefix
'Figure N description' lets the LLM tell two figures on the same
page apart, even though the bbox renderer still emits one image
per page today (multi-figure clustering is a follow-up).
- When the page text has no figure lines (DOCX/HTML/TXT or rare
PDF layouts) the old end-of-page appendix is kept as a fallback.
page.get_images() only returns raster blobs embedded in the PDF's
resource dictionary, so vector schematics like Figure 1 — drawn purely
with paths/lines — were never extracted, and the VLM only ever saw
incidental embedded photos that happened to live near figures.
Replace the xref-based extraction with bbox rendering: union the
bounding rects of all vector drawings and raster image_info entries on
each page, expand a few points, and render the region with
get_pixmap(clip=bbox, matrix=2x). The captioner now receives the
actual figure — schematic arrows, box labels, legend text, and any
inset photos — and produces a caption that describes the figure as a
whole, not just one embedded sub-image.
Also sharpen the captioner prompt: explicitly tell the VLM the image
is a single figure cropped from a PDF page, and not to describe page
chrome or body paragraphs.
Dense vectors don't preserve numbers (BGE-small treats 'Figure 1' and
'Figure 10' as nearly identical), so a query like 'what does Figure 1
show' got out-ranked by chunks describing other figures that share more
vocabulary with the question — even after the figure-boundary chunker
ensured Figure 1's chunk started with the literal caption.
Detect 'Figure N' / 'Table N' (numbered, decimal, appendix-style)
references in the query, look up chunks that start with those captions
directly, and feed the result as a third RRF source. RRF gives them
rank-0 in the third ranking and the fused score lifts them above the
dense-vocabulary noise. No-ops when the query has no figure ref.
Dense embedders mean-pool over a whole chunk, so a 'Figure 1:' caption
buried at the end of a 500-token body chunk gets washed out by the
surrounding theory text and never surfaces for queries about that
figure. Pre-split each page's markdown at the start of every
Figure/Table caption line so the caption anchors its own chunk, which
gives both BM25 and the dense vector a focused, figure-dominated
target. Handles numbered, decimal, and appendix-style labels
(Figure 1, Figure 1.2, Figure B.1, Table 4, Fig./Tab. abbreviations).
Reasoning models (gemma-4, qwen3-thinking) burn the entire max_tokens
budget on <thinking> output and return empty visible content, so the
captioner produced zero captions for every image. Pass
chat_template_kwargs={enable_thinking: false} per-request to skip the
reasoning phase, and bump max_tokens 120 -> 200 as headroom.
Both modules used stdlib logging.getLogger which is not bridged to the
project's structlog config, so every probe / captioner log was silently
dropped. Switch to loggers.get_logger and convert %-format calls to
structlog kwargs so the captioning path becomes observable.
Four valid review comments from gemini-code-assist[bot] on #5759:
1. core/rag/bm25.py:_load — wrap bm25s.BM25.load + json.loads with
specific exception handlers (FileNotFoundError, OSError,
JSONDecodeError, ValueError) and log a warning instead of
propagating a 500. Corrupt/partial bm25 dirs now degrade to
empty-search rather than crashing the request.
2. core/rag/tool.py was importing _resolve_scope_embedder from
routes/rag.py — a layering violation (core depending on
routes). Move the resolver into a new core/rag/scope.py module
along with the chat-settings key constants; routes/rag.py
now re-imports it under the same name. Same behaviour, no
cycle, one source of truth for the resolution logic.
3. core/rag/bm25.py:rebuild_index — call delete_scope before saving
the new index so stale files from a previous build (or a
bm25s naming change) never coexist with current files. The
library's save() doesn't unlink files it doesn't write.
4. routes/rag.py:_save_upload was running f.write() synchronously
inside an async def. Switch to anyio.open_file() so each chunk
write runs in a worker thread instead of blocking the event
loop on multi-MB uploads. Cleanup unlink happens after the
async-with closes the handle so Windows is happy.
Skipped one (vector_store.py:133 'hasattr query_points' redundancy)
— that comment was on the pre-rewrite Qdrant code; the file is
now sqlite-vec backed and the hasattr check is gone.
core/rag/db.py, vector_store.py, tool.py, bm25.py, and reranker.py
all run only in the FastAPI parent process. Switch their loggers
from Python stdlib to studio's structlog get_logger so their output
shows up in the same JSON stream as the rest of the backend (the
request_completed / RAG search lines).
embeddings.py and ingestion.py stay on stdlib because they execute
inside the mp.spawn ingestion subprocess, which doesn't inherit the
parent's structlog configuration.
asg017/sqlite-vec is Apache-2.0 and OSI-approved. Replaces
qdrant-client (~30 MB) with a small SQLite extension loaded into a
dedicated rag.db file. Single file holds RAG vectors; bm25s indexes
and chat-side studio.db are unaffected.
- New core/rag/db.py owns the rag.db connection and sqlite-vec load.
Extension load runs once at first open. Process-wide singleton
protected by a lock; check_same_thread=False + WAL handles the
FastAPI thread pool.
- core/rag/vector_store.py keeps the same public API
(ensure_collection / upsert_chunks / search / collection_exists /
delete_scope / delete_document) so callers in routes/rag.py,
core/rag/ingestion.py, core/rag/tool.py, and core/rag/retrieval.py
don't change. ensure_collection is now a no-op; collection_exists
returns True iff the scope has at least one indexed vector.
- search uses sqlite-vec's vec_distance_cosine and converts distance
to similarity in [0, 1] so the per-scope min_score threshold
semantics stay identical.
- Mixed-dim scopes coexist behind WHERE scope = ? — the per-scope
embedder resolver guarantees one embedder per scope.
- requirements/rag.txt swaps qdrant-client for sqlite-vec.
- utils/paths/storage_roots.py drops rag_vectordb_root() (the old
qdrant directory); rag.db lives directly under rag_root().
- Rewritten tests/python/test_rag_vector_store.py for the new
semantics (collection_exists tracks populated scopes; new tests
for filtered search and upsert conflict resolution).
Python build requirement: connection.enable_load_extension(True)
must be available. install.sh creates the venv via uv-managed
python-build-standalone, which is compiled with
--enable-loadable-sqlite-extensions, so this works on standard
installs. core/rag/db.py raises an actionable error on the rare
custom-interpreter case.
The query was always going through the default text embedder
(bge-small, 384-d) regardless of how the scope was ingested. A
multimodal thread indexed by Qwen3-VL (2048-d) crashed at cosine
similarity with 'shapes (104,2048) and (384,) not aligned'.
Resolve the scope's embedder at search time:
- kb_<id> -> rag_knowledge_bases.embedding_model column
- thread_<id> -> thread settings (with fallback to defaults +
RAG_EMBEDDER_MATRIX matrix lookup)
Pass it through retrieve_hybrid/retrieve_dense to embeddings.encode
so the query lands in the same vector space as the docs. Both the
/api/rag/search route and the search_knowledge_base tool use the
same resolver.
