Commit graph

3 commits

Author SHA1 Message Date
Daniel Han
ab0828b976 Studio: fix RAG correctness bugs
Backend:
- Deterministic SQLite connection cleanup. The RAG code used bare
  `with get_connection() as conn:`, which commits but never closes, leaning
  on GC to release handles (the rest of studio_db closes explicitly). Add a
  closing_connection() context manager that commits/rolls back like sqlite3's
  own manager and always closes, and route all 30 RAG call sites through it.
- filter_by_min_score no longer drops BM25-only and figure-ref hits. min_score
  is a cosine floor, so it now gates only hits that carry a dense_score;
  lexical and figure-ref hits (dense_score is None) pass through instead of
  being silently discarded when the floor is raised.
- Fix two tests that could not pass against the production code: the RRF
  fusion test asserted the wrong winner (c edges out b: 0.032266 vs 0.032258),
  and two tool-handler scope tests stubbed retrieve_hybrid without accepting
  the embedder_model kwarg the handler now passes (TypeError was swallowed,
  leaving captured["scope"] unset).

Frontend:
- Removing an in-flight upload chip now routes through the teardown thunk
  already registered for the aggregate-progress toast (abort, unsubscribe,
  release the index slot, delete the backend doc with the correct kb/thread
  scope key it closed over) and clears the toast entry. Deleting directly
  leaked the concurrency slot and hardcoded the thread scope, mis-targeting
  KB-scoped docs. Applied in both the composer hook and the compare-view
  composer; drop the now-vestigial chip-scope-key tracking and unused
  activeThreadId selectors. Add index-progress-store.remove(id).
2026-05-31 09:56:23 +00:00
Daniel Han
d1348cac3f Studio: tighten RAG code comments
Shorten and condense comments across the RAG backend, frontend, and
tests for readability. Comment text only; no code, strings, identifiers,
or logic changed. License headers and lint/type pragmas are preserved.
2026-05-31 08:31:08 +00:00
Roland Tannous
92994e8b83 Studio: add RAG with hybrid search, reranker, chat integration
Backend (studio/backend/):
- core/rag/: parsers (PDF/TXT/MD/DOCX/HTML via pypdf/python-docx/bs4),
  recursive token-aware chunker, embeddings singleton via
  FastSentenceTransformer.from_pretrained(for_inference=True), Qdrant
  local vector store, bm25s lexical index, RRF hybrid retrieval,
  spawn-subprocess ingestion job with SSE progress, optional
  CrossEncoder reranker (off-by-default).
- routes/rag.py: KB CRUD, doc upload (KB + per-thread), doc list/delete,
  ingestion SSE, hybrid+rerank search, thread-index list/clear.
- routes/chat_history.py: purge thread RAG artifacts on thread delete
  and clear-all (rag_documents has no FK cascade to chat_threads so
  uploads work on un-persisted threads).
- studio.db gains 4 RAG tables; storage_roots gains rag_*() helpers.
- auth/authentication.py: get_current_subject_sse accepts ?token=... so
  EventSource can stream ingestion progress.

Frontend (studio/frontend/):
- features/rag/: api client, Zustand store, hooks, dropzone, KB list,
  doc rows, ingestion-progress, thread-index list components.
- Settings dialog gains a Knowledge Bases tab (master/detail + thread
  documents list); /knowledge-bases deep-links to it.
- features/chat/: per-thread ragSource/enableRerank/ragTopK state in
  chat-runtime-store; Retrieval section in chat-settings-sheet with KB
  DropdownMenu (active highlight + per-row trash), thread doc list with
  Clear-thread-index button, RAG Top K slider, reranker toggle;
  chat-adapter retrieves before /v1/chat/completions and injects hits
  as a system block; shared-composer + button routes documents into
  pendingDocs (auto-uploads, send blocked while indexing).
2026-05-23 18:46:15 +04:00