Commit graph

4 commits

Author SHA1 Message Date
Roland Tannous
86b52503dd Studio: trim verbose comments/docstrings across RAG code 2026-05-26 13:51:11 +04:00
Roland Tannous
7b3a13fea4 Studio: address gemini-code-assist PR review
Four valid review comments from gemini-code-assist[bot] on #5759:

1. core/rag/bm25.py:_load — wrap bm25s.BM25.load + json.loads with
   specific exception handlers (FileNotFoundError, OSError,
   JSONDecodeError, ValueError) and log a warning instead of
   propagating a 500. Corrupt/partial bm25 dirs now degrade to
   empty-search rather than crashing the request.

2. core/rag/tool.py was importing _resolve_scope_embedder from
   routes/rag.py — a layering violation (core depending on
   routes). Move the resolver into a new core/rag/scope.py module
   along with the chat-settings key constants; routes/rag.py
   now re-imports it under the same name. Same behaviour, no
   cycle, one source of truth for the resolution logic.

3. core/rag/bm25.py:rebuild_index — call delete_scope before saving
   the new index so stale files from a previous build (or a
   bm25s naming change) never coexist with current files. The
   library's save() doesn't unlink files it doesn't write.

4. routes/rag.py:_save_upload was running f.write() synchronously
   inside an async def. Switch to anyio.open_file() so each chunk
   write runs in a worker thread instead of blocking the event
   loop on multi-MB uploads. Cleanup unlink happens after the
   async-with closes the handle so Windows is happy.

Skipped one (vector_store.py:133 'hasattr query_points' redundancy)
— that comment was on the pre-rewrite Qdrant code; the file is
now sqlite-vec backed and the hasattr check is gone.
2026-05-25 16:43:39 +04:00
Roland Tannous
005234c953 Studio: route RAG parent-process loggers through structlog
core/rag/db.py, vector_store.py, tool.py, bm25.py, and reranker.py
all run only in the FastAPI parent process. Switch their loggers
from Python stdlib to studio's structlog get_logger so their output
shows up in the same JSON stream as the rest of the backend (the
request_completed / RAG search lines).

embeddings.py and ingestion.py stay on stdlib because they execute
inside the mp.spawn ingestion subprocess, which doesn't inherit the
parent's structlog configuration.
2026-05-25 15:25:11 +04:00
Roland Tannous
92994e8b83 Studio: add RAG with hybrid search, reranker, chat integration
Backend (studio/backend/):
- core/rag/: parsers (PDF/TXT/MD/DOCX/HTML via pypdf/python-docx/bs4),
  recursive token-aware chunker, embeddings singleton via
  FastSentenceTransformer.from_pretrained(for_inference=True), Qdrant
  local vector store, bm25s lexical index, RRF hybrid retrieval,
  spawn-subprocess ingestion job with SSE progress, optional
  CrossEncoder reranker (off-by-default).
- routes/rag.py: KB CRUD, doc upload (KB + per-thread), doc list/delete,
  ingestion SSE, hybrid+rerank search, thread-index list/clear.
- routes/chat_history.py: purge thread RAG artifacts on thread delete
  and clear-all (rag_documents has no FK cascade to chat_threads so
  uploads work on un-persisted threads).
- studio.db gains 4 RAG tables; storage_roots gains rag_*() helpers.
- auth/authentication.py: get_current_subject_sse accepts ?token=... so
  EventSource can stream ingestion progress.

Frontend (studio/frontend/):
- features/rag/: api client, Zustand store, hooks, dropzone, KB list,
  doc rows, ingestion-progress, thread-index list components.
- Settings dialog gains a Knowledge Bases tab (master/detail + thread
  documents list); /knowledge-bases deep-links to it.
- features/chat/: per-thread ragSource/enableRerank/ragTopK state in
  chat-runtime-store; Retrieval section in chat-settings-sheet with KB
  DropdownMenu (active highlight + per-row trash), thread doc list with
  Clear-thread-index button, RAG Top K slider, reranker toggle;
  chat-adapter retrieves before /v1/chat/completions and injects hits
  as a system block; shared-composer + button routes documents into
  pendingDocs (auto-uploads, send blocked while indexing).
2026-05-23 18:46:15 +04:00