diff --git a/studio/RAG_FAST_INDEXING.md b/studio/RAG_FAST_INDEXING.md new file mode 100644 index 0000000000..97757e3525 --- /dev/null +++ b/studio/RAG_FAST_INDEXING.md @@ -0,0 +1,47 @@ +# Studio RAG: fast incremental indexing + +Progress log for an optimization stacked on the `feature/rag` branch (PR #5759). The goal is to +keep that PR's UX and retrieval accuracy while cutting per-document indexing from ~1 minute to a +few seconds, validated through the real Studio UI. + +## Root cause of slow indexing (measured on feature/rag) + +1. **Embedder reload on every upload.** Ingestion spawns a fresh `spawn` subprocess per job + (`core/rag/ingestion.py` `_CTX.Process(target=_subprocess_worker)`), and the embedder is loaded + *inside* that subprocess (`get_embedder` -> `from unsloth import FastSentenceTransformer`). A + spawned interpreter re-imports unsloth and reloads the model from scratch every time, so the + ~12-30s cold load recurs on each upload, not just the first. `lifespan()` never warms it. +2. **Per-document BM25 rebuild.** On every document, `ingestion.py` reads *all* chunks in the scope + (`_all_scope_chunks`) and rebuilds the whole `bm25s` index (`bm25.rebuild_index`). That is + O(N^2) tokenization as a knowledge base fills. + +## The change (all behind a single `UNSLOTH_RAG_FAST=1` flag; off = byte-identical to PR) + +1. **Warm the embedder at startup.** `main.py` `lifespan()` warms `get_embedder()` in a daemon + thread (mirrors the existing GGUF precache thread). +2. **In-process ingestion.** Run the existing `_subprocess_worker` in a thread in the warm main + process instead of a fresh subprocess, so `get_embedder` returns the already-loaded singleton. + The worker, queue protocol, persistence and SSE progress are otherwise unchanged. +3. **Incremental SQLite FTS5 BM25.** Reimplement `core/rag/bm25.py` on an FTS5 virtual table in the + existing `rag.db`, with an incremental `add_chunks(scope, chunks)`. Ingestion inserts only the + new document's chunks (O(N) total) instead of rebuilding the scope. FTS5 `MATCH` returns only + matching rows (no zero-score pollution) and adds `porter` stemming. The dense leg already uses + sqlite-vec, so it is unchanged; RRF fusion in `retrieval.py` is unchanged. + +## Validation + +Two isolated Studios (separate `UNSLOTH_STUDIO_HOME` + port), same prebuilt frontend, same corpus, +same model. Baseline = `UNSLOTH_RAG_FAST` unset; improved = set. Driven through the real UI with +Playwright (`studio_test_kit`). Local GGUF model `unsloth/Qwen3.5-9B-GGUF` for the chat tool path. + +Authoritative corpus (stable URLs): arXiv 1706.03762 (Attention), 1810.04805 (BERT), +2005.11401 (RAG); RFC 9110. Retrieval scored independently of generation with gold queries. + +Metrics: per-document upload->ready latency (cold + warm), Recall@1/3/5 and MRR via `POST +/api/rag/search`, scaling (per-doc index time vs document count), and end-to-end UI (upload -> +indexed -> RAG answer with citation). Robustness: delete-then-query, restart persistence, thread +isolation. + +## Results + +Pending - populated as runs complete (see commits below).