Studio: RAG fast incremental indexing progress log

studio/RAG_FAST_INDEXING.md

@@ -0,0 +1,47 @@
+# Studio RAG: fast incremental indexing
+
+Progress log for an optimization stacked on the `feature/rag` branch (PR #5759). The goal is to
+keep that PR's UX and retrieval accuracy while cutting per-document indexing from ~1 minute to a
+few seconds, validated through the real Studio UI.
+
+## Root cause of slow indexing (measured on feature/rag)
+
+1. **Embedder reload on every upload.** Ingestion spawns a fresh `spawn` subprocess per job
+   (`core/rag/ingestion.py` `_CTX.Process(target=_subprocess_worker)`), and the embedder is loaded
+   *inside* that subprocess (`get_embedder` -> `from unsloth import FastSentenceTransformer`). A
+   spawned interpreter re-imports unsloth and reloads the model from scratch every time, so the
+   ~12-30s cold load recurs on each upload, not just the first. `lifespan()` never warms it.
+2. **Per-document BM25 rebuild.** On every document, `ingestion.py` reads *all* chunks in the scope
+   (`_all_scope_chunks`) and rebuilds the whole `bm25s` index (`bm25.rebuild_index`). That is
+   O(N^2) tokenization as a knowledge base fills.
+
+## The change (all behind a single `UNSLOTH_RAG_FAST=1` flag; off = byte-identical to PR)
+
+1. **Warm the embedder at startup.** `main.py` `lifespan()` warms `get_embedder()` in a daemon
+   thread (mirrors the existing GGUF precache thread).
+2. **In-process ingestion.** Run the existing `_subprocess_worker` in a thread in the warm main
+   process instead of a fresh subprocess, so `get_embedder` returns the already-loaded singleton.
+   The worker, queue protocol, persistence and SSE progress are otherwise unchanged.
+3. **Incremental SQLite FTS5 BM25.** Reimplement `core/rag/bm25.py` on an FTS5 virtual table in the
+   existing `rag.db`, with an incremental `add_chunks(scope, chunks)`. Ingestion inserts only the
+   new document's chunks (O(N) total) instead of rebuilding the scope. FTS5 `MATCH` returns only
+   matching rows (no zero-score pollution) and adds `porter` stemming. The dense leg already uses
+   sqlite-vec, so it is unchanged; RRF fusion in `retrieval.py` is unchanged.
+
+## Validation
+
+Two isolated Studios (separate `UNSLOTH_STUDIO_HOME` + port), same prebuilt frontend, same corpus,
+same model. Baseline = `UNSLOTH_RAG_FAST` unset; improved = set. Driven through the real UI with
+Playwright (`studio_test_kit`). Local GGUF model `unsloth/Qwen3.5-9B-GGUF` for the chat tool path.
+
+Authoritative corpus (stable URLs): arXiv 1706.03762 (Attention), 1810.04805 (BERT),
+2005.11401 (RAG); RFC 9110. Retrieval scored independently of generation with gold queries.
+
+Metrics: per-document upload->ready latency (cold + warm), Recall@1/3/5 and MRR via `POST
+/api/rag/search`, scaling (per-doc index time vs document count), and end-to-end UI (upload ->
+indexed -> RAG answer with citation). Robustness: delete-then-query, restart persistence, thread
+isolation.
+
+## Results
+
+Pending - populated as runs complete (see commits below).