2093fb1608
asg017/sqlite-vec is Apache-2.0 and OSI-approved. Replaces qdrant-client (~30 MB) with a small SQLite extension loaded into a dedicated rag.db file. Single file holds RAG vectors; bm25s indexes and chat-side studio.db are unaffected. - New core/rag/db.py owns the rag.db connection and sqlite-vec load. Extension load runs once at first open. Process-wide singleton protected by a lock; check_same_thread=False + WAL handles the FastAPI thread pool. - core/rag/vector_store.py keeps the same public API (ensure_collection / upsert_chunks / search / collection_exists / delete_scope / delete_document) so callers in routes/rag.py, core/rag/ingestion.py, core/rag/tool.py, and core/rag/retrieval.py don't change. ensure_collection is now a no-op; collection_exists returns True iff the scope has at least one indexed vector. - search uses sqlite-vec's vec_distance_cosine and converts distance to similarity in [0, 1] so the per-scope min_score threshold semantics stay identical. - Mixed-dim scopes coexist behind WHERE scope = ? — the per-scope embedder resolver guarantees one embedder per scope. - requirements/rag.txt swaps qdrant-client for sqlite-vec. - utils/paths/storage_roots.py drops rag_vectordb_root() (the old qdrant directory); rag.db lives directly under rag_root(). - Rewritten tests/python/test_rag_vector_store.py for the new semantics (collection_exists tracks populated scopes; new tests for filtered search and upsert conflict resolution). Python build requirement: connection.enable_load_extension(True) must be available. install.sh creates the venv via uv-managed python-build-standalone, which is compiled with --enable-loadable-sqlite-extensions, so this works on standard installs. core/rag/db.py raises an actionable error on the rare custom-interpreter case.
35 lines
1.3 KiB
Text
35 lines
1.3 KiB
Text
# Studio RAG dependencies.
|
|
# Installed by studio/install_python_stack.py in the normal (with-torch)
|
|
# path. Skipped in NO_TORCH (Intel Mac GGUF-only) mode because RAG
|
|
# embedding relies on sentence-transformers, which requires torch.
|
|
|
|
# Vector store + lexical index.
|
|
#
|
|
# sqlite-vec is an Apache-2.0 SQLite extension (asg017/sqlite-vec) that
|
|
# adds vector functions (vec_distance_cosine, serialize_float32, vec0
|
|
# virtual tables). The studio loads it into a dedicated rag.db file and
|
|
# stores vectors as BLOB columns alongside the RAG metadata — no separate
|
|
# vector server, no second client library. bm25s persists per-scope
|
|
# lexical indexes to disk.
|
|
sqlite-vec>=0.1.5
|
|
bm25s>=0.2
|
|
|
|
# Image preprocessing helpers required by Qwen3-VL-Embedding-2B (the
|
|
# multimodal embedder). Not used in text-only mode.
|
|
qwen-vl-utils>=0.0.14
|
|
|
|
# Layout-aware Markdown extraction (Phase 3A) so the chunker can split
|
|
# on real headings instead of running paragraphs together. pymupdf4llm
|
|
# preserves headings + pipe-tables; mammoth handles DOCX Heading styles;
|
|
# markdownify converts HTML <h*>/<table>/<ul> faithfully.
|
|
pymupdf>=1.24
|
|
pymupdf4llm>=0.0.17
|
|
mammoth>=1.7
|
|
markdownify>=0.13
|
|
|
|
# pypdf is kept as a fallback for malformed PDFs that defeat pymupdf.
|
|
pypdf>=4.0
|
|
python-docx>=1.1
|
|
beautifulsoup4>=4.12
|
|
lxml>=5.0
|
|
chardet>=5.2
|