Studio: move RAG deps to dedicated rag.txt and install in normal path
no-torch-runtime.txt is only consumed in NO_TORCH (Intel Mac GGUF-only) mode, so qdrant-client / bm25s / pymupdf etc. were never installed in the normal install path. Split them into rag.txt and add a step to install_python_stack.py that installs it after studio deps, skipped only when NO_TORCH is set.
This commit is contained in:
parent
30856de739
commit
6c694d2ef8
3 changed files with 40 additions and 21 deletions
|
|
@ -76,24 +76,3 @@ trl>=0.18.2,!=0.19.0,<=0.24.0
|
|||
sentence-transformers
|
||||
cut_cross_entropy
|
||||
pillow
|
||||
|
||||
# RAG: vector store, lexical index, document parsers.
|
||||
# qdrant-client supports a pure-Python local mode (QdrantClient(path=...))
|
||||
# that we use to keep the studio install self-contained — no separate
|
||||
# server. bm25s persists per-scope indices to disk.
|
||||
qdrant-client>=1.12
|
||||
bm25s>=0.2
|
||||
# RAG parsers (Phase 3A): layout-aware Markdown extraction so the chunker
|
||||
# can split on real headings instead of running paragraphs together.
|
||||
# pymupdf4llm preserves headings + pipe-tables; mammoth handles DOCX
|
||||
# Heading styles; markdownify converts HTML <h*>/<table>/<ul> faithfully.
|
||||
pymupdf>=1.24
|
||||
pymupdf4llm>=0.0.17
|
||||
mammoth>=1.7
|
||||
markdownify>=0.13
|
||||
# pypdf is kept as a fallback for malformed PDFs that defeat pymupdf.
|
||||
pypdf>=4.0
|
||||
python-docx>=1.1
|
||||
beautifulsoup4>=4.12
|
||||
lxml>=5.0
|
||||
chardet>=5.2
|
||||
|
|
|
|||
27
studio/backend/requirements/rag.txt
Normal file
27
studio/backend/requirements/rag.txt
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
# Studio RAG dependencies.
|
||||
# Installed by studio/install_python_stack.py in the normal (with-torch)
|
||||
# path. Skipped in NO_TORCH (Intel Mac GGUF-only) mode because RAG
|
||||
# embedding relies on sentence-transformers, which requires torch.
|
||||
|
||||
# Vector store + lexical index. qdrant-client supports a pure-Python
|
||||
# local mode (QdrantClient(path=...)) that we use to keep the studio
|
||||
# install self-contained — no separate server. bm25s persists per-scope
|
||||
# indices to disk.
|
||||
qdrant-client>=1.12
|
||||
bm25s>=0.2
|
||||
|
||||
# Layout-aware Markdown extraction (Phase 3A) so the chunker can split
|
||||
# on real headings instead of running paragraphs together. pymupdf4llm
|
||||
# preserves headings + pipe-tables; mammoth handles DOCX Heading styles;
|
||||
# markdownify converts HTML <h*>/<table>/<ul> faithfully.
|
||||
pymupdf>=1.24
|
||||
pymupdf4llm>=0.0.17
|
||||
mammoth>=1.7
|
||||
markdownify>=0.13
|
||||
|
||||
# pypdf is kept as a fallback for malformed PDFs that defeat pymupdf.
|
||||
pypdf>=4.0
|
||||
python-docx>=1.1
|
||||
beautifulsoup4>=4.12
|
||||
lxml>=5.0
|
||||
chardet>=5.2
|
||||
|
|
@ -915,6 +915,8 @@ def install_python_stack() -> int:
|
|||
base_total -= 1 # triton step is skipped on macOS
|
||||
if not IS_WINDOWS and not IS_MACOS and not NO_TORCH:
|
||||
base_total += 3
|
||||
if not NO_TORCH:
|
||||
base_total += 1 # studio RAG deps (rag.txt)
|
||||
_TOTAL = (base_total - 1) if skip_base else base_total
|
||||
|
||||
# 1. Try to use uv for faster installs (must happen before pip upgrade
|
||||
|
|
@ -1203,6 +1205,17 @@ def install_python_stack() -> int:
|
|||
req = REQ_ROOT / "studio.txt",
|
||||
)
|
||||
|
||||
# 8b. RAG dependencies (vector store + lexical index + layout-aware
|
||||
# parsers). Skipped in no-torch mode because RAG embeddings go
|
||||
# through sentence-transformers, which requires torch.
|
||||
if not NO_TORCH:
|
||||
_progress("rag deps")
|
||||
pip_install(
|
||||
"Installing RAG dependencies",
|
||||
"--no-cache-dir",
|
||||
req = REQ_ROOT / "rag.txt",
|
||||
)
|
||||
|
||||
# 9. Data-designer dependencies
|
||||
_progress("data designer deps")
|
||||
pip_install(
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue