unsloth/studio/backend/core/rag
Roland Tannous ba0fd85e8b Studio: break RAG chunks at figure/table caption boundaries
Dense embedders mean-pool over a whole chunk, so a 'Figure 1:' caption
buried at the end of a 500-token body chunk gets washed out by the
surrounding theory text and never surfaces for queries about that
figure. Pre-split each page's markdown at the start of every
Figure/Table caption line so the caption anchors its own chunk, which
gives both BM25 and the dense vector a focused, figure-dominated
target. Handles numbered, decimal, and appendix-style labels
(Figure 1, Figure 1.2, Figure B.1, Table 4, Fig./Tab. abbreviations).
2026-05-27 16:09:23 +04:00
..
parsers Studio: text-mode default with VLM-captioned figure splicing; helper VLM fallback 2026-05-27 11:16:56 +04:00
__init__.py Studio: add RAG with hybrid search, reranker, chat integration 2026-05-23 18:46:15 +04:00
bm25.py Studio: trim verbose comments/docstrings across RAG code 2026-05-26 13:51:11 +04:00
captioner.py Studio: disable thinking for RAG captioner requests 2026-05-27 15:33:48 +04:00
chunking.py Studio: break RAG chunks at figure/table caption boundaries 2026-05-27 16:09:23 +04:00
db.py Studio: trim verbose comments/docstrings across RAG code 2026-05-26 13:51:11 +04:00
embeddings.py Studio: pass BytesIO (not PIL Image) to BGE-VL encode so model.data_process can re-open 2026-05-27 05:44:40 +04:00
ingestion.py Studio: route RAG ingestion + captioner loggers through structlog 2026-05-27 15:20:22 +04:00
reranker.py Studio: trim verbose comments/docstrings across RAG code 2026-05-26 13:51:11 +04:00
retrieval.py Studio: trim verbose comments/docstrings across RAG code 2026-05-26 13:51:11 +04:00
scope.py Studio: text-mode default with VLM-captioned figure splicing; helper VLM fallback 2026-05-27 11:16:56 +04:00
tool.py Studio: VLM-caption figures at ingest + pass image hits to LLM + render in card 2026-05-26 20:17:52 +04:00
vector_store.py Studio: trim verbose comments/docstrings across RAG code 2026-05-26 13:51:11 +04:00