unsloth/studio/backend/core/rag
Daniel Han 34d24bcf97 Studio RAG trim: remove late-chunking strategy
Late chunking (the Jina single-pass technique) is a non-default embedding
path (chunking_strategy defaults to 'standard'). It adds a second full-document
forward pass with windowed token pooling for a marginal long-document gain that
the standard per-chunk encoder already covers at R@5 = 1.0 on the gold set.

- remove late_chunk_encode / _windowed_late_chunk_encode / _pool_spans /
  _encode_tokens from embeddings.py (205 lines)
- remove the chunking_strategy == 'late' branch and _run_late_chunking from
  ingestion.py (75 lines); 'late' now degrades to standard chunking

42 RAG tests pass.
2026-05-31 14:18:03 +00:00
..
parsers Studio: tighten RAG code comments 2026-05-31 08:31:08 +00:00
__init__.py Studio: add RAG with hybrid search, reranker, chat integration 2026-05-23 18:46:15 +04:00
authorization.py Studio: fix RAG correctness bugs 2026-05-31 09:56:23 +00:00
bm25.py Studio: fast RAG path (warm embedder + in-process ingest + FTS5 BM25) behind UNSLOTH_RAG_FAST 2026-05-31 13:31:01 +00:00
bm25_fts.py Studio: fast RAG path (warm embedder + in-process ingest + FTS5 BM25) behind UNSLOTH_RAG_FAST 2026-05-31 13:31:01 +00:00
captioner.py Studio: tighten RAG code comments 2026-05-31 08:31:08 +00:00
chunking.py Studio: tighten RAG code comments 2026-05-31 08:31:08 +00:00
db.py Studio: trim verbose comments/docstrings across RAG code 2026-05-26 13:51:11 +04:00
embeddings.py Studio RAG trim: remove late-chunking strategy 2026-05-31 14:18:03 +00:00
ingestion.py Studio RAG trim: remove late-chunking strategy 2026-05-31 14:18:03 +00:00
locators.py Studio: fix RAG correctness bugs 2026-05-31 09:56:23 +00:00
retrieval.py Studio: fix RAG correctness bugs 2026-05-31 09:56:23 +00:00
scope.py Studio: fix RAG correctness bugs 2026-05-31 09:56:23 +00:00
tool.py Studio RAG trim: remove cross-encoder reranker 2026-05-31 14:14:52 +00:00
vector_store.py Studio: WIP — RAG preview UI, locator/auth refactor, tests, fixtures (pre-merge snapshot) 2026-05-28 00:13:43 +03:00