34d24bcf97
Late chunking (the Jina single-pass technique) is a non-default embedding path (chunking_strategy defaults to 'standard'). It adds a second full-document forward pass with windowed token pooling for a marginal long-document gain that the standard per-chunk encoder already covers at R@5 = 1.0 on the gold set. - remove late_chunk_encode / _windowed_late_chunk_encode / _pool_spans / _encode_tokens from embeddings.py (205 lines) - remove the chunking_strategy == 'late' branch and _run_late_chunking from ingestion.py (75 lines); 'late' now degrades to standard chunking 42 RAG tests pass. |
||
|---|---|---|
| .. | ||
| data_recipe | ||
| export | ||
| inference | ||
| rag | ||
| training | ||
| __init__.py | ||
| _torchao_stub.py | ||
| tool_healing.py | ||