810e3a80be
BGE-VL inherits CLIP's 77-token text positional embedding table — longer chunks crash inside the text model with a shape mismatch. Pre-tokenize with truncation=True, max_length=77 and call get_text_features directly so the high-level encode() (which does not truncate) is bypassed. Log when truncation happens — text chunks beyond the cap are silently cut, so multimodal mode is lossy on the text channel. Image channel is unaffected. |
||
|---|---|---|
| .. | ||
| parsers | ||
| __init__.py | ||
| bm25.py | ||
| chunking.py | ||
| embeddings.py | ||
| ingestion.py | ||
| reranker.py | ||
| retrieval.py | ||
| tool.py | ||
| vector_store.py | ||