62e9644266* Studio RAG: fix RTL/Indic PDF corruption and dropped DOCX tables The RAG parser prefers pymupdf4llm.to_markdown for PDFs, but that rebuilds text from positioned glyphs and mangles complex-shaping scripts (RTL Arabic/Hebrew come back as shaped Presentation Forms, Indic matras drop to U+FFFD) and can silently drop most of a heavy-RTL page. _pdf now compares the Markdown against PyMuPDF's logical-order get_text() per page and falls back to it when the Markdown looks corrupted (shaped Presentation Forms or U+FFFD above a small floor/ratio) or holds far fewer letters than the raw layer. Latin PDFs are unaffected and keep their Markdown tables/headings. _docx walked document.paragraphs, which excludes table cells, so DOCX tables were dropped entirely. It now walks body content in document order via iter_inner_content, emitting each table row as pipe-joined cells (deduped across merged cells); the preview locator already anchors on pipes. Adds parser tests for the corruption and incompleteness fallbacks and for DOCX table extraction. These mirror the chat document-extractor guard raised in the unslothai/ unsloth#5351 review; the RAG parser is a separate module and needed its own fix. * RAG DOCX: keep empty table cells and collapse in-cell newlines Skipping empty cells shifted later cells left and broke column alignment across rows; a cell with internal paragraphs (newlines) also broke the pipe-joined row. Keep every cell (dropping the row only when all are empty) and normalize each cell with " ".join(split()) so multi-paragraph cells stay on one row. Adds a test for both. * RAG DOCX: dedup merged table cells on the <w:tc> element directly Store the shared <w:tc> lxml element in the seen set instead of its id(); it is hashable and compares by the underlying node, so it dedups spanned/merged cells the same way without relying on id(). Adds a merged-cell test. * RAG DOCX: align merged cells, pad skipped grid columns, flatten nested tables * RAG DOCX: walk cells in document order so nested tables keep in-cell position * RAG DOCX: dedup vertically merged cells so a spanning label is indexed once --------- Co-authored-by: danielhanchen <michaelhan2050@gmail.com> Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| assets | ||
| auth | ||
| core | ||
| hub | ||
| loggers | ||
| models | ||
| plugins | ||
| requirements | ||
| routes | ||
| state | ||
| storage | ||
| tests | ||
| utils | ||
| __init__.py | ||
| _platform_compat.py | ||
| cloudflare_tunnel.py | ||
| colab.py | ||
| main.py | ||
| run.py | ||
| startup_banner.py | ||