* Studio RAG: fix RTL/Indic PDF corruption and dropped DOCX tables
The RAG parser prefers pymupdf4llm.to_markdown for PDFs, but that rebuilds text from
positioned glyphs and mangles complex-shaping scripts (RTL Arabic/Hebrew come back as
shaped Presentation Forms, Indic matras drop to U+FFFD) and can silently drop most of a
heavy-RTL page. _pdf now compares the Markdown against PyMuPDF's logical-order
get_text() per page and falls back to it when the Markdown looks corrupted (shaped
Presentation Forms or U+FFFD above a small floor/ratio) or holds far fewer letters than
the raw layer. Latin PDFs are unaffected and keep their Markdown tables/headings.
_docx walked document.paragraphs, which excludes table cells, so DOCX tables were
dropped entirely. It now walks body content in document order via iter_inner_content,
emitting each table row as pipe-joined cells (deduped across merged cells); the preview
locator already anchors on pipes.
Adds parser tests for the corruption and incompleteness fallbacks and for DOCX table
extraction. These mirror the chat document-extractor guard raised in the unslothai/
unsloth#5351 review; the RAG parser is a separate module and needed its own fix.
* RAG DOCX: keep empty table cells and collapse in-cell newlines
Skipping empty cells shifted later cells left and broke column alignment across rows;
a cell with internal paragraphs (newlines) also broke the pipe-joined row. Keep every
cell (dropping the row only when all are empty) and normalize each cell with
" ".join(split()) so multi-paragraph cells stay on one row. Adds a test for both.
* RAG DOCX: dedup merged table cells on the <w:tc> element directly
Store the shared <w:tc> lxml element in the seen set instead of its id(); it is
hashable and compares by the underlying node, so it dedups spanned/merged cells the
same way without relying on id(). Adds a merged-cell test.
* RAG DOCX: align merged cells, pad skipped grid columns, flatten nested tables
* RAG DOCX: walk cells in document order so nested tables keep in-cell position
* RAG DOCX: dedup vertically merged cells so a spanning label is indexed once
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
62e9644266