Commit graph

2 commits

Author SHA1 Message Date
oobabooga
c4e6dd4f6c
Studio: extract text from PDF web results (#7154) 2026-07-16 19:48:12 -03:00
oobabooga
770f92e250
Studio: reject binary web_search fetches instead of decoding them into replacement chars (#7130)
* Studio: reject binary web_search fetches instead of decoding them into replacement chars

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Match web-fetch MIME subtypes exactly and detect control-char binary

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Sniff binary magic bytes and retry undeclared non-UTF-8 pages as text

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Harden web fetch binary sniffing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Simplify web fetch binary guard

* Sniff unknown MIME types and handle Latin-1

* Sniff ambiguous Office MIME and prefixed magic

* Decode BOM-marked Unicode web content

* Studio: require ASCII evidence for declared Latin-1/cp1252 web fetches

Latin-1 and cp1252 decode every byte to a printable character, so a high-byte
binary body declared as iso-8859-1/windows-1252 decoded cleanly and slipped
past the control-character binary check. Apply the existing ASCII-structure gate
to those declared decodes as well. Scoped to the Latin family so legitimate
non-Latin single-byte pages (Cyrillic, Greek) are not rejected.

* Revert "Studio: require ASCII evidence for declared Latin-1/cp1252 web fetches"

This reverts commit c7fbec216c.

* Studio: tighten web-fetch binary guard comments

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
2026-07-15 10:43:48 -07:00