Commit graph

1 commit

Author SHA1 Message Date
dylanschroers
51f1c8732d
fix: decode subprocess output as UTF-8 in save.py on Windows (#6218)
* Fix UnicodeDecodeError on Windows reading subprocess output in save path

On Windows the default text encoding is the locale code page (cp1252), not
UTF-8. The text-mode subprocess calls in save.py (text=True /
universal_newlines=True) set no explicit encoding, so they decode
llama.cpp / Ollama output with cp1252. When a child process emits a byte
undefined in cp1252 -- e.g. 0x9d, which appears inside the UTF-8 encoding
of common punctuation / box-drawing glyphs and in non-ASCII file paths --
the read raises UnicodeDecodeError and aborts GGUF export.

Add encoding="utf-8", errors="replace" to all 8 text-mode subprocess calls.
errors="replace" also avoids silent mojibake for inputs whose bytes happen
to be valid-but-wrong in cp1252.

Add tests/saving/test_save_subprocess_utf8_encoding.py:
- an AST drift detector asserting every text-mode subprocess call in
  save.py pins encoding="utf-8" (runs without importing torch/unsloth_zoo)
- a behavioural test reproducing the cp1252 failure and the utf-8 fix

Relates-to: #2660

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-06-12 01:31:31 -07:00