51f1c8732d* Fix UnicodeDecodeError on Windows reading subprocess output in save path On Windows the default text encoding is the locale code page (cp1252), not UTF-8. The text-mode subprocess calls in save.py (text=True / universal_newlines=True) set no explicit encoding, so they decode llama.cpp / Ollama output with cp1252. When a child process emits a byte undefined in cp1252 -- e.g. 0x9d, which appears inside the UTF-8 encoding of common punctuation / box-drawing glyphs and in non-ASCII file paths -- the read raises UnicodeDecodeError and aborts GGUF export. Add encoding="utf-8", errors="replace" to all 8 text-mode subprocess calls. errors="replace" also avoids silent mojibake for inputs whose bytes happen to be valid-but-wrong in cp1252. Add tests/saving/test_save_subprocess_utf8_encoding.py: - an AST drift detector asserting every text-mode subprocess call in save.py pins encoding="utf-8" (runs without importing torch/unsloth_zoo) - a behavioural test reproducing the cp1252 failure and the utf-8 fix Relates-to: #2660 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| gpt-oss-merge | ||
| language_models | ||
| non_peft | ||
| text_to_speech_models | ||
| vision_models | ||
| test_fix_sentencepiece_gguf_robustness.py | ||
| test_patch_saving_none_tokenizer.py | ||
| test_preserve_tokenizer_eos_token.py | ||
| test_qwen3_5_vlm_full_finetune_key_remap.py | ||
| test_save_shell_injection.py | ||
| test_save_subprocess_utf8_encoding.py | ||
| test_unsloth_save.py | ||