unsloth/studio/backend/core
Saicharan Ramineni 7ecbf5a770
Use UTF-8 for Python code-execution subprocess I/O (#6489 class) (#6548)
* Use UTF-8 for Python code-execution subprocess I/O

Studio's code-execution tool already tells the child to emit UTF-8
(PYTHONIOENCODING=utf-8 in _build_safe_env), but _python_exec writes the
temp script and decodes the subprocess pipe with the OS default codec.
On Windows (cp1252), non-ASCII in model-written code or its output --
arrows, CJK, emoji -- raises UnicodeEncodeError / UnicodeDecodeError and
breaks execution.

Complete the UTF-8 wiring in core/inference/tools.py:
- write the temp script with encoding="utf-8"
- decode _python_exec stdout as utf-8, errors="replace"
- set PYTHONIOENCODING=utf-8 in _build_bypass_env too (matches
  _build_safe_env, so the bypass path's child also emits utf-8)

The child is python with PYTHONIOENCODING=utf-8, so it emits UTF-8
regardless of the console code page and the decode is always correct.
Shell execution via cmd.exe has a separate console-code-page story and
is left to a follow-up.

Refs unslothai/unsloth#6489

* Scope Python exec UTF-8 env to Python tool

* Make bash bypass test robust to a host-set PYTHONIOENCODING for PR #6548

Bypass mode preserves benign host env vars, so a host-set PYTHONIOENCODING was
inherited into the bash bypass env and tripped the new assertion even though
_bash_exec never adds it. Clear it in the test so the assertion checks _bash_exec,
not the runner environment.

---------

Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
2026-06-22 09:06:03 -07:00
..
data_recipe Studio: use an isolated Node.js for the frontend build instead of replacing the system Node/npm (#6533) 2026-06-21 21:17:29 -07:00
export Generalize transformers tier selection by probing AutoConfig (#6550) 2026-06-22 08:20:06 -07:00
inference Use UTF-8 for Python code-execution subprocess I/O (#6489 class) (#6548) 2026-06-22 09:06:03 -07:00
rag Reap Studio child processes when the parent dies abnormally (#6425) 2026-06-18 05:51:22 -07:00
training Generalize transformers tier selection by probing AutoConfig (#6550) 2026-06-22 08:20:06 -07:00
__init__.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
_torchao_stub.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
import_guards.py Studio: self-heal unsloth namespace shadows; clearer failed-load messages (#6532) 2026-06-21 22:43:31 -07:00
tool_healing.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00