ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Pin utf-8 on shipping-code text I/O instead of the operator locale (#7486)

Browse source 
* Pin utf-8 on shipping-code text I/O instead of the operator locale

113 read_text/write_text/open call sites across unsloth, studio and
unsloth_cli let locale.getencoding() decide the encoding. That is utf-8 on
the Linux and macOS runners and cp1252 on a stock Windows install, so the
same file decodes differently for a Windows user and silently produces
mojibake or raises UnicodeDecodeError.

Adds tests/test_runtime_text_encoding.py to keep it that way. It resolves
openers through each file's own imports rather than a fixed list of module
names, so an aliased tarfile.open or a local from PIL.Image import open is
not asked for an encoding it does not take.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Scan tracked files only and resolve the unbound Path calling forms

* Honour PEP 263 when scanning sources and migrate a legacy JSONL before appending

* Scope guard imports lexically and only migrate a legacy file when it round-trips

* Leave a legacy JSONL untouched and resolve path aliases in the foreign-opener check

* Tighten comments

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Daniel Han 2026-07-27 02:14:20 -07:00 • committed by GitHub
commit 3fd948eb95
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
39 changed files with 563 additions and 114 deletions
Download patch file Download diff file Expand all files Collapse all files

10
studio/install_python_stack.py
View file

@ -503,7 +503,7 @@ def _detect_rocm_version() -> tuple[int, int] | None:
os.path.join(rocm_root, "lib", "rocm_version"),
):
try:
with open(path) as fh:
with open(path, encoding = "utf-8") as fh:
parts = fh.read().strip().split("-")[0].split(".")
# Explicit length guard: don't rely on the broad except below to
# swallow IndexError on a single-component version (e.g. "6\n").
@ -852,9 +852,9 @@ def _linux_amd_display_device_present() -> bool:
try:
for dev in Path("/sys/bus/pci/devices").iterdir():
try:
if (dev / "vendor").read_text().strip() != "0x1002":
if (dev / "vendor").read_text(encoding = "utf-8").strip() != "0x1002":
continue
if (dev / "class").read_text().strip().startswith("0x03"):
if (dev / "class").read_text(encoding = "utf-8").strip().startswith("0x03"):
return True
except OSError:
continue
@ -1067,7 +1067,7 @@ def _has_rocm_gpu() -> bool:
for entry in os.listdir(kfd_nodes):
gpu_id_path = os.path.join(kfd_nodes, entry, "gpu_id")
try:
with open(gpu_id_path) as fh:
with open(gpu_id_path, encoding = "utf-8") as fh:
gpu_id = fh.read().strip()
except OSError:
continue
@ -1079,7 +1079,7 @@ def _has_rocm_gpu() -> bool:
# false positive (e.g. NVIDIA open-driver KFD nodes lacking it).
props_path = os.path.join(kfd_nodes, entry, "properties")
try:
with open(props_path) as fh:
with open(props_path, encoding = "utf-8") as fh:
props = fh.read()
except OSError:
continue # can't confirm vendor -- skip