* tests: read checked-in files as UTF-8 instead of the platform default
Path.read_text() with no encoding uses locale.getpreferredencoding(), which
is UTF-8 on the Linux runners and cp1252 on a stock Windows install. Nine
module-level reads of checked-in source files were relying on that default.
studio/backend/routes/inference.py carries the DeepSeek tool-call token
regexes, so it holds U+FF5C and U+2581. Under cp1252 that read raised
UnicodeDecodeError on byte 0x81 at position 97806, and because the reads run
at import time it took test_cancel_atomicity.py and test_cancel_id_wiring.py
out at collection, not as failures. Green on CI, permanently broken for a
Windows contributor running the suite locally.
Adds a guard: at module scope there is no tmp_path fixture, so a bare
read_text()/write_text()/open() there is always touching a checked-in file.
That makes the rule mechanical enough to enforce with no allowlist, while
staying quiet about temp-dir I/O inside test bodies where the platform
default is harmless.
The repo already spells this correctly in 464 other places; this only stops
the stragglers coming back.
* tests: cover import-time helper reads and keep the guard py3.9-safe
Follows up on the Codex review:
- add `from __future__ import annotations`, since `str | None` in
`_offender` is evaluated at import on Python 3.9 and pyproject declares
requires-python ">=3.9,<3.15".
- widen the guard from module scope to import time. Class bodies and the
bodies of module-level helpers called from an executing statement run
during collection too, so `CODE = _extract_mixed_precision_code()` was
the same hazard as an inline read. `if __name__ == "__main__":` blocks
are skipped: pytest never executes them.
- scan studio/backend/tests/ as well as tests/. Both trees are collected
on Windows by separate CI jobs, and the offender that started this,
test_tool_xml_strip.py reading routes/inference.py, lives there.
Widening it surfaced seven more import-time reads of checked-in sources;
all now name utf-8.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden the import-time encoding guard for PR #7438
Close the detector gaps raised in review, all of which I reproduced against
the actual AST before changing anything.
False negatives (the guard let a real hazard through):
- _is_main_guard ignored the comparison operator, so if __name__ != "__main__"
counted as script-only even though its body runs at import.
- The else arm of a main guard was discarded with the rest of the If node.
- Decorators and argument defaults on a module-level def were skipped with the
body, though both are evaluated when the def executes.
- Path.open() in text mode was invisible; only builtin open() was matched.
- encoding = None and encoding = "locale" both re-select the platform default,
but the keyword merely being present counted as pinned.
False positives (the guard would have blocked a compliant contributor):
- A non-literal mode fell through to the "r" default, so open(p, mode) was
flagged even when mode is "rb", where adding encoding= is a ValueError and
there is no edit that satisfies the rule.
- Same for open(*args) and a **kwargs splat, which hide the mode and can hide
an encoding.
- Lambda bodies and comprehension elements were walked even though neither runs
at definition.
Verified: still reports the same 22 offenders on unpatched main, green on this
branch and on the tree merged with latest main (557 files), and an adversarial
corpus of 33 cases now scores zero false positives and zero false negatives.
Also corrected two docstring claims: neither collecting job runs on Windows,
and the read is governed by locale.getencoding().
* Walk eager comprehensions and treat io.open as the builtin
Two regressions from the previous commit, both reproduced against the AST
before changing anything.
Lumping list, set and dict comprehensions in with generator expressions was
wrong. Only a genexp is lazy; the other three run their element expression,
their filters and their nested iterators immediately, so
CONTENTS = [p.read_text() for p in PATHS] at module scope is an import-time
read the guard was silently missing. Comprehensions are now walked in full and
only the genexp keeps the outermost-iterable-only treatment.
io was also in the not-a-path-opener list, but io.open is the builtin, with the
same mode position and the same platform default. io.open(CHECKED_IN_FILE) is
exactly the hazard this guard exists for, so it is matched now, with binary
modes and a pinned encoding still exempt. tarfile.open and fitz.open stay
exempt since neither has an encoding to name.
Verified: 13 targeted cases covering all five eager comprehension forms and
io.open in text, binary and pinned shapes all classify correctly; still 22
offenders on unpatched main; green on this branch and on the tree merged with
latest main.
* Close three more walker gaps in the import-time guard
All three reproduced against the AST first.
A generator expression handed straight to a call is consumed there, so
DATA = "".join(p.read_text() for p in paths) runs its element at import. Only
an unconsumed genexp bound to a name stays lazy, so the walker now follows the
consumed ones in full and keeps the outermost-iterable-only treatment for the
rest.
if "__main__" == __name__ is an equivalent and accepted spelling of the main
guard, but requiring __name__ on the left meant its body was treated as
import-time code. That is a false positive on a block pytest never runs, so
both operand orders are recognised now.
The helper table was built from module-level defs only, so a def in a class
body invoked while the class is constructed was never followed, contradicting
the walker's stated coverage of class bodies. Helpers are now collected from
the module body and from class bodies at any nesting.
Verified: 15 targeted cases including all three fixes and the earlier ones
still classify correctly; still 22 offenders on unpatched main; green on this
branch and on the tree merged with latest main.
* Handle positional read_text encodings, lazy generators and nested helpers
* Guard reads reached from test bodies, unbound Path calls and __file__ paths
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Follow derived paths, skip lazy generator helpers, cover compressed openers
* Guard the CLI tests, helper parameters and unbound Path arguments
* Discover test roots and follow literal, in-place and tuple-derived paths
* Identify module openers by import, unwrap starred paths, pin subprocess snippets
* Resolve import origins, seed helper locals, follow named generators and parametrize
* Scope imports lexically, list tracked test files, bind unpacked names
* Resolve aliased openers, keyword-only params, destructured targets, next()
* Pin the encoding on subprocess snippets, workflow lint and CLI output for PR #7438
* Harden the CLI encoding guard against detached streams for PR #7438
* Tighten the encoding guard's path and scope analysis for PR #7438
* Resolve path provenance more precisely and keep POSIX stream encodings for PR #7438
* Resolve qualified path classes and scope conditional imports for PR #7438
* Scope CLI stream setup to the entry point and align two encoding pairs for PR #7438
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* studio: fix Backend CI red on main from an ambiguous ordering anchor
test_load_marker_precedes_hub_guard_and_unload fails on main, so every
open PR against the repo inherits the failure.
Root cause. #7239 (a7761e174) reworked the GGUF GPU-pool validation in
_load_model_impl from "if config.is_gguf and effective_gpu_ids is not
None:" to a bare "if config.is_gguf:", placed earlier in the function
than the GGUF load branch. The test anchors on
source.index("if config.is_gguf:"), a first-match search, so it silently
re-anchored onto the GPU-pool statement. #7251 (95f42bcce) then restored
the assertion "= _resolve_inherited_extra_args(" before
"if config.is_gguf:" against a tree where that anchor already pointed at
the wrong statement, and main went red. Checking out 95f42bcce and
running the suite reproduces the same single failure.
The code is correct. _resolve_inherited_extra_args still runs before the
GGUF load branch and before the hub-download guard that consumes
extra_llama_args for require_mmproj, so the guarantee #7251 protects is
intact; only the assertion is wrong.
Fix. Assert that guarantee behaviourally instead of by source offsets.
The new test drives _load_model_impl over a vision GGUF with a stored
--no-mmproj from a previous same-model load and captures the
require_mmproj the hub guard is called with: inherited --no-mmproj gives
False, nothing to inherit gives True, and an explicit request list wins
over the stored one both ways. Moving the resolution call after the
guard makes the inherited case report True and the test fails, so it
detects the reorder the old assertion was meant to catch, without
depending on how many "if config.is_gguf:" statements the endpoint has.
The surviving marker-before-guard-before-unload assertion had the same
ambiguous anchor for its slice start, silently widening the slice past
the GPU-pool block. It now slices from the "if config.is_gguf:" nearest
above the in-flight marker, which pins the load branch.
The structlog test stub gains a get_logger factory so routes/inference.py
is importable when structlog is absent.
34 pass in tests/test_gguf_load_cache_reuse.py (was 32 pass, 1 fail);
350 pass across it plus test_llama_cpp_mmproj_fallback.py and
test_llama_cpp_mtp_detection.py. A full backend run before and after is
identical apart from this test going from fail to pass.
* studio/tests: repair a pre-existing bare structlog stub before importing routes
* studio/tests: tighten the comments on the new load-ordering coverage
* Tighten comments on the load-ordering coverage for PR #7442
_load_model_impl contains more than one `if config.is_gguf:`, so
source.index() returned the earlier one, which belongs to a different check
than the branch the assertion is reasoning about. The inheritance call sits at
line 4543, the earlier branch at 4508 and the branch holding the load marker at
4567, so the comparison read 186995 < 185014 and failed on main.
The branch is now located from the load marker itself, which is the landmark
the rest of the test already relies on, so the assertion compares the
inheritance call against the branch that actually guards it. The slice used by
the following assertions is anchored the same way, which also tightens them:
they previously searched from the earlier branch to end of file.
The invariant is unchanged and still has teeth: moving the inheritance call
after the branch makes the assertion fail.
Co-authored-by: danielhanchen <unslothai@gmail.com>
The gguf order fix that landed on main dropped the only assertion
covering the prerequisite that llama_extra_args inheritance runs before
the GGUF branch: the inherited value (a carried --no-mmproj) shapes the
hub guard's require_mmproj, so a future reorder could reject a load
over an mmproj download the inherited arguments would disable. The
comment also misattributed the inheritance site to
_guard_chat_load_against_training.
The assertion is restored anchored on the call form
"= _resolve_inherited_extra_args(", which pins the endpoint's call site
(the bare name would match the function definition, which always
precedes the endpoint, making the check vacuous), and the comment now
names the real inheritance site. 32 tests pass.
Adds a configurable Hugging Face model download cache location to Unsloth Studio, selectable from Settings, with per-cache download manifests, scoped deletion, and read-only inventory of previously selected caches.
#6414 moved the llama_extra_args inheritance out of the GGUF branch in
_load_model_impl into _guard_chat_load_against_training, which runs before the
branch, so 'if request.llama_extra_args is None' is no longer inside the
gguf_branch slice that test_load_marker_precedes_hub_guard_and_unload checks.
The assertion failed on that now-missing landmark even though the guarantee it
protects (the gguf_load_in_flight marker is entered before the hub-download
guard and the unload) is intact. Drop the relocated landmark from the ordering
so the test matches the current structure.
Co-authored-by: danielhanchen <unslothshared@gmail.com>
