* fix(tests/sh): accept pinned tokenizers line after #5359#5359 pinned the tokenizers line in
studio/backend/requirements/no-torch-runtime.txt from bare
`tokenizers` to `tokenizers<=0.23.0` to stop pip from resolving to
0.23.1+ (which transformers rejects at import time). The shell test
in tests/sh/test_torch_constraint.sh was still asserting the literal
`^tokenizers$` regex, which fails on the pinned form. Surfaced as
a hard fail on PR #5312's Backend CI Repo tests (CPU) step:
=== Structural: tokenizers in no-torch-runtime.txt ===
FAIL: tokenizers present as standalone line (expected '1', got '0')
FAIL: tokenizers before transformers (expected 'yes', got 'no')
Relax the regex to `^tokenizers([<>=!,~ ]|$)` so it matches both bare
and version-constrained forms, which preserves the original intent
(verify tokenizers is present in the file, before transformers).
Verified locally: 24 PASS, 0 FAIL.
* fixup(tests/sh): tighten tokenizers check to guard the safe bound
Address bot review feedback on #5361:
* Codex (P2): the previous relaxed regex `^tokenizers([<>=!,~ ]|$)`
accepted any version operator, so `tokenizers>=0.23.1` would still
pass the test even though that line would re-introduce the import
failure #5359 fixed.
* Gemini (medium): the boundary char class did not cover all PEP 508
separators (`[`, `;`, `@`).
Replace the single check with two:
1. Loose: `^tokenizers([^a-zA-Z0-9._-]|$)` confirms the package is
listed (covers extras, env markers, URLs, bare line).
2. Tight regression guard: pipe those lines through a second grep
that requires `<=0.23.0` or the functionally equivalent
`<0.23.1`. Rejects bare `tokenizers`, `>=0.22.0` (no upper
bound), `>=0.23.1`, `!=0.23.0`, `<=0.24.0`, etc.
Verified locally:
- Current main (tokenizers<=0.23.0): 25 PASS, 0 FAIL.
- Spot-check with the bug reverted (bare `tokenizers`): the new
"tokenizers pinned with upper bound excluding 0.23.1+" check
FAILS as intended; the original "listed" check still passes.
* fix: add tokenizers to no-torch runtime deps and add TORCH_CONSTRAINT for arm64 macOS py313+
Two installer fixes:
1. Add `tokenizers` to `no-torch-runtime.txt` before `transformers`.
Without it, `from transformers import AutoConfig` crashes on startup
because `--no-deps` skips transitive dependencies.
2. Add `TORCH_CONSTRAINT` variable to `install.sh`. On arm64 macOS with
Python 3.13+, tighten the torch requirement to `>=2.6` since torch
<2.6 has no cp313 arm64 wheels. The variable replaces the previously
hard-coded constraint in the uv pip install line.
Includes 66 tests (42 pytest + 24 bash) covering:
- Structural checks on install.sh, install.ps1, no-torch-runtime.txt
- Shell snippet tests with mocked python for 13 platform/version combos
- Mock uv integration verifying correct constraint string
- E2E venv tests on Python 3.12 and 3.13 confirming AutoConfig works
- Negative control proving AutoConfig fails without tokenizers
- Full no-torch sandbox regression guards (safetensors, huggingface_hub)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix incomplete no-torch manifest and align E2E tests with real --no-deps path
- Add missing transitive deps to no-torch-runtime.txt that are required
under --no-deps: regex, typing_extensions, filelock, httpx, httpcore,
certifi, idna, anyio, sniffio, h11. Without these, `from transformers
import AutoConfig` still fails after install.sh --no-torch.
- Change all E2E tests to use --no-deps (matching what install.sh does)
instead of normal dep resolution. Previous tests passed even with an
incomplete manifest because uv backfilled transitive deps.
- Rewrite negative control to derive from the real no-torch-runtime.txt
with tokenizers stripped, proving the specific fix matters.
- Replace GNU-only sed -i with heredoc in shell test for macOS compat.
- Remove unused os/sys imports from Python test file.
- Quote SKIP_TORCH and mock uv paths in bash -c strings.
* Assert install succeeds before checking import results in E2E tests
Address review feedback: test_torch_not_importable and
test_tokenizers_directly_importable in Group 3 now assert that
uv pip install returns 0 before checking import behavior. This
prevents false positives when the install itself fails silently.
* Assert install succeeds in negative control and tighten error check
- Add missing install-success assertion in test_negative_control_no_tokenizers
to prevent false positives from network/install failures.
- Tighten error message check to look for "tokenizers" in stderr or
ModuleNotFoundError, rather than the generic "No module" substring
which could match unrelated import failures.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Daniel Han