* fix(tests/sh): accept pinned tokenizers line after #5359#5359 pinned the tokenizers line in
studio/backend/requirements/no-torch-runtime.txt from bare
`tokenizers` to `tokenizers<=0.23.0` to stop pip from resolving to
0.23.1+ (which transformers rejects at import time). The shell test
in tests/sh/test_torch_constraint.sh was still asserting the literal
`^tokenizers$` regex, which fails on the pinned form. Surfaced as
a hard fail on PR #5312's Backend CI Repo tests (CPU) step:
=== Structural: tokenizers in no-torch-runtime.txt ===
FAIL: tokenizers present as standalone line (expected '1', got '0')
FAIL: tokenizers before transformers (expected 'yes', got 'no')
Relax the regex to `^tokenizers([<>=!,~ ]|$)` so it matches both bare
and version-constrained forms, which preserves the original intent
(verify tokenizers is present in the file, before transformers).
Verified locally: 24 PASS, 0 FAIL.
* fixup(tests/sh): tighten tokenizers check to guard the safe bound
Address bot review feedback on #5361:
* Codex (P2): the previous relaxed regex `^tokenizers([<>=!,~ ]|$)`
accepted any version operator, so `tokenizers>=0.23.1` would still
pass the test even though that line would re-introduce the import
failure #5359 fixed.
* Gemini (medium): the boundary char class did not cover all PEP 508
separators (`[`, `;`, `@`).
Replace the single check with two:
1. Loose: `^tokenizers([^a-zA-Z0-9._-]|$)` confirms the package is
listed (covers extras, env markers, URLs, bare line).
2. Tight regression guard: pipe those lines through a second grep
that requires `<=0.23.0` or the functionally equivalent
`<0.23.1`. Rejects bare `tokenizers`, `>=0.22.0` (no upper
bound), `>=0.23.1`, `!=0.23.0`, `<=0.24.0`, etc.
Verified locally:
- Current main (tokenizers<=0.23.0): 25 PASS, 0 FAIL.
- Spot-check with the bug reverted (bare `tokenizers`): the new
"tokenizers pinned with upper bound excluding 0.23.1+" check
FAILS as intended; the original "listed" check still passes.
2fd21a737d