d22b2a18f9* fix: add tokenizers to no-torch runtime deps and add TORCH_CONSTRAINT for arm64 macOS py313+ Two installer fixes: 1. Add `tokenizers` to `no-torch-runtime.txt` before `transformers`. Without it, `from transformers import AutoConfig` crashes on startup because `--no-deps` skips transitive dependencies. 2. Add `TORCH_CONSTRAINT` variable to `install.sh`. On arm64 macOS with Python 3.13+, tighten the torch requirement to `>=2.6` since torch <2.6 has no cp313 arm64 wheels. The variable replaces the previously hard-coded constraint in the uv pip install line. Includes 66 tests (42 pytest + 24 bash) covering: - Structural checks on install.sh, install.ps1, no-torch-runtime.txt - Shell snippet tests with mocked python for 13 platform/version combos - Mock uv integration verifying correct constraint string - E2E venv tests on Python 3.12 and 3.13 confirming AutoConfig works - Negative control proving AutoConfig fails without tokenizers - Full no-torch sandbox regression guards (safetensors, huggingface_hub) * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix incomplete no-torch manifest and align E2E tests with real --no-deps path - Add missing transitive deps to no-torch-runtime.txt that are required under --no-deps: regex, typing_extensions, filelock, httpx, httpcore, certifi, idna, anyio, sniffio, h11. Without these, `from transformers import AutoConfig` still fails after install.sh --no-torch. - Change all E2E tests to use --no-deps (matching what install.sh does) instead of normal dep resolution. Previous tests passed even with an incomplete manifest because uv backfilled transitive deps. - Rewrite negative control to derive from the real no-torch-runtime.txt with tokenizers stripped, proving the specific fix matters. - Replace GNU-only sed -i with heredoc in shell test for macOS compat. - Remove unused os/sys imports from Python test file. - Quote SKIP_TORCH and mock uv paths in bash -c strings. * Assert install succeeds before checking import results in E2E tests Address review feedback: test_torch_not_importable and test_tokenizers_directly_importable in Group 3 now assert that uv pip install returns 0 before checking import behavior. This prevents false positives when the install itself fails silently. * Assert install succeeds in negative control and tighten error check - Add missing install-success assertion in test_negative_control_no_tokenizers to prevent false positives from network/install failures. - Tighten error message check to look for "tokenizers" in stderr or ModuleNotFoundError, rather than the generic "No module" substring which could match unrelated import failures. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
50 lines
1.2 KiB
Text
50 lines
1.2 KiB
Text
# Runtime dependencies for no-torch (GGUF-only) mode.
|
|
# Installed with --no-deps to prevent transitive torch resolution
|
|
# from packages like accelerate, peft, trl, sentence-transformers.
|
|
#
|
|
# Includes unsloth's own direct deps (typer, pydantic, pyyaml,
|
|
# nest-asyncio) since unsloth is also installed with --no-deps
|
|
# (current PyPI metadata still declares torch as a hard dep).
|
|
|
|
# unsloth direct deps (from pyproject.toml [project].dependencies)
|
|
typer
|
|
pydantic
|
|
pyyaml
|
|
nest-asyncio
|
|
|
|
# HF ecosystem (from [huggingfacenotorch] extras in pyproject.toml)
|
|
wheel>=0.42.0
|
|
packaging
|
|
numpy
|
|
tqdm
|
|
psutil
|
|
tyro
|
|
protobuf
|
|
sentencepiece>=0.2.0
|
|
safetensors>=0.4.3
|
|
datasets>=3.4.1,!=4.0.*,!=4.1.0,<4.4.0
|
|
accelerate>=0.34.1
|
|
peft>=0.18.0,!=0.11.0
|
|
huggingface_hub>=0.34.0
|
|
hf_transfer
|
|
diffusers
|
|
|
|
# Transitive deps required because this file is installed with --no-deps.
|
|
# Without these, `from transformers import AutoConfig` fails at import time.
|
|
regex
|
|
typing_extensions
|
|
filelock
|
|
httpx
|
|
httpcore
|
|
certifi
|
|
idna
|
|
anyio
|
|
sniffio
|
|
h11
|
|
|
|
tokenizers
|
|
transformers>=4.51.3,!=4.52.0,!=4.52.1,!=4.52.2,!=4.52.3,!=4.53.0,!=4.54.0,!=4.55.0,!=4.55.1,!=4.57.0,!=4.57.4,!=4.57.5,!=5.0.0,!=5.1.0,<=5.3.0
|
|
trl>=0.18.2,!=0.19.0,<=0.24.0
|
|
sentence-transformers
|
|
cut_cross_entropy
|
|
pillow
|