798e73e770
Qwen-family models (Qwen3.5, etc.) use a tiktoken-based tokenizer. When Studio routes these models through the transformers 5.x overlay (.venv_t5), the tokenizer instantiation fails with: ValueError: Couldn't instantiate the backend tokenizer... You need to have sentencepiece or tiktoken installed This happens because .venv_t5 was set up with only transformers and huggingface_hub (both --no-deps). The main venv usually has tiktoken, but on Windows it sometimes does not. Fix: install tiktoken into .venv_t5 in both setup.sh and the runtime fallback (_ensure_venv_t5_exists). Uses --no-deps in the runtime path since regex/requests are always present in the main venv. |
||
|---|---|---|
| .. | ||
| datasets | ||
| hardware | ||
| inference | ||
| models | ||
| paths | ||
| .gitkeep | ||
| __init__.py | ||
| cache_cleanup.py | ||
| transformers_version.py | ||
| utils.py | ||