798e73e770
Qwen-family models (Qwen3.5, etc.) use a tiktoken-based tokenizer. When Studio routes these models through the transformers 5.x overlay (.venv_t5), the tokenizer instantiation fails with: ValueError: Couldn't instantiate the backend tokenizer... You need to have sentencepiece or tiktoken installed This happens because .venv_t5 was set up with only transformers and huggingface_hub (both --no-deps). The main venv usually has tiktoken, but on Windows it sometimes does not. Fix: install tiktoken into .venv_t5 in both setup.sh and the runtime fallback (_ensure_venv_t5_exists). Uses --no-deps in the runtime path since regex/requests are always present in the main venv. |
||
|---|---|---|
| .. | ||
| backend | ||
| frontend | ||
| __init__.py | ||
| install_python_stack.py | ||
| LICENSE.AGPL-3.0 | ||
| setup.bat | ||
| setup.ps1 | ||
| setup.sh | ||
| Unsloth_Studio_Colab.ipynb | ||