ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

[fix] qwen3-guard tokenizer (#3959)

Browse source 
* fix for qwen3-guard tokenizer

* Better qwen3guard check

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This commit is contained in:
Datta Nimmaturi 2026-02-02 11:39:15 +05:30 • committed by GitHub
commit 5d95a23273
1 changed files with 4 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

5
unsloth/tokenizer_utils.py
View file

@ -602,7 +602,10 @@ def load_correct_tokenizer(
old_chat_template = getattr(tokenizer, "chat_template", None)
# Ignore mistral type models since they don't have an add_generation_prompt
if "mistral" in str(getattr(tokenizer, "name_or_path", "")).lower():
if any(
s in str(getattr(tokenizer, "name_or_path", "")).lower()
for s in ["mistral", "qwen3guard"]
):
chat_template = old_chat_template
# Also check Llama-2 old style models