ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Fix tokenizer reload from LoRA path on transformers 5.x

Browse source 
Transformers 5.x no longer saves special_tokens_map.json when calling
tokenizer.save_pretrained(). The three-file check in loader.py
(tokenizer_config.json + tokenizer.json + special_tokens_map.json)
fails, causing Unsloth to fall back to the base model's tokenizer
which may not have a chat_template set.

Accept chat_template.jinja as an alternative signal that the LoRA path
contains valid tokenizer data. This file is always saved by Unsloth's
save_pretrained and contains the chat template.
This commit is contained in:
Daniel Han 2026-02-24 08:08:03 +00:00
commit 0028e2f6d7
1 changed files with 8 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

10
unsloth/models/loader.py
View file

@ -568,7 +568,10 @@ class FastLanguageModel(FastLlamaModel):
if (
os.path.exists(os.path.join(old_model_name, "tokenizer_config.json"))
and os.path.exists(os.path.join(old_model_name, "tokenizer.json"))
and os.path.exists(os.path.join(old_model_name, "special_tokens_map.json"))
and (
os.path.exists(os.path.join(old_model_name, "special_tokens_map.json"))
or os.path.exists(os.path.join(old_model_name, "chat_template.jinja"))
)
):
tokenizer_name = old_model_name
else:
@ -1237,7 +1240,10 @@ class FastModel(FastBaseModel):
if (
os.path.exists(os.path.join(old_model_name, "tokenizer_config.json"))
and os.path.exists(os.path.join(old_model_name, "tokenizer.json"))
and os.path.exists(os.path.join(old_model_name, "special_tokens_map.json"))
and (
os.path.exists(os.path.join(old_model_name, "special_tokens_map.json"))
or os.path.exists(os.path.join(old_model_name, "chat_template.jinja"))
)
):
tokenizer_name = old_model_name
else: