ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update tokenizer_utils.py

Browse source
This commit is contained in:
Daniel Han-Chen 2024-03-24 23:15:33 +11:00
commit 4e43875670
1 changed files with 1 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
unsloth/tokenizer_utils.py
View file

@ -275,6 +275,6 @@ def load_correct_tokenizer(
if assert_same_tokenization(slow_tokenizer, fast_tokenizer):
return fast_tokenizer
else:
return slow_tokenizer
return convert_to_fast_tokenizer(slow_tokenizer)
pass
pass