fix(tokenizer): check for tokenizer.model after saving it, not before (#7194)

* fix(tokenizer): check for tokenizer.model after saving it, not before

`fix_sentencepiece_tokenizer` creates its temporary directory, then returns
early unless that directory already contains a tokenizer.model:

    if not os.path.exists(temporary_location):
        os.makedirs(temporary_location)          # fresh, empty

    if not os.path.isfile(f"{temporary_location}/tokenizer.model"):
        return new_tokenizer                     # always true

    old_tokenizer.save_pretrained(temporary_location)   # writes that file

The file only appears on the line after the check, so the guard is always
true and the body never runs. Nothing else writes that path either --
`convert_to_fast_tokenizer` saves into a per-name subdirectory, not
`{temporary_location}/tokenizer.model`.

Both call sites are in `get_chat_template` and are commented "Must fix the
sentence piece tokenizer since there's no tokenizer.model file!" -- the
guard defeats the exact intent the caller states. The effect is silent: the
caller still gets a working `new_tokenizer`, but the sentencepiece piece
rename is skipped, so the mapped token (e.g. the eos token remapped to
`<|im_end|>`) is missing from tokenizer.model and GGUF/llama.cpp exports
carry the old piece.

`check_if_sentencepiece_model` in save.py does the same probe in the right
order -- makedirs, save_pretrained, then isfile. Match it.

Tests are added under tests/saving/ next to the existing sentencepiece
coverage, and to the two Bucket-A lists in consolidated-tests-ci.yml, since
Repo tests (CPU) --ignores tests/saving and these need protobuf.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Clear stale tokenizer.model before the sentencepiece guard

The guard now runs after old_tokenizer.save_pretrained, but the default
temporary_location is a fixed reusable directory. A fast-only tokenizer writes
no tokenizer.model, so a stale file from an earlier sentencepiece call could
pass the guard and patch the wrong model (e.g. mixing models in one process,
like a long-running server). Remove any existing tokenizer.model first, and add
a regression test.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Empty the reusable sentencepiece scratch directory each call

The final AutoTokenizer.from_pretrained reloads the whole temporary_location, so
removing only a stale tokenizer.model still let other artifacts from a previous
tokenizer (added_tokens.json, chat template, etc.) leak into the reload when the
default reusable directory is used across models in one process. Recreate the
directory instead, and add a regression test for the leaked-artifact case.

* Clear only top-level scratch files, keep subdirectories

Recreating the whole reusable directory deleted the {name} subtree that
convert_to_fast_tokenizer stores a converted tokenizer's source vocab in, so
old_tokenizer.save_pretrained could not copy tokenizer.model and the guard
returned the tokenizer unpatched for those legacy converted tokenizers. Remove
only stale top-level files (all the final reload reads) and leave subdirectories
intact. Add a regression test for the converted-source subdirectory.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Keep the current tokenizer's own source vocab when clearing

On a repeated get_chat_template(map_eos_token=True) call, the returned tokenizer's
vocab_file points back at the top-level tokenizer.model, and the cleanup deleted
that source before old_tokenizer.save_pretrained could re-emit it, so the guard
returned the tokenizer unpatched. Skip removing the old tokenizer's own source
vocab while still clearing stale files from a different tokenizer, and add a
regression test.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Use a per-call temporary directory for the sentencepiece fix

The scratch directory defaulted to a single shared path, so concurrent or repeated
get_chat_template(map_eos_token=True) calls could delete or overwrite each other's
tokenizer.model between save and reload (tripping the piece assertion or reloading
the wrong model), and stale files from an earlier tokenizer could leak into the
reload. Work in a unique per-call subdirectory instead: this isolates every call
without deleting anything the caller owns, and replaces the earlier per-file cleanup.
Tests updated to read the patched model from the reloaded directory and to cover
isolation and source-vocab preservation.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Pass only the applied token mappings into the sentencepiece fix

get_chat_template mirrors token remaps into tokenizer.model via
fix_sentencepiece_tokenizer, but two caller paths passed a mapping that did not
match what they wrote to the fast tokenizer JSON, so once the sentencepiece patch
runs the model and JSON disagree:
- the mapped-token path skipped entries whose target already existed but still
  passed the full mapping, renaming a piece the JSON never changed;
- the EOS-swap path swapped both tokens in the JSON but passed only one direction,
  leaving two stop_word pieces and no old EOS piece.
Pass the applied mapping (and both swap directions) instead. Add regression tests.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Tighten sentencepiece guard comments

* Add SPDX license identifier to sentencepiece guard test

* Reclaim the per-call sentencepiece scratch directory

The per-call tempfile.mkdtemp fixed the shared-directory race but never cleaned
up, so a long-running process leaked one scratch dir per call. The dir cannot be
deleted eagerly for sentencepiece tokenizers because the returned tokenizer's
vocab_file points into it (a later save_pretrained copies the patched
tokenizer.model from there). Reclaim it correctly instead: remove the dir right
away on the fast-only path (the returned tokenizer never references it), and
attach a weakref.finalize so the sentencepiece dir is removed once its tokenizer
is garbage collected. Add regression tests for both.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Tighten the scratch-dir reclaim comment

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: danielhanchen <unslothshared@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>

.github/workflows/consolidated-tests-ci.yml

@@ -268,6 +268,7 @@ jobs:
             tests/saving/test_save_shell_injection.py \
             tests/saving/test_patch_saving_none_tokenizer.py \
             tests/saving/test_fix_sentencepiece_gguf_robustness.py \
+            tests/saving/test_fix_sentencepiece_tokenizer_guard.py \
             tests/saving/test_compressed_export_schemes.py \
             tests/saving/test_export_api_surface.py \
             tests/saving/test_export_dispatch.py \
@@ -358,6 +359,7 @@ jobs:
             tests/saving/test_save_shell_injection.py \
             tests/saving/test_patch_saving_none_tokenizer.py \
             tests/saving/test_fix_sentencepiece_gguf_robustness.py \
+            tests/saving/test_fix_sentencepiece_tokenizer_guard.py \
             tests/saving/test_compressed_export_schemes.py \
             tests/saving/test_export_api_surface.py \
             tests/saving/test_export_dispatch.py \

tests/saving/test_fix_sentencepiece_tokenizer_guard.py

@@ -0,0 +1,307 @@
+# SPDX-License-Identifier: AGPL-3.0-only
+import gc
+import os
+
+os.environ.setdefault("PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION", "python")
+
+import transformers
+from transformers.utils import sentencepiece_model_pb2
+
+from unsloth.tokenizer_utils import fix_sentencepiece_tokenizer
+
+
+NORMAL, CONTROL = 1, 3
+
+
+def _spm_bytes(pieces):
+    m = sentencepiece_model_pb2.ModelProto()
+    for piece, score, typ in pieces:
+        p = m.pieces.add()
+        p.piece = piece
+        p.score = score
+        p.type = typ
+    return m.SerializeToString()
+
+
+def _read_pieces(path):
+    m = sentencepiece_model_pb2.ModelProto()
+    with open(path, "rb") as f:
+        m.ParseFromString(f.read())
+    return [p.piece for p in m.pieces]
+
+
+class _FakeTokenizer:
+    """Minimal stand-in for a sentencepiece-backed slow tokenizer.
+
+    ``save_pretrained`` writes a tokenizer.model, which is what the real slow
+    tokenizers do and what fix_sentencepiece_tokenizer reads back.
+    """
+
+    def __init__(
+        self,
+        name,
+        spm_bytes = None,
+        vocab = None,
+    ):
+        self.name = name
+        self.eos_token = "</s>"
+        self.pad_token = "<pad>"
+        self._spm_bytes = spm_bytes
+        self._vocab = vocab or {}
+        self.saved_to = []
+
+    def save_pretrained(self, location):
+        self.saved_to.append(location)
+        os.makedirs(location, exist_ok = True)
+        if self._spm_bytes is not None:
+            with open(os.path.join(location, "tokenizer.model"), "wb") as f:
+                f.write(self._spm_bytes)
+
+    def __call__(
+        self,
+        texts,
+        add_special_tokens = False,
+    ):
+        class _Encoded:
+            pass
+
+        encoded = _Encoded()
+        encoded.input_ids = [[self._vocab[text]] for text in texts]
+        return encoded
+
+
+def _tokenizers():
+    pieces = [("<s>", 0.0, CONTROL), ("a", -1.0, NORMAL), ("</s>", 0.0, CONTROL)]
+    old = _FakeTokenizer("old", spm_bytes = _spm_bytes(pieces), vocab = {"</s>": 2})
+    new = _FakeTokenizer("new")
+    return old, new
+
+
+class _ReloadedTokenizer:
+    """Weakref-able stand-in for the tokenizer AutoTokenizer.from_pretrained returns."""
+
+    def __init__(self, location):
+        self.location = location
+
+
+def _stub_auto_tokenizer(monkeypatch):
+    """fix_sentencepiece_tokenizer reloads the patched directory through
+    AutoTokenizer at the end; that needs a full tokenizer on disk, which is
+    out of scope here. Record the reload location and hand back a sentinel.
+    """
+    loaded = []
+
+    class _StubAutoTokenizer:
+        @staticmethod
+        def from_pretrained(location, **kwargs):
+            loaded.append(location)
+            return _ReloadedTokenizer(location)
+
+    monkeypatch.setattr(transformers, "AutoTokenizer", _StubAutoTokenizer)
+    return loaded
+
+
+def test_old_tokenizer_is_saved_so_its_model_can_be_read(tmp_path, monkeypatch):
+    """The guard must not skip the body on a fresh temporary directory.
+
+    fix_sentencepiece_tokenizer creates its scratch directory itself and then
+    checks for a tokenizer.model inside it, but that file only appears once
+    old_tokenizer.save_pretrained() has run.
+    """
+    _stub_auto_tokenizer(monkeypatch)
+    old, new = _tokenizers()
+    location = str(tmp_path / "_unsloth_sentencepiece_temp")
+
+    fix_sentencepiece_tokenizer(old, new, {"</s>": "<|im_end|>"}, temporary_location = location)
+
+    assert old.saved_to, "old tokenizer was never saved: the body did not run"
+
+
+def test_token_mapping_is_applied_to_the_sentencepiece_model(tmp_path, monkeypatch):
+    loaded = _stub_auto_tokenizer(monkeypatch)
+    old, new = _tokenizers()
+    location = str(tmp_path / "_unsloth_sentencepiece_temp")
+
+    # Hold the returned tokenizer so its scratch dir survives until we read it.
+    tok = fix_sentencepiece_tokenizer(old, new, {"</s>": "<|im_end|>"}, temporary_location = location)
+
+    assert "<|im_end|>" in _read_pieces(f"{loaded[-1]}/tokenizer.model")
+    assert tok is not None
+
+
+def test_tokenizer_without_a_sentencepiece_model_is_returned_untouched(tmp_path, monkeypatch):
+    """A fast-only tokenizer writes no tokenizer.model, so the guard still
+    short-circuits and the caller gets new_tokenizer back unchanged. Its scratch
+    dir is unreferenced and reclaimed immediately.
+    """
+    _stub_auto_tokenizer(monkeypatch)
+    old = _FakeTokenizer("old", spm_bytes = None)
+    new = _FakeTokenizer("new")
+    location = str(tmp_path / "_unsloth_sentencepiece_temp")
+
+    result = fix_sentencepiece_tokenizer(
+        old, new, {"</s>": "<|im_end|>"}, temporary_location = location
+    )
+
+    assert result is new
+    assert not any(
+        name.startswith("tokenizer_") for name in os.listdir(location)
+    ), "the fast-only scratch dir was not reclaimed"
+
+
+def test_each_call_uses_a_fresh_isolated_subdirectory(tmp_path, monkeypatch):
+    """Each call must work in its own unique subdirectory, so concurrent or
+    repeated calls never share scratch files, stale artifacts never leak into
+    the reload, and nothing the caller left in the scratch location is deleted.
+    """
+    loaded = _stub_auto_tokenizer(monkeypatch)
+    location = str(tmp_path / "_unsloth_sentencepiece_temp")
+    os.makedirs(location, exist_ok = True)
+
+    # A pre-existing artifact in the shared scratch location.
+    marker = os.path.join(location, "leftover.json")
+    with open(marker, "w") as f:
+        f.write("{}")
+
+    old1, new1 = _tokenizers()
+    old2, new2 = _tokenizers()
+    # Hold both returned tokenizers so their scratch dirs stay alive.
+    tok1 = fix_sentencepiece_tokenizer(
+        old1, new1, {"</s>": "<|im_end|>"}, temporary_location = location
+    )
+    tok2 = fix_sentencepiece_tokenizer(
+        old2, new2, {"</s>": "<|im_end|>"}, temporary_location = location
+    )
+
+    work1, work2 = loaded[0], loaded[1]
+    assert work1 != work2, "two calls reused the same directory"
+    assert os.path.dirname(work1) == location and os.path.dirname(work2) == location
+    assert os.path.isdir(work1) and os.path.isdir(work2)
+    # Nothing the caller left behind is deleted, and it never leaks into a work dir.
+    assert os.path.isfile(marker), "a pre-existing scratch file was deleted"
+    assert not os.path.isfile(os.path.join(work1, "leftover.json"))
+    assert not os.path.isfile(os.path.join(work2, "leftover.json"))
+    assert tok1 is not None and tok2 is not None
+
+
+def test_sentencepiece_scratch_dir_is_reclaimed_once_the_tokenizer_is_gone(tmp_path, monkeypatch):
+    """The scratch dir must live as long as the returned tokenizer (its vocab_file
+    points there), then be reclaimed when the tokenizer is garbage collected.
+    """
+    loaded = _stub_auto_tokenizer(monkeypatch)
+    old, new = _tokenizers()
+    location = str(tmp_path / "_unsloth_sentencepiece_temp")
+
+    tok = fix_sentencepiece_tokenizer(old, new, {"</s>": "<|im_end|>"}, temporary_location = location)
+    work = loaded[-1]
+    assert os.path.isdir(work), "scratch dir vanished while the tokenizer was alive"
+
+    del tok
+    gc.collect()
+    assert not os.path.isdir(work), "scratch dir was not reclaimed after the tokenizer was freed"
+
+
+class _CopyFromSubdirTokenizer:
+    """A slow tokenizer whose sentencepiece source lives elsewhere (like the
+    tokenizers convert_to_fast_tokenizer produces under {location}/{name}).
+    save_pretrained copies that source into the destination, as HF slow
+    tokenizers copy their vocab_file.
+    """
+
+    def __init__(self, source_model_path):
+        self.eos_token = "</s>"
+        self.pad_token = "<pad>"
+        self._source_model_path = source_model_path
+
+    def save_pretrained(self, location):
+        os.makedirs(location, exist_ok = True)
+        if os.path.isfile(self._source_model_path):
+            with open(self._source_model_path, "rb") as src:
+                data = src.read()
+            with open(os.path.join(location, "tokenizer.model"), "wb") as dst:
+                dst.write(data)
+
+    def __call__(
+        self,
+        texts,
+        add_special_tokens = False,
+    ):
+        class _Encoded:
+            pass
+
+        encoded = _Encoded()
+        encoded.input_ids = [[2] for _ in texts]
+        return encoded
+
+
+def test_source_vocab_outside_the_work_directory_is_not_disturbed(tmp_path, monkeypatch):
+    """A tokenizer whose sentencepiece source lives elsewhere (e.g. the subtree
+    convert_to_fast_tokenizer created) is copied into the fresh work directory
+    and patched there; the original source is left untouched.
+    """
+    loaded = _stub_auto_tokenizer(monkeypatch)
+    location = str(tmp_path / "_unsloth_sentencepiece_temp")
+    subdir = os.path.join(location, "some_model")
+    os.makedirs(subdir, exist_ok = True)
+
+    pieces = [("<s>", 0.0, CONTROL), ("a", -1.0, NORMAL), ("</s>", 0.0, CONTROL)]
+    source_model = os.path.join(subdir, "tokenizer.model")
+    with open(source_model, "wb") as f:
+        f.write(_spm_bytes(pieces))
+
+    old = _CopyFromSubdirTokenizer(source_model)
+    new = _FakeTokenizer("new")
+    tok = fix_sentencepiece_tokenizer(old, new, {"</s>": "<|im_end|>"}, temporary_location = location)
+
+    assert _read_pieces(source_model) == [
+        "<s>",
+        "a",
+        "</s>",
+    ], "the original source vocab was modified"
+    assert "<|im_end|>" in _read_pieces(f"{loaded[-1]}/tokenizer.model")
+    assert tok is not None
+
+
+def test_swap_mapping_swaps_both_pieces_without_duplicating(tmp_path, monkeypatch):
+    """When the caller swaps eos and stop_word in the fast JSON it must pass both
+    directions here; a one-way mapping would leave two stop_word pieces and no eos.
+    """
+    loaded = _stub_auto_tokenizer(monkeypatch)
+    location = str(tmp_path / "_unsloth_sentencepiece_temp")
+
+    pieces = [("<s>", 0.0, CONTROL), ("<|im_end|>", -1.0, NORMAL), ("</s>", 0.0, CONTROL)]
+    old = _FakeTokenizer("old", spm_bytes = _spm_bytes(pieces), vocab = {"</s>": 2, "<|im_end|>": 1})
+    new = _FakeTokenizer("new")
+
+    tok = fix_sentencepiece_tokenizer(
+        old, new, {"</s>": "<|im_end|>", "<|im_end|>": "</s>"}, temporary_location = location
+    )
+
+    result = _read_pieces(f"{loaded[-1]}/tokenizer.model")
+    assert result.count("<|im_end|>") == 1 and result.count("</s>") == 1, result
+    assert tok is not None
+
+
+def test_only_applied_mappings_are_patched(tmp_path, monkeypatch):
+    """When the caller skips a mapping whose target already exists, it must not
+    pass that mapping here, or the skipped source token gets renamed anyway and
+    duplicates the existing target in the model.
+    """
+    loaded = _stub_auto_tokenizer(monkeypatch)
+    location = str(tmp_path / "_unsloth_sentencepiece_temp")
+
+    pieces = [
+        ("<s>", 0.0, CONTROL),
+        ("aa", -1.0, NORMAL),
+        ("bb", -1.0, NORMAL),
+        ("X", -1.0, NORMAL),
+    ]
+    old = _FakeTokenizer("old", spm_bytes = _spm_bytes(pieces), vocab = {"aa": 1, "bb": 2})
+    new = _FakeTokenizer("new")
+
+    # Caller skipped aa->X (X already exists) and applied bb->Y, so only bb->Y is passed.
+    tok = fix_sentencepiece_tokenizer(old, new, {"bb": "Y"}, temporary_location = location)
+
+    result = _read_pieces(f"{loaded[-1]}/tokenizer.model")
+    assert result.count("X") == 1 and "Y" in result and "aa" in result, result
+    assert tok is not None

unsloth/chat_templates.py

@@ -1929,6 +1929,9 @@ def get_chat_template(
             string_vocab = tokenizer._tokenizer.to_str()
 
             skipped = 0
+            # Only mirror applied mappings into the spm model; a skipped one would
+            # rename a piece the JSON never changed and desync the two.
+            applied_mapping = {}
             for old_token, new_token in token_mapping.items():
                 old_count = string_vocab.count(f'"{old_token}"')
                 new_count = string_vocab.count(f'"{new_token}"')
@@ -1939,6 +1942,7 @@ def get_chat_template(
                     raise RuntimeError(f"{old_token} was not part of the tokenizer!")
                 else:
                     string_vocab = string_vocab.replace(f'"{old_token}"', f'"{new_token}"')
+                    applied_mapping[old_token] = new_token
                 pass
             pass
 
@@ -1973,7 +1977,7 @@ def get_chat_template(
 
                 # Must fix the sentence piece tokenizer since there's no tokenizer.model file!
                 from .tokenizer_utils import fix_sentencepiece_tokenizer
-                tokenizer = fix_sentencepiece_tokenizer(tokenizer, new_tokenizer, token_mapping,)
+                tokenizer = fix_sentencepiece_tokenizer(tokenizer, new_tokenizer, applied_mapping,)
             else:
                 pass
 
@@ -1997,8 +2001,11 @@ def get_chat_template(
                 string_vocab = string_vocab.replace(old_eos_token, temporary_stop_token)
                 string_vocab = string_vocab.replace(stop_word, old_eos_token)
                 string_vocab = string_vocab.replace(temporary_stop_token, stop_word)
+                # JSON swapped both, so swap both here too; a one-way map leaves two stop_word pieces.
+                sentencepiece_mapping = { old_eos_token : stop_word, stop_word : old_eos_token, }
             else:
                 string_vocab = string_vocab.replace(old_eos_token, stop_word)
+                sentencepiece_mapping = { old_eos_token : stop_word, }
             pass
             new_tokenizer = tokenizer._tokenizer.from_str(string_vocab)
 
@@ -2017,9 +2024,8 @@ def get_chat_template(
             )
 
             # Must fix the sentence piece tokenizer since there's no tokenizer.model file!
-            token_mapping = { old_eos_token : stop_word, }
             from .tokenizer_utils import fix_sentencepiece_tokenizer
-            tokenizer = fix_sentencepiece_tokenizer(tokenizer, new_tokenizer, token_mapping,)
+            tokenizer = fix_sentencepiece_tokenizer(tokenizer, new_tokenizer, sentencepiece_mapping,)
         pass
 
     else:

unsloth/tokenizer_utils.py

@@ -17,6 +17,9 @@ from transformers.convert_slow_tokenizer import convert_slow_tokenizer
 from transformers import PreTrainedTokenizerFast
 import re
 import os
+import shutil
+import tempfile
+import weakref
 from transformers.models.llama.modeling_llama import logger
 from peft import PeftModelForCausalLM
 import torch
@@ -370,13 +373,19 @@ def fix_sentencepiece_tokenizer(
     if not os.path.exists(temporary_location):
         os.makedirs(temporary_location)
 
-    # Check if tokenizer.model exists
-    if not os.path.isfile(f"{temporary_location}/tokenizer.model"):
-        return new_tokenizer
+    # Fresh per-call subdir so concurrent/repeated calls can't clobber each other's
+    # tokenizer.model or leak stale files, without deleting anything the caller owns.
+    temporary_location = tempfile.mkdtemp(prefix = "tokenizer_", dir = temporary_location)
 
     # First save the old tokenizer
     old_tokenizer.save_pretrained(temporary_location)
 
+    # Only sentencepiece tokenizers write tokenizer.model, so check after the save.
+    if not os.path.isfile(f"{temporary_location}/tokenizer.model"):
+        # new_tokenizer was built in memory and never references this dir, so drop it.
+        shutil.rmtree(temporary_location, ignore_errors = True)
+        return new_tokenizer
+
     tokenizer_file = sentencepiece_model_pb2.ModelProto()
     tokenizer_file.ParseFromString(open(f"{temporary_location}/tokenizer.model", "rb").read())
 
@@ -414,6 +423,9 @@ def fix_sentencepiece_tokenizer(
         eos_token = new_tokenizer.eos_token,
         pad_token = new_tokenizer.pad_token,
     )
+    # vocab_file points here, so the dir must outlive the tokenizer (a later
+    # save_pretrained copies the patched tokenizer.model from it); reclaim it on GC.
+    weakref.finalize(tokenizer, shutil.rmtree, temporary_location, ignore_errors = True)
     return tokenizer