ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Keep pad-named pad_tokens (e.g. <|vision_pad|>); fix Qwen3-Base load crash (#6652)

Browse source 
* Keep pad-named pad_tokens; defer pad repair to shared unsloth_zoo.pad_token

A pad-named token (e.g. <|vision_pad|>) is a valid pad. The narrow fallback that
stripped vision pad tokens on text-only models is now a no-op; the active path
delegates to the shared fix_pad_token in unsloth_zoo, which keeps pad-named tokens
and only heals missing / eos-collision / out-of-range pads.

This fixes the Qwen3-4B-Base load crash (its config ships pad_token=<|vision_pad|>):
the old swap could not find a safe text pad (eos is <|endoftext|>, no unk_token) and
left the tokenizer broken. Removes the unused _VISION_PAD_TOKENS / _SAFE_TEXT_PAD_TOKENS
sets. Tests updated.

Pairs with unslothai/unsloth-zoo#831.

* Remove _fix_vision_pad_token; inline the no-op fallback

A pad-named token (e.g. <|vision_pad|>) is a valid pad, so the old vision-pad swap
helper has no purpose. _fix_pad_token now returns the tokenizer unchanged when the
shared unsloth_zoo.pad_token module is unavailable, instead of routing through a
no-op helper. Test WANTED set updated.
This commit is contained in:
Daniel Han 2026-06-25 04:41:09 -07:00 • committed by GitHub
commit 4929c5f769
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 13 additions and 66 deletions
Download patch file Download diff file Expand all files Collapse all files

15
tests/python/test_pad_token_fix.py
View file

@ -1,9 +1,9 @@
"""_fix_pad_token dispatch in unsloth/tokenizer_utils.py.
It must delegate to unsloth_zoo's shared fix_pad_token when present (single
source of truth), and fall back to the narrow vision-token swap against an
older unsloth_zoo. Static + CPU-only: the two helpers are exec'd in isolation
so the test never imports torch / transformers / unsloth.
source of truth), and fall back to a no-op against an older unsloth_zoo. Static
+ CPU-only: _fix_pad_token is exec'd in isolation so the test never imports
torch / transformers / unsloth.
"""
import ast
@ -15,9 +15,6 @@ REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
TOK_PATH = os.path.join(REPO_ROOT, "unsloth", "tokenizer_utils.py")
WANTED = {
"_VISION_PAD_TOKENS",
"_SAFE_TEXT_PAD_TOKENS",
"_fix_vision_pad_token",
"_fix_pad_token",
}
@ -69,17 +66,17 @@ def test_fix_pad_token_none_is_noop():
assert ns["_fix_pad_token"](None) is None
def test_fix_pad_token_falls_back_without_shared_module(monkeypatch):
def test_fallback_keeps_pad_named_token(monkeypatch):
ns = _load_pad_helpers()
_block_shared_module(monkeypatch)
# Qwen3 text tokenizer shipping a vision pad_token -> narrow swap heals it.
# A pad-named token (e.g. <|vision_pad|>) is a valid pad -> fallback keeps it.
tok = FakeTok(
{"<|endoftext|>": 1, "<|im_end|>": 2, "<|vision_pad|>": 3},
pad = "<|vision_pad|>",
eos = "<|im_end|>",
)
ns["_fix_pad_token"](tok)
assert tok.pad_token == "<|endoftext|>"
assert tok.pad_token == "<|vision_pad|>"
assert tok.pad_token != tok.eos_token