ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Sync FORCE_FLOAT32 fallback with unsloth-zoo (gemma4, glm4_moe, qwen3_moe) (#6865)

Browse source 
* Add gemma4, glm4_moe and qwen3_moe to the FORCE_FLOAT32 fallback list

Keeps the fallback list (used only if the unsloth_zoo import fails) in sync with
unsloth_zoo/model_lists.py, which now force-float32s these MoE archs so a float16
request loads bf16 and trains finite instead of NaNing the grad_norm.

* Union FORCE_FLOAT32 fallback so new archs force float32 with older unsloth_zoo
This commit is contained in:
Daniel Han 2026-07-06 05:48:10 -07:00 • committed by GitHub
commit efcaffb17b
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 20 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

34
unsloth/models/loader.py
View file

@ -113,22 +113,28 @@ from ._utils import (
maybe_prefetch_hf_snapshot,
)
# Single source of truth is unsloth_zoo.model_lists. Re-exported so callers
# doing `from unsloth.models.loader import FORCE_FLOAT32` keep working.
# Fallback list mirrors zoo for users who upgrade unsloth without upgrading
# unsloth_zoo (so this module never fails at import).
# Source of truth is unsloth_zoo.model_lists. Re-exported so callers doing
# `from unsloth.models.loader import FORCE_FLOAT32` keep working. The fallback
# list is also unioned in so a newer unsloth still forces float32 for these
# archs when paired with an older unsloth_zoo that predates them (upgrade skew).
_FORCE_FLOAT32_FALLBACK = [
"gemma3,", # Add comma bc gemma3 will match gemma3n
"gemma3text", # Gemma3TextModel (EmbeddingGemma, standalone text-only Gemma3)
"gemma3n",
"gemma4", # Gemma4 (gemma4 / gemma4_text): float16 NaNs grad norms in the backward
"glm4_moe", # GLM-4.x MoE (glm4_moe / glm4_moe_lite): float16 NaNs grad norms
"gpt_oss",
"qwen3_5", # Qwen3.5 GDN layers produce NaN grad norms in float16 training
"qwen3_moe", # Qwen3-MoE (Qwen3-30B-A3B): float16 NaNs grad norms in the backward
]
try:
from unsloth_zoo import FORCE_FLOAT32 # noqa: F401
from unsloth_zoo import FORCE_FLOAT32 as _ZOO_FORCE_FLOAT32
FORCE_FLOAT32 = list(_ZOO_FORCE_FLOAT32)
except ImportError:
global FORCE_FLOAT32
# Forces float32 precision since float16 goes to infinity
FORCE_FLOAT32 = [
"gemma3,", # Add comma bc gemma3 will match gemma3n
"gemma3text", # Gemma3TextModel (EmbeddingGemma, standalone text-only Gemma3)
"gemma3n",
"gpt_oss",
"qwen3_5", # Qwen3.5 GDN layers produce NaN grad norms in float16 training
]
FORCE_FLOAT32 = []
for _mt in _FORCE_FLOAT32_FALLBACK:
if not any(_mt in _entry for _entry in FORCE_FLOAT32):
FORCE_FLOAT32.append(_mt)
global DISABLE_COMPILE_MODEL_NAMES
# Must be alphabetically sorted for each entry