Route Gemma-4 float16 through UNSLOTH_FORCE_FLOAT32
Add "gemma4" to FORCE_FLOAT32 so FastVisionModel.from_pretrained with dtype=torch.float16 flips UNSLOTH_FORCE_FLOAT32=1 and swaps the load dtype to bf16, matching how Gemma-3 and Gemma-3n are handled. Without this entry, fp16 Gemma-4 GRPO crashes at step 2 with a CUDA device-side assert because the text-decoder MLP saturates in fp16 and produces non-finite logits during generation. The actual stability work lives in the paired unsloth-zoo patches on Gemma4RMSNorm, Gemma4TextMLP, Gemma4TextAttention, and Gemma4TextScaledWordEmbedding, all of which gate on UNSLOTH_FORCE_FLOAT32 and stay no-ops for bf16 runs.
This commit is contained in:
parent
7d0d2f256c
commit
027b7f8551
1 changed files with 1 additions and 0 deletions
|
|
@ -105,6 +105,7 @@ FORCE_FLOAT32 = [
|
|||
"gemma3,", # Add comma bc gemma3 will match gemma3n
|
||||
"gemma3text", # Gemma3TextModel (EmbeddingGemma, standalone text-only Gemma3)
|
||||
"gemma3n",
|
||||
"gemma4", # Gemma-4 E2B/E4B GRPO NaNs under float16 (MLP gate*up overflow)
|
||||
"gpt_oss",
|
||||
"qwen3_5", # Qwen3.5 GDN layers produce NaN grad norms in float16 training
|
||||
]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue