ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

tests/studio: clarify why MLX smoke pins max_grad_value=0

Browse source 
Refresh the rationale comment to reflect the new default landing in
unslothai/unsloth-zoo#652 (max_grad_value=1.0, not 5.0). The smoke
still needs the explicit pin because neither default value reliably
converges in 7 steps at seed=3407:

  max_grad_value=5.0 -- diverges after step 4 (loss 7.3 -> 8.4)
  max_grad_value=1.0 -- stalls (loss ~3.2 plateau across seeds)
  max_grad_value=0.5/0.25/0.1 -- noisier still
  max_grad_norm=1.0  -- cleanly drops loss to <0.01, emits "Unsloth!"

Mention both the historical 5.0 default and the new 1.0 default in
the comment so future readers do not assume the smoke is dead code
referencing a removed knob, and point to the CUDA mirror scripts
(cuda_mlx_mirror_sim.py + cuda_mlx_clip1_vs_norm1.py) for the
empirical evidence.

No behaviour change; comment-only refresh.
This commit is contained in:
Daniel Han 2026-05-15 10:45:18 +00:00 • committed by danielhanchen
commit e05d6c7df6
1 changed files with 4 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

13
tests/studio/run_real_mlx_smoke.py
View file

@ -278,15 +278,10 @@ def cmd_train(args) -> int:
optim = "adamw",
weight_decay = 0.0,
max_grad_norm = 1.0,
# Explicitly disable the new per-element clip introduced in
# #5340 (default max_grad_value=5.0). When both are set the
# MLX trainer silently drops max_grad_norm in favour of the
# per-element clip, but +-5.0 is far too loose for this
# 270M LoRA setup -- losses diverge after step 4 and the
# model never memorises "Unsloth!" (verified via the CUDA
# mirror at scripts/cuda_mlx_mirror_sim.py). Pinning
# max_grad_value=0 makes the smoke depend on the same
# max_grad_norm=1.0 the test was originally written for.
# Disable per-element clip so the trainer uses max_grad_norm.
# No value converges in 7 steps at seed=3407 (5.0 diverges,
# 1.0 stalls ~3.2); only norm clip drops loss <0.01 and
# emits "Unsloth!". See scripts/cuda_mlx_*.
max_grad_value = 0.0,
logging_steps = 1,
max_seq_length = 64,