56e9046b2fIn full FT, AdamW weight decay shrinks the parameter directly so the implicit prior is W -> 0. In LoRA the trained parameters are A and B while the effective weight is W = W_init + (alpha/r) * B @ A; decaying A and B separately drives BA -> 0, hence W -> W_init rather than 0. The previous default of 0.01 inherited from full-FT recipes adds a measurable pull on the merged adapter back toward the base model over a few thousand steps. 0.001 keeps a small Frobenius-norm prior on ||A||^2 + ||B||^2 for numerical stability without meaningfully biasing the merged weight toward init, and aligns with the value used across the unsloth notebook templates. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| _utils.py | ||
| cohere.py | ||
| dpo.py | ||
| falcon_h1.py | ||
| gemma.py | ||
| gemma2.py | ||
| glm4_moe.py | ||
| granite.py | ||
| llama.py | ||
| llama4.py | ||
| loader.py | ||
| loader_utils.py | ||
| mapper.py | ||
| mistral.py | ||
| qwen2.py | ||
| qwen3.py | ||
| qwen3_moe.py | ||
| rl.py | ||
| rl_replacements.py | ||
| sentence_transformer.py | ||
| vision.py | ||