56e9046b2fIn full FT, AdamW weight decay shrinks the parameter directly so the implicit prior is W -> 0. In LoRA the trained parameters are A and B while the effective weight is W = W_init + (alpha/r) * B @ A; decaying A and B separately drives BA -> 0, hence W -> W_init rather than 0. The previous default of 0.01 inherited from full-FT recipes adds a measurable pull on the merged adapter back toward the base model over a few thousand steps. 0.001 keeps a small Frobenius-norm prior on ||A||^2 + ||B||^2 for numerical stability without meaningfully biasing the merged weight toward init, and aligns with the value used across the unsloth notebook templates. |
||
|---|---|---|
| .. | ||
| dataprep | ||
| kernels | ||
| models | ||
| optimizers | ||
| registry | ||
| utils | ||
| __init__.py | ||
| _auto_install.py | ||
| _gpu_init.py | ||
| chat_templates.py | ||
| device_type.py | ||
| import_fixes.py | ||
| ollama_template_mappers.py | ||
| save.py | ||
| tokenizer_utils.py | ||
| trainer.py | ||