diff --git a/Home.md b/Home.md index 3b73883..dac899c 100644 --- a/Home.md +++ b/Home.md @@ -18,7 +18,8 @@ trainer = Trainer(...) trainer.train() ``` -### Finetuning the `lm_head` and `embed_tokens` matrices: +### Continued Pretraining & Finetuning the `lm_head` and `embed_tokens` matrices +Add `lm_head` and `embed_tokens`. For Colab, sometimes you will go out of memory for Llama-3 8b. If so, just add `lm_head`. ```python model = FastLanguageModel.get_peft_model( model, @@ -29,6 +30,20 @@ model = FastLanguageModel.get_peft_model( lora_alpha = 16, ) ``` +Then use 2 different learning rates - a 2-10x smaller one for the `lm_head` or `embed_tokens` like so: + +```python +from unsloth import UnslothTrainer, UnslothTrainingArguments + +trainer = UnslothTrainer( + .... + args = UnslothTrainingArguments( + .... + learning_rate = 5e-5, + embedding_learning_rate = 5e-6, # 2-10x smaller than learning_rate + ), +) +``` ### Finetuning from your last checkpoint You must edit the `Trainer` first to add `save_strategy` and `save_steps`. Below saves a checkpoint every 50 steps to the folder `outputs`.