diff --git a/README.md b/README.md index 200ef3ff3b..f3e2a19891 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ -### Finetune Mistral, Llama 2-5x faster with 70% less memory! +### Finetune Mistral, Gemma, Llama 2-5x faster with 70% less memory! ![](https://i.ibb.co/sJ7RhGG/image-41.png) @@ -22,28 +22,30 @@ All notebooks are **beginner friendly**! Add your dataset, click "Run All", and | Unsloth supports | Free Notebooks | Performance | Memory use | |-----------------|--------------------------------------------------------------------------------------------------------------------------|-------------|----------| +| **Gemma 7b** | [▶️ Start on Colab](https://colab.research.google.com/drive/1lBzz5KeZJKXjvivbYvmGarix9Ao6Wxe5?usp=sharing) | 2.5x faster | 24% less | | **Mistral 7b** | [▶️ Start on Colab](https://colab.research.google.com/drive/1Dyauq4kTZoLewQ1cApceUQVNcnnNTzg_?usp=sharing) | 2.2x faster | 62% less | | **Llama-2 7b** | [▶️ Start on Colab](https://colab.research.google.com/drive/1lBzz5KeZJKXjvivbYvmGarix9Ao6Wxe5?usp=sharing) | 2.2x faster | 43% less | -| **DPO - Zephyr** | [▶️ Start on Colab](https://colab.research.google.com/drive/15vttTpzzVXv_tJwEk-hIcQ0S9FcEWvwP?usp=sharing) | 1.9x faster | 19% less | | **TinyLlama** | [▶️ Start on Colab](https://colab.research.google.com/drive/1AZghoNBQaMDgWJpi4RbffGM1h6raLUj9?usp=sharing) | 3.9x faster | 74% less | | **CodeLlama 34b** A100 | [▶️ Start on Colab](https://colab.research.google.com/drive/1y7A0AxE3y8gdj4AVkl2aZX47Xu3P1wJT?usp=sharing) | 1.9x faster | 27% less | | **Mistral 7b** 1xT4 | [▶️ Start on Kaggle](https://www.kaggle.com/code/danielhanchen/kaggle-mistral-7b-unsloth-notebook) | 5x faster\* | 62% less | +| **DPO - Zephyr** | [▶️ Start on Colab](https://colab.research.google.com/drive/15vttTpzzVXv_tJwEk-hIcQ0S9FcEWvwP?usp=sharing) | 1.9x faster | 19% less | - This [conversational notebook](https://colab.research.google.com/drive/1Aau3lgPzeZKQ-98h69CCu1UJcvIBLmy2?usp=sharing) is useful for ShareGPT ChatML / Vicuna templates. - This [text completion notebook](https://colab.research.google.com/drive/1ef-tab5bhkvWmBOObepl1WgJvfvSzn5Q?usp=sharing) is for raw text. This [DPO notebook](https://colab.research.google.com/drive/15vttTpzzVXv_tJwEk-hIcQ0S9FcEWvwP?usp=sharing) replicates Zephyr. -- Colab provides a free GPU sometimes. Kaggle has 30 hrs free per week on a 12 hr running cap. -- \* Kaggle has 2x T4s, but we use 1. Due to overhead, 1x T4 is 5x faster. Use Colab as Kaggle takes 10 mins to install. +- \* Kaggle has 2x T4s, but we use 1. Due to overhead, 1x T4 is 5x faster. ## 🦥 Unsloth.ai News -- 📣 [DPO support](https://colab.research.google.com/drive/15vttTpzzVXv_tJwEk-hIcQ0S9FcEWvwP?usp=sharing) is now included. [More info](#DPO) on DPO. -- 📣 [TinyLlama 1.1b](https://colab.research.google.com/drive/1AZghoNBQaMDgWJpi4RbffGM1h6raLUj9?usp=sharing) on 3T tokens now works. -- 📣 We did a [blog](https://huggingface.co/blog/unsloth-trl) with 🤗Hugging Face, and we're in their official docs! Check out the [SFT docs](https://huggingface.co/docs/trl/main/en/sft_trainer#accelerate-fine-tuning-2x-using-unsloth) and [DPO docs](https://huggingface.co/docs/trl/main/en/dpo_trainer#accelerate-dpo-fine-tuning-using-unsloth). -- 📣 Now supports **Llama, Yi, Mistral, CodeLlama, Qwen (llamafied), Deepseek** and their derived models (**Open Hermes** etc). Llama 7, 13, 70b; CodeLlama 7, 13, 34, 70b; Yi 6, 34b are all supported! -- 📣 **Download models 4x faster** from 🤗Hugging Face! Eg: `unsloth/mistral-7b-bnb-4bit` See our [HF collection](https://huggingface.co/collections/unsloth/load-4bit-models-4x-faster-659042e3a41c3cbad582e734) for more! +- 📣 [Gemma 7b](https://colab.research.google.com/drive/1AZghoNBQaMDgWJpi4RbffGM1h6raLUj9?usp=sharing) on 6T tokens now works. And [Gemma 2b notebook](https://colab.research.google.com/drive/1AZghoNBQaMDgWJpi4RbffGM1h6raLUj9?usp=sharing) +- 📣 Added [conversational notebooks](https://colab.research.google.com/drive/1ef-tab5bhkvWmBOObepl1WgJvfvSzn5Q?usp=sharing) and [raw text notebooks](https://colab.research.google.com/drive/1bMOKOBzxQWUIGZBs_B0zm8pimuEnZdfM?usp=sharing) +- 📣 [2x faster inference](https://colab.research.google.com/drive/15vttTpzzVXv_tJwEk-hIcQ0S9FcEWvwP?usp=sharing) added for all our models +- 📣 [DPO support](https://colab.research.google.com/drive/15vttTpzzVXv_tJwEk-hIcQ0S9FcEWvwP?usp=sharing) is now included. [More info](#DPO) on DPO +- 📣 We did a [blog](https://huggingface.co/blog/unsloth-trl) with 🤗Hugging Face and are in their official docs! Check out the [SFT docs](https://huggingface.co/docs/trl/main/en/sft_trainer#accelerate-fine-tuning-2x-using-unsloth) and [DPO docs](https://huggingface.co/docs/trl/main/en/dpo_trainer#accelerate-dpo-fine-tuning-using-unsloth) +- 📣 [Download models 4x faster](https://huggingface.co/collections/unsloth/) from 🤗Hugging Face. Eg: `unsloth/mistral-7b-bnb-4bit` ## 🔗 Links and Resources | Type | Links | | ------------------------------- | --------------------------------------- | +| 📚 **Wiki & FAQ** | [Read Our Wiki](https://github.com/unslothai/unsloth/wiki) | | 📜 **Documentation** | [Read The Doc](https://github.com/unslothai/unsloth/tree/main#-documentation) | | 💾 **Installation** | [unsloth/README.md](https://github.com/unslothai/unsloth/tree/main#installation-instructions)| |   **Twitter (aka X)** | [Follow us on X](https://twitter.com/unslothai)| diff --git a/unsloth/chat_templates.py b/unsloth/chat_templates.py index eb61056320..6f0237c27b 100644 --- a/unsloth/chat_templates.py +++ b/unsloth/chat_templates.py @@ -217,6 +217,35 @@ alpaca_eos_token = "eos_token" CHAT_TEMPLATES["alpaca"] = (alpaca_template, alpaca_eos_token,) +# https://huggingface.co/google/gemma-7b-it +# Notice we must use |trim for lstrip and rstrip. maps to 106. +# maps to 107. user and model are normal 1 word tokens. +gemma_template = \ + "{% for message in messages %}"\ + "{% if message['role'] == 'user' %}"\ + "{{'user\n' + message['content'] | trim + '\n'}}"\ + "{% elif message['role'] == 'assistant' %}"\ + "{{'model\n' + message['content'] | trim + '\n' }}"\ + "{% else %}"\ + "{{ 'system\n' + message['content'] | trim + '\n' }}"\ + "{% endif %}"\ + "{% endfor %}"\ + "{% if add_generation_prompt %}"\ + "{{ 'model\n' }}"\ + "{% endif %}" +gemma_eos_token = "" +CHAT_TEMPLATES["gemma"] = (gemma_template, gemma_eos_token,) + + +# Gemma with ChatML instead +gemma_chatml_template = chatml_template +gemma_chatml_eos_token = ( + {"" : "<|im_start|>", "" : "<|im_end|>"}, + "<|im_end|>", +) +CHAT_TEMPLATES["gemma_chatml"] = (gemma_chatml_template, gemma_chatml_eos_token,) + + def get_chat_template( tokenizer, chat_template = "chatml", @@ -229,7 +258,7 @@ def get_chat_template( old_padding_side = tokenizer.padding_side - if type(chat_template) in (list, tuple): + if type(chat_template) in (list, tuple,): chat_template, stop_word = chat_template assert(type(chat_template) is str) assert(type(stop_word) is str) @@ -238,7 +267,38 @@ def get_chat_template( chat_template, stop_word = CHAT_TEMPLATES[chat_template] - if stop_word != "eos_token": + if type(stop_word) in (list, tuple,): + token_mapping, stop_word = stop_word + assert(type(token_mapping) is dict) + else: + token_mapping = None + + assert(type(stop_word) is str) + + # token_mapping = {"" : "<|im_start|>", "" : "<|im_end|>"} + # For Gemma :) + if token_mapping is not None: + + string_vocab = tokenizer._tokenizer.to_str() + + for old_token, new_token in token_mapping.items(): + old_count = string_vocab.count(f'"{old_token}"') + new_count = string_vocab.count(f'"{new_token}"') + if new_count != 0: + print(f"{new_token} is already a token. Skipping.") + elif old_count == 0: + raise RuntimeError(f"{old_token} was not part of the tokenizer!") + else: + string_vocab = string_vocab.replace(f'"{old_token}"', f'"{new_token}"') + pass + pass + + logger.warning_once(f"Unsloth: Will map {stop_word} to EOS = {tokenizer.eos_token}.") + string_vocab = string_vocab.replace(tokenizer.eos_token, stop_word) + new_tokenizer = tokenizer._tokenizer.from_str(string_vocab) + tokenizer = tokenizer.__class__(tokenizer_object = new_tokenizer, eos_token = stop_word) + + elif stop_word != "eos_token": logger.warning_once(f"Unsloth: Will map {stop_word} to EOS = {tokenizer.eos_token}.") # Replaces the old EOS token with a new one. @@ -252,6 +312,7 @@ def get_chat_template( new_tokenizer = tokenizer._tokenizer.from_str(string_vocab) tokenizer = tokenizer.__class__(tokenizer_object = new_tokenizer, eos_token = stop_word) pass + else: raise TypeError( f"Unsloth: `chat_template` must be a tuple of (your_template, eos_token,) or one of\n"\ @@ -318,6 +379,7 @@ def test_chat_templates(): {"role": "user", "content": " No it's 100% 5! "}, ] + # Zephyr from transformers import AutoTokenizer template = zephyr_template correct_tokenizer = AutoTokenizer.from_pretrained("HuggingFaceH4/zephyr-7b-beta") @@ -326,6 +388,7 @@ def test_chat_templates(): our_prompt = correct_tokenizer.apply_chat_template(messages, tokenize = False, add_generation_prompt = True) assert(correct_prompt == our_prompt) + # Chatml template = chatml_template correct_tokenizer = AutoTokenizer.from_pretrained("teknium/OpenHermes-2.5-Mistral-7B") correct_prompt = correct_tokenizer.apply_chat_template(messages, tokenize = False, add_generation_prompt = True) @@ -333,6 +396,7 @@ def test_chat_templates(): our_prompt = correct_tokenizer.apply_chat_template(messages, tokenize = False, add_generation_prompt = True) assert(correct_prompt == our_prompt) + # Mistral template = mistral_template correct_tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2") correct_prompt = correct_tokenizer.apply_chat_template(messages[1:], tokenize = False, add_generation_prompt = True) @@ -340,6 +404,7 @@ def test_chat_templates(): our_prompt = correct_tokenizer.apply_chat_template(messages[1:], tokenize = False, add_generation_prompt = True) assert(correct_prompt == our_prompt) + # Llama template = llama_template correct_tokenizer = AutoTokenizer.from_pretrained("unsloth/llama-2-7b-chat") correct_prompt = correct_tokenizer.apply_chat_template(messages, tokenize = False, add_generation_prompt = True) @@ -347,6 +412,7 @@ def test_chat_templates(): our_prompt = correct_tokenizer.apply_chat_template(messages, tokenize = False, add_generation_prompt = True) assert(correct_prompt == our_prompt) + # Vicuna try: from fastchat.conversation import get_conv_template except: @@ -381,4 +447,11 @@ def test_chat_templates(): our_prompt = correct_tokenizer.apply_chat_template(messages[1:], tokenize = False, add_generation_prompt = True) # We add ourselves assert(correct_prompt == our_prompt.replace("", "")) + + # Gemma + correct_tokenizer = AutoTokenizer.from_pretrained("unsloth/gemma-7b-it") + correct_prompt = correct_tokenizer.apply_chat_template(messages[1:], tokenize = False, add_generation_prompt = True) + correct_tokenizer.chat_template = gemma_template + our_prompt = correct_tokenizer.apply_chat_template(messages[1:], tokenize = False, add_generation_prompt = True) + assert(our_prompt == correct_prompt) pass diff --git a/unsloth/models/gemma.py b/unsloth/models/gemma.py index 664a647f47..4aa634a4bd 100644 --- a/unsloth/models/gemma.py +++ b/unsloth/models/gemma.py @@ -264,7 +264,7 @@ class FastGemmaModel(FastLlamaModel): param.requires_grad_(False) pass - print("Unsloth: Patching Gemma RMS Layernorm + 1") + # Patch RMS Layernorm for name, module in model.named_modules(): if isinstance(module, GemmaRMSNorm): module.weight += 1.0 # return output * (1 + self.weight)