diff --git a/README.md b/README.md index 2a3322d629..3e56ec1307 100644 --- a/README.md +++ b/README.md @@ -1,68 +1,105 @@ -

- - - - unsloth logo - -

-

- - - -

+
-

- Finetune Mistral, Llama 2-5x faster with 50% less memory! -

-
+ + + + unsloth logo + + + + + -| Llama 2 7b | Mistral 7b | CodeLlama 34b | Llama 7b Kaggle 2x T4 | -|-----------------------------|-----------------------------|-------------------------|------------------------| -| **2.2x faster 43% less VRAM** | **2.2x faster 62% less VRAM** | **1.9x faster 27% less VRAM** | **5.5x faster 44% less VRAM** | -| [⭐Llama **free** Colab notebook](https://colab.research.google.com/drive/1lBzz5KeZJKXjvivbYvmGarix9Ao6Wxe5?usp=sharing") | [⭐Mistral **free** Colab notebook](https://colab.research.google.com/drive/1Dyauq4kTZoLewQ1cApceUQVNcnnNTzg_?usp=sharing) | [CodeLlama A100 Colab notebook](https://colab.research.google.com/drive/1y7A0AxE3y8gdj4AVkl2aZX47Xu3P1wJT?usp=sharing) | [⭐Kaggle **free** Alpaca notebook](https://www.kaggle.com/danielhanchen/unsloth-alpaca-t4-ddp) -| [Llama A100 Colab notebook](https://colab.research.google.com/drive/1YIPY_18xm-K0iJDgvNkRoJsgkPMPAO3G?usp=sharing) | [Mistral A100 Colab notebook](https://colab.research.google.com/drive/1SKrKGV-BZoU4kv5q3g0jtE_OhRgPtrrQ?usp=sharing) | 50+ more examples below! | [⭐Kaggle **free** Slim Orca notebook](https://www.kaggle.com/danielhanchen/unsloth-slimorca-t4-ddp) | +### Finetune Mistral, Llama 2-5x faster with 70% less memory! -* **NEW!** [DPO](https://arxiv.org/abs/2305.18290) support. ⭐**Free!** DPO Zephyr, Mistral example! [More info](#DPO) on DPO -* **NEW!** [TinyLlama 1.1b](https://github.com/jzhang38/TinyLlama) on 3T tokens! ⭐**Free!** example -* **NEW!** We're in 🤗 Huggingface's official docs! We're on the [SFT docs](https://huggingface.co/docs/trl/main/en/sft_trainer#accelerate-fine-tuning-2x-using-unsloth) and the [DPO docs](https://huggingface.co/docs/trl/main/en/dpo_trainer#accelerate-dpo-fine-tuning-using-unsloth)! -* Supports Llama, Yi, Mistral, CodeLlama, Qwen (llamafied), Deepseek and their derived models (Open Hermes etc). -* All kernels written in [OpenAI's Triton](https://openai.com/research/triton) language. **Manual backprop engine**. -* **0% loss in accuracy** - no approximation methods - all exact. -* No change of hardware. Supports NVIDIA GPUs since 2018+. Minimum CUDA Capability 7.0 (V100, T4, Titan V, RTX 20, 30, 40x, A100, H100, L40 etc) [Check your GPU!](https://developer.nvidia.com/cuda-gpus) GTX 1070, 1080 works, but is slow. -* Works on **Linux** and **Windows** via WSL. -* **NEW!** Download 4 bit models 4x faster from 🤗 Huggingface! Eg: `unsloth/mistral-7b-bnb-4bit` -* Supports 4bit and 16bit QLoRA / LoRA finetuning via [bitsandbytes](https://github.com/TimDettmers/bitsandbytes). -* **NEW!** Want a UI for finetuning? Try [Llama-Factory](https://github.com/hiyouga/LLaMA-Factory) and use `--use_unsloth`! -* Open source trains 5x faster - see [Unsloth Pro](https://unsloth.ai/) for **30x faster training**! +![](https://i.ibb.co/sJ7RhGG/image-41.png) -| 1 A100 40GB | 🤗 Hugging Face | Flash Attention | 🦥 Unsloth Open Source | [🦥 Unsloth Pro](https://unsloth.ai/pricing) | +
+ +## ✨ Finetune for Free + +All notebooks are **beginner friendly**! Colab provides a free GPU. Kaggle provides 30 hours for free per week. +| Unsloth supports | Free Notebooks | Performance | Memory use | +|-----------------|--------------------------------------------------------------------------------------------------------------------------|-------------|----------| +| **Mistral 7b** | [▶️ Start on Colab](https://colab.research.google.com/drive/1Dyauq4kTZoLewQ1cApceUQVNcnnNTzg_?usp=sharing) | 2.2x faster | 62% less | +| **Llama-2 7b** | [▶️ Start on Colab](https://colab.research.google.com/drive/1lBzz5KeZJKXjvivbYvmGarix9Ao6Wxe5?usp=sharing) | 2.2x faster | 43% less | +| **DPO - Zephyr** | [▶️ Start on Colab](https://colab.research.google.com/drive/15vttTpzzVXv_tJwEk-hIcQ0S9FcEWvwP?usp=sharing) | 1.9x faster | 19% less | +| **TinyLlama** | [▶️ Start on Colab](https://colab.research.google.com/drive/1AZghoNBQaMDgWJpi4RbffGM1h6raLUj9?usp=sharing) | 3.9x faster | 74% less | +| **CodeLlama 34b** A100 | [▶️ Start on Colab](https://colab.research.google.com/drive/1y7A0AxE3y8gdj4AVkl2aZX47Xu3P1wJT?usp=sharing) | 1.9x faster | 27% less | +| **Mistral 7b** 2xT4 | [▶️ Start on Kaggle](https://www.kaggle.com/code/danielhanchen/kaggle-mistral-7b-unsloth-notebook) | 5x faster | 60% less | + +- This [conversational notebook](https://colab.research.google.com/drive/1ef-tab5bhkvWmBOObepl1WgJvfvSzn5Q?usp=sharing) is useful for ShareGPT ChatML datatsets. +- Our [raw text notebook](https://colab.research.google.com/drive/1bMOKOBzxQWUIGZBs_B0zm8pimuEnZdfM?usp=sharing) is useful for text completion. + +## 🦥 Unsloth.ai News +- 📣 [DPO support](https://colab.research.google.com/drive/15vttTpzzVXv_tJwEk-hIcQ0S9FcEWvwP?usp=sharing) is now included. [More info](#DPO) on DPO. +- 📣 [TinyLlama 1.1b](https://colab.research.google.com/drive/1AZghoNBQaMDgWJpi4RbffGM1h6raLUj9?usp=sharing) on 3T tokens now works. +- 📣 We did a [blog](https://huggingface.co/blog/unsloth-trl) with 🤗Hugging Face! We're in 🤗Hugging Face's official docs! Check out the [SFT docs](https://huggingface.co/docs/trl/main/en/sft_trainer#accelerate-fine-tuning-2x-using-unsloth) and [DPO docs](https://huggingface.co/docs/trl/main/en/dpo_trainer#accelerate-dpo-fine-tuning-using-unsloth). +- 📣 Now supports **Llama, Yi, Mistral, CodeLlama, Qwen (llamafied), Deepseek** and their derived models (**Open Hermes** etc). Llama 7, 13, 70b; CodeLlama 7, 13, 34, 70b; Yi 6, 34b are all supported! +- 📣 **Download models 4x faster** from 🤗Hugging Face! Eg: `unsloth/mistral-7b-bnb-4bit` See our [HF collection](https://huggingface.co/collections/unsloth/load-4bit-models-4x-faster-659042e3a41c3cbad582e734) for more! + +## 🔗 Links and Resources +| Type | Links | +| ------------------------------- | --------------------------------------- | +| 📜 **Documentation** | [Read The Doc](https://github.com/unslothai/unsloth/tree/main#-documentation) | +| 💾 **Installation** | [unsloth/README.md](https://github.com/unslothai/unsloth/tree/main#installation-instructions)| +|   **Twitter (aka X)** | [Follow us on X](https://twitter.com/unslothai)| +| 🥇 **Benchmarking** | [Performance Tables](https://github.com/unslothai/unsloth/tree/main#-performance-benchmarking) +| 🌐 **Released Models** | [Unsloth Releases](https://huggingface.co/unsloth)| +| ✍️ **Blog** | [Read our Blogs](https://unsloth.ai/blog)| + +## ⭐ Key Features +- All kernels written in [OpenAI's Triton](https://openai.com/research/triton) language. **Manual backprop engine**. +- **0% loss in accuracy** - no approximation methods - all exact. +- No change of hardware. Supports NVIDIA GPUs since 2018+. Minimum CUDA Capability 7.0 (V100, T4, Titan V, RTX 20, 30, 40x, A100, H100, L40 etc) [Check your GPU!](https://developer.nvidia.com/cuda-gpus) GTX 1070, 1080 works, but is slow. +- Works on **Linux** and **Windows** via WSL. +- Supports 4bit and 16bit QLoRA / LoRA finetuning via [bitsandbytes](https://github.com/TimDettmers/bitsandbytes). +- Open source trains 5x faster - see [Unsloth Pro](https://unsloth.ai/) for **30x faster training**! +- If you trained a model with 🦥Unsloth, you can use this cool sticker!   + + +## 🥇 Performance Benchmarking +- For the full list of **reproducable** benchmarking tables, [go to our website](https://unsloth.ai/blog/mistral-benchmark#Benchmark%20tables) + +| 1 A100 40GB | 🤗Hugging Face | Flash Attention | 🦥Unsloth Open Source | 🦥[Unsloth Pro](https://unsloth.ai/pricing) | |--------------|--------------|-----------------|---------------------|-----------------| | Alpaca | 1x | 1.04x | 1.98x | **15.64x** | | LAION Chip2 | 1x | 0.92x | 1.61x | **20.73x** | | OASST | 1x | 1.19x | 2.17x | **14.83x** | | Slim Orca | 1x | 1.18x | 2.22x | **14.82x** | -Join our [Discord](https://discord.gg/nsS4V5Z6ge)! +- Benchmarking table below was conducted by [🤗Hugging Face](https://huggingface.co/blog/unsloth-trl). - -If you trained a model with 🦥 Unsloth, we made a cool sticker if you want to use it! +| Free Colab T4 | Dataset | 🤗Hugging Face | Pytorch 2.1.1 | 🦥Unsloth | 🦥 VRAM reduction | +| --- | --- | --- | --- | --- | --- | +| Llama-2 7b | OASST | 1x | 1.19x | 1.95x | -43.3% | +| Mistral 7b | Alpaca | 1x | 1.07x | 1.56x | -13.7% | +| Tiny Llama 1.1b | Alpaca | 1x | 2.06x | 3.87x | -73.8% | +| DPO with Zephyr | Ultra Chat | 1x | 1.09x | 1.55x | -18.6% | -# Installation Instructions - Conda -Select either `pytorch-cuda=11.8` for CUDA 11.8 or `pytorch-cuda=12.1` for CUDA 12.1. +![](https://i.ibb.co/sJ7RhGG/image-41.png) + +## 💾 Installation Instructions +### Conda Installation +Select either `pytorch-cuda=11.8` for CUDA 11.8 or `pytorch-cuda=12.1` for CUDA 12.1. If you have `mamba`, use `mamba` instead of `conda` for faster solving. See this [Github issue](https://github.com/unslothai/unsloth/issues/73) for help on debugging Conda installs. ```bash -conda install cudatoolkit xformers bitsandbytes pytorch pytorch-cuda=12.1 \ - -c pytorch -c nvidia -c xformers -c conda-forge -y +conda install pytorch torchvision torchaudio pytorch-cuda=<12.1/11.8> -c pytorch -c nvidia + +conda install xformers -c xformers -y + +pip install bitsandbytes + pip install "unsloth[conda] @ git+https://github.com/unslothai/unsloth.git" ``` -# Installation Instructions - Pip +### Pip Installation Do **NOT** use this if you have Anaconda. You must use the Conda install method, or else stuff will BREAK. 1. Find your CUDA version via ```python import torch; torch.version.cuda ``` -2. For Pytorch 2.1.0: You can update Pytorch via Pip (interchange `cu121` / `cu118`). Go to https://pytorch.org/ to learn more. Select either `cu118` for CUDA 11.8 or `cu121` for CUDA 12.1. If you have a RTX 3060 or higher (A100, H100 etc), use the `"ampere"` path. For Pytorch 2.1.1: got to step 3. +2. For Pytorch 2.1.0: You can update Pytorch via Pip (interchange `cu121` / `cu118`). Go to https://pytorch.org/ to learn more. Select either `cu118` for CUDA 11.8 or `cu121` for CUDA 12.1. If you have a RTX 3060 or higher (A100, H100 etc), use the `"ampere"` path. For Pytorch 2.1.1: go to step 3. For Pytorch 2.2.0: go to step 4. ```bash pip install --upgrade --force-reinstall --no-cache-dir torch==2.1.0 triton \ --index-url https://download.pytorch.org/whl/cu121 @@ -84,16 +121,25 @@ pip install "unsloth[cu121_torch211] @ git+https://github.com/unslothai/unsloth. pip install "unsloth[cu118_ampere_torch211] @ git+https://github.com/unslothai/unsloth.git" pip install "unsloth[cu121_ampere_torch211] @ git+https://github.com/unslothai/unsloth.git" ``` -4. We're working on Pytorch 2.1.2 support. +4. For Pytorch 2.2.0: Use the `"ampere"` path for newer RTX 30xx GPUs or higher. +```bash +pip install --upgrade --force-reinstall --no-cache-dir torch==2.2.0 triton \ + --index-url https://download.pytorch.org/whl/cu121 +``` +```bash +pip install "unsloth[cu118_torch220] @ git+https://github.com/unslothai/unsloth.git" +pip install "unsloth[cu121_torch220] @ git+https://github.com/unslothai/unsloth.git" +pip install "unsloth[cu118_ampere_torch220] @ git+https://github.com/unslothai/unsloth.git" +pip install "unsloth[cu121_ampere_torch220] @ git+https://github.com/unslothai/unsloth.git" +``` 5. If you get errors, try the below first, then go back to step 1: ```bash pip install --upgrade pip ``` -# Documentation -We support Huggingface's TRL, Trainer, Seq2SeqTrainer or even Pytorch code! - -We're in 🤗 Huggingface's official docs! We're on the [SFT docs](https://huggingface.co/docs/trl/main/en/sft_trainer#accelerate-fine-tuning-2x-using-unsloth) and the [DPO docs](https://huggingface.co/docs/trl/main/en/dpo_trainer#accelerate-dpo-fine-tuning-using-unsloth)! +## 📜 Documentation +- We support Huggingface's TRL, Trainer, Seq2SeqTrainer or even Pytorch code! +- We're in 🤗Hugging Face's official docs! Check out the [SFT docs](https://huggingface.co/docs/trl/main/en/sft_trainer#accelerate-fine-tuning-2x-using-unsloth) and [DPO docs](https://huggingface.co/docs/trl/main/en/dpo_trainer#accelerate-dpo-fine-tuning-using-unsloth)! ```python from unsloth import FastLanguageModel @@ -159,10 +205,10 @@ trainer.train() ``` -# DPO (Direct Preference Optimization) Support -DPO, PPO, Reward Modelling all seem to work as per 3rd party independent testing from [Llama-Factory](https://github.com/hiyouga/LLaMA-Factory). We have a preliminary Google Colab notebook for reproducing Zephyr on Tesla T4 here: [notebook](https://colab.research.google.com/drive/15vttTpzzVXv_tJwEk-hIcQ0S9FcEWvwP?usp=sharing). +## DPO Support +DPO (Direct Preference Optimization), PPO, Reward Modelling all seem to work as per 3rd party independent testing from [Llama-Factory](https://github.com/hiyouga/LLaMA-Factory). We have a preliminary Google Colab notebook for reproducing Zephyr on Tesla T4 here: [notebook](https://colab.research.google.com/drive/15vttTpzzVXv_tJwEk-hIcQ0S9FcEWvwP?usp=sharing). -We're in 🤗 Huggingface's official docs! We're on the [SFT docs](https://huggingface.co/docs/trl/main/en/sft_trainer#accelerate-fine-tuning-2x-using-unsloth) and the [DPO docs](https://huggingface.co/docs/trl/main/en/dpo_trainer#accelerate-dpo-fine-tuning-using-unsloth)! +We're in 🤗Hugging Face's official docs! We're on the [SFT docs](https://huggingface.co/docs/trl/main/en/sft_trainer#accelerate-fine-tuning-2x-using-unsloth) and the [DPO docs](https://huggingface.co/docs/trl/main/en/dpo_trainer#accelerate-dpo-fine-tuning-using-unsloth)! ```python from unsloth import FastLanguageModel, PatchDPOTrainer @@ -217,17 +263,76 @@ dpo_trainer = DPOTrainer( dpo_trainer.train() ``` -# Support us! -We're currently 2 brothers trying to make LLMs for everyone! It'll be super cool if you can support our work!! - +## 🥇 Detailed Benchmarking Tables +- Click "Code" for fully reproducible examples +- "Unsloth Equal" is a preview of our PRO version, with code stripped out. All settings and the loss curve remains identical. +- For the full list of benchmarking tables, [go to our website](https://unsloth.ai/blog/mistral-benchmark#Benchmark%20tables) + +| 1 A100 40GB | 🤗Hugging Face | Flash Attention 2 | 🦥Unsloth Open | Unsloth Equal | Unsloth Pro | Unsloth Max | +|--------------|-------------|-------------|-----------------|--------------|---------------|-------------| +| Alpaca | 1x | 1.04x | 1.98x | 2.48x | 5.32x | **15.64x** | +| code | [Code](https://colab.research.google.com/drive/1u4dBeM-0vGNVmmO6X7cScAut-Hyt4KDF?usp=sharing) | [Code](https://colab.research.google.com/drive/1fgTOxpMbVjloQBvZyz4lF4BacKSZOB2A?usp=sharing) | [Code](https://colab.research.google.com/drive/1YIPY_18xm-K0iJDgvNkRoJsgkPMPAO3G?usp=sharing) | [Code](https://colab.research.google.com/drive/1ANW8EFL3LVyTD7Gq4TkheC1Z7Rxw-rHp?usp=sharing) | | | +| seconds| 1040 | 1001 | 525 | 419 | 196 | 67 | +| memory MB| 18235 | 15365 | 9631 | 8525 | | | +| % saved| | 15.74 | 47.18 | 53.25 | | | | -# Future Milestones and limitations -1. Support Mixtral. -2. Supports all Mistral, Llama type models, but some are unoptimized (Qwen with biases) -3. Dropout, bias in LoRA matrices are supported, just not optimized. +### Llama-Factory 3rd party benchmarking +- [Link to performance table.](https://github.com/hiyouga/LLaMA-Factory/wiki/Performance-Comparison) TGS: tokens per GPU per second. Model: LLaMA2-7B. GPU: NVIDIA A100 * 1. Batch size: 4. Gradient accumulation: 2. LoRA rank: 8. Max length: 1024. -# Performance comparisons on 1 Tesla T4 GPU: -**Time taken for 1 epoch** +| Method | Bits | TGS | GRAM | Speed | +| --- | --- | --- | --- | --- | +| HF | 16 | 2392 | 18GB | 100% | +| HF+FA2 | 16 | 2954 | 17GB | 123% | +| Unsloth+FA2 | 16 | 4007 | 16GB | **168%** | +| HF | 4 | 2415 | 9GB | 101% | +| Unsloth+FA2 | 4 | 3726 | 7GB | **160%** | + +### Performance comparisons between popular models +
+ Click for specific model benchmarking tables (Mistral 7b, CodeLlama 34b etc.) + +### Mistral 7b +| 1 A100 40GB | Hugging Face | Flash Attention 2 | Unsloth Open | Unsloth Equal | Unsloth Pro | Unsloth Max | +|--------------|-------------|-------------|-----------------|--------------|---------------|-------------| +| Mistral 7B Slim Orca | 1x | 1.15x | 2.15x | 2.53x | 4.61x | **13.69x** | +| code | [Code](https://colab.research.google.com/drive/1mePk3KzwTD81hr5mcNcs_AX3Kbg_Ha0x?usp=sharing) | [Code](https://colab.research.google.com/drive/1dgHxjvTmX6hb0bPcLp26RXSE6_n9DKj7?usp=sharing) | [Code](https://colab.research.google.com/drive/1SKrKGV-BZoU4kv5q3g0jtE_OhRgPtrrQ?usp=sharing) | [Code](https://colab.research.google.com/drive/18yOiyX0T81mTwZqOALFSCX_tSAqju6aD?usp=sharing) | | +| seconds | 1813 | 1571 | 842 | 718 | 393 | 132 | +| memory MB | 32853 | 19385 | 12465 | 10271 | | | +| % saved| | 40.99 | 62.06 | 68.74 | | | + +### CodeLlama 34b +| 1 A100 40GB | Hugging Face | Flash Attention 2 | Unsloth Open | Unsloth Equal | Unsloth Pro | Unsloth Max | +|--------------|-------------|-------------|-----------------|--------------|---------------|-------------| +| Code Llama 34B | OOM ❌ | 0.99x | 1.87x | 2.61x | 4.27x | 12.82x | +| code | [▶️ Code](https://colab.research.google.com/drive/1ykfz3BqrtC_AUFegCzUQjjfUNlxp6Otc?usp=sharing) | [Code](https://colab.research.google.com/drive/12ZypxQh7OC6kBXvWZI-5d05I4m-B_hoR?usp=sharing) | [Code](https://colab.research.google.com/drive/1gdHyAx8XJsz2yNV-DHvbHjR1iCef5Qmh?usp=sharing) | [Code](https://colab.research.google.com/drive/1fm7wqx9MJ0kRrwKOfmLkK1Rmw-pySahB?usp=sharing) | | +| seconds | 1953 | 1982 | 1043 | 748 | 458 | 152 | +| memory MB | 40000 | 33217 | 27413 | 22161 | | | +| % saved| | 16.96| 31.47 | 44.60 | | | | + +### 1 Tesla T4 + +| 1 T4 16GB | Hugging Face | Flash Attention | Unsloth Open | Unsloth Pro Equal | Unsloth Pro | Unsloth Max | +|--------------|-------------|-----------------|-----------------|---------------|---------------|-------------| +| Alpaca | 1x | 1.09x | 1.69x | 1.79x | 2.93x | **8.3x** | +| code | [▶️ Code](https://colab.research.google.com/drive/1XpLIV4s8Bj5uryB-X2gqM88oRGHEGdaB?usp=sharing) | [Code](https://colab.research.google.com/drive/1LyXu6CjuymQg6ddHX8g1dpUvrMa1nn4L?usp=sharing) | [Code](https://colab.research.google.com/drive/1gsv4LpY7C32otl1rgRo5wXTk4HIitXoM?usp=sharing) | [Code](https://colab.research.google.com/drive/1VtULwRQwhEnVdNryjm27zXfdSM1tNfFK?usp=sharing) | | | +| seconds | 1599 | 1468 | 942 | 894 | 545 | 193 | +| memory MB | 7199 | 7059 | 6459 | 5443 | | | +| % saved | | 1.94 | 10.28 | 24.39 | | | + +### 2 Tesla T4s via DDP + + | 2 T4 DDP | Hugging Face | Flash Attention | Unsloth Open | Unsloth Equal | Unsloth Pro | Unsloth Max | +|--------------|----------|-------------|-----------------|--------------|---------------|-------------| +| Alpaca | 1x | 0.99x | 4.95x | 4.44x | 7.28x | **20.61x** | +| code | [▶️ Code](https://www.kaggle.com/danielhanchen/hf-original-alpaca-t4-ddp) | [Code](https://www.kaggle.com/danielhanchen/hf-sdpa-alpaca-t4-ddp) | [Code](https://www.kaggle.com/danielhanchen/unsloth-alpaca-t4-ddp) | | | +| seconds | 9882 | 9946 | 1996 | 2227 | 1357 | 480 | +| memory MB| 9176 | 9128 | 6904 | 6782 | | | +| % saved | | 0.52 | 24.76 | 26.09 | | | | +
+ +### Performance comparisons on 1 Tesla T4 GPU: +
+ Click for Time taken for 1 epoch One Tesla T4 on Google Colab `bsz = 2, ga = 4, max_grad_norm = 0.3, num_train_epochs = 1, seed = 3047, lr = 2e-4, wd = 0.01, optim = "adamw_8bit", schedule = "linear", schedule_steps = 10` @@ -247,8 +352,10 @@ One Tesla T4 on Google Colab | Unsloth Open | 1 T4 | 6.8GB | 5.7GB | 7.8GB | 7.7GB | | Unsloth Pro | 1 T4 | 6.4GB | 6.4GB | 6.4GB | 6.4GB | | Unsloth Max | 1 T4 | 11.4GB | 12.4GB | 11.9GB | 14.4GB | +
-# Performance comparisons on 2 Tesla T4 GPUs via DDP: +
+ Click for Performance Comparisons on 2 Tesla T4 GPUs via DDP: **Time taken for 1 epoch** Two Tesla T4s on Kaggle @@ -269,173 +376,12 @@ Two Tesla T4s on Kaggle | Unsloth Max | 2 T4 | 10.5GB \| 5GB | 10.6GB \| 5GB | 10.6GB \| 5GB | 10.5GB \| 5GB * | * Slim Orca `bsz=1` for all benchmarks since `bsz=2` OOMs. We can handle `bsz=2`, but we benchmark it with `bsz=1` for consistency. +
-# Llama-Factory 3rd party benchmarking +![](https://i.ibb.co/sJ7RhGG/image-41.png) +
-| Method | Bits | TGS | GRAM | Speed | -| --- | --- | --- | --- | --- | -| HF | 16 | 2392 | 18GB | 100% | -| HF+FA2 | 16 | 2954 | 17GB | 123% | -| Unsloth+FA2 | 16 | 4007 | 16GB | **168%** | -| HF | 4 | 2415 | 9GB | 101% | -| Unsloth+FA2 | 4 | 3726 | 7GB | **160%** | - -[Link](https://github.com/hiyouga/LLaMA-Factory/wiki/Performance-Comparison) to performance table. TGS: tokens per GPU per second. Model: LLaMA2-7B. GPU: NVIDIA A100 * 1. Batch size: 4. Gradient accumulation: 2. LoRA rank: 8. Max length: 1024. - -# How did we make it faster? -Manual autograd, Triton kernels etc. See our [Benchmark Breakdown](https://unsloth.ai/blog/mistral-benchmark) for more info! - -# Troubleshooting -1. Sometimes `bitsandbytes` or `xformers` does not link properly. Try running: -```bash -!ldconfig /usr/lib64-nvidia -``` -2. Windows is not supported as of yet - we rely on Xformers and Triton support, so until both packages support Windows officially, Unsloth will then support Windows. - -3. If it doesn't install - maybe try updating `pip`. - - -# Full benchmarking tables -Click "Code" for a fully reproducible example. -"Unsloth Equal" is a preview of our PRO version, with code stripped out. All settings and the loss curve remains identical. -| 1 A100 40GB | Hugging Face | Flash Attention 2 | Unsloth Open | Unsloth Equal | Unsloth Pro | Unsloth Max | -|--------------|-------------|-------------|-----------------|--------------|---------------|-------------| -| Alpaca | 1x | 1.04x | 1.98x | 2.48x | 5.32x | **15.64x** | -| code | [Code](https://colab.research.google.com/drive/1u4dBeM-0vGNVmmO6X7cScAut-Hyt4KDF?usp=sharing) | [Code](https://colab.research.google.com/drive/1fgTOxpMbVjloQBvZyz4lF4BacKSZOB2A?usp=sharing) | [Code](https://colab.research.google.com/drive/1YIPY_18xm-K0iJDgvNkRoJsgkPMPAO3G?usp=sharing) | [Code](https://colab.research.google.com/drive/1ANW8EFL3LVyTD7Gq4TkheC1Z7Rxw-rHp?usp=sharing) | | | -| seconds| 1040 | 1001 | 525 | 419 | 196 | 67 | -| memory MB| 18235 | 15365 | 9631 | 8525 | | | -| % saved| | 15.74 | 47.18 | 53.25 | | | | - - -| 1 A100 40GB | Hugging Face | Flash Attention 2 | Unsloth Open | Unsloth Equal | Unsloth Pro | Unsloth Max | -|--------------|-------------|-------------|-----------------|--------------|---------------|-------------| -| LAION Chip2 | 1x | 0.92x | 1.61x | 1.84x | 7.05x | **20.73x** | -| code |[Code](https://colab.research.google.com/drive/1gjL1TaKwc_xv2TcxJC8QWEWBG1msh3g2?usp=sharing) | [Code](https://colab.research.google.com/drive/15vlPjMr8xDj5BFhGdqunGaOQSMqXPEXU?usp=sharing) | [Code](https://colab.research.google.com/drive/1zPwvf-BmHyHlPMBxDsY8zS0BnQ-KKbCc?usp=sharing) | [Code](https://colab.research.google.com/drive/1X2uHy-arRsZxqWHvKHwwW102JaMwChD2?usp=sharing) | | | -| seconds| 581 | 631 | 361 | 315 | 82 | 28 | -| memory MB| 7763 | 8047 | 7763 | 6441 | | | -| % saved| | -3.66 | 0.00 | 17.03 | | | | - - -| 1 A100 40GB | Hugging Face | Flash Attention 2 | Unsloth Open | Unsloth Equal | Unsloth Pro | Unsloth Max | -|--------------|-------------|-------------|-----------------|--------------|---------------|-------------| -| OASST | 1x | 1.19x | 2.17x | 2.66x | 5.04x | **14.83x** | -| code |[Code](https://colab.research.google.com/drive/10NzDreFbuWELGUuBv0MOoC7y3MBewaNx?usp=sharing) | [Code](https://colab.research.google.com/drive/1TwdkJ1sHsuEH-kgeCPqSFeCpOnCfz6Ou?usp=sharing) | [Code](https://colab.research.google.com/drive/1AkwjUkOF0XeRBMT_S8Uhh74kitEsZHla?usp=sharing) | [Code](https://colab.research.google.com/drive/1roMkp2UjbeK2t3DkNz50cRs1MT92RPFT?usp=sharing) | | | -| seconds| 1852 | 1558 | 852 | 696 | 367 | 125 | -| memory MB| 26431 | 16565 | 12267| 11223| | | -| % saved| | 37.33 | 53.59 | 57.54 | | | - -| 1 A100 40GB | Hugging Face | Flash Attention 2 | Unsloth Open | Unsloth Equal | Unsloth Pro | Unsloth Max | -|--------------|-------------|-------------|-----------------|--------------|---------------|-------------| -| Slim Orca | 1x | 1.18x | 2.22x | 2.64x | 5.04x | **14.82x** | -| code |[Code](https://colab.research.google.com/drive/1UNo1xsMl8YH7xnWnIVjDFnCAPfc0RGgu?usp=sharing) | [Code](https://colab.research.google.com/drive/1zbphER-SKhbSWGjHTfnBLPFyTgIVvaeH?usp=sharing) | [Code](https://colab.research.google.com/drive/156si33585iv4Uh-VILFglUmIMrNCNuc2?usp=sharing) | [Code](https://colab.research.google.com/drive/1_mhZy7dfl9jEnJRuJBZJ5y3OwW06jgQA?usp=sharing) | | | -| seconds| 1824 | 1545 | 821 | 691 | 362 | 123 | -| memory MB| 24557 | 15681 | 10595| 9007 | | | -| % saved| | 36.14 | 56.86 | 63.32 | | | - -### Mistral 7b -| 1 A100 40GB | Hugging Face | Flash Attention 2 | Unsloth Open | Unsloth Equal | Unsloth Pro | Unsloth Max | -|--------------|-------------|-------------|-----------------|--------------|---------------|-------------| -| Mistral 7B Slim Orca | 1x | 1.15x | 2.15x | 2.53x | 4.61x | **13.69x** | -| code | [Code](https://colab.research.google.com/drive/1mePk3KzwTD81hr5mcNcs_AX3Kbg_Ha0x?usp=sharing) | [Code](https://colab.research.google.com/drive/1dgHxjvTmX6hb0bPcLp26RXSE6_n9DKj7?usp=sharing) | [Code](https://colab.research.google.com/drive/1SKrKGV-BZoU4kv5q3g0jtE_OhRgPtrrQ?usp=sharing) | [Code](https://colab.research.google.com/drive/18yOiyX0T81mTwZqOALFSCX_tSAqju6aD?usp=sharing) | | -| seconds | 1813 | 1571 | 842 | 718 | 393 | 132 | -| memory MB | 32853 | 19385 | 12465 | 10271 | | | -| % saved| | 40.99 | 62.06 | 68.74 | | | - -### CodeLlama 34b -| 1 A100 40GB | Hugging Face | Flash Attention 2 | Unsloth Open | Unsloth Equal | Unsloth Pro | Unsloth Max | -|--------------|-------------|-------------|-----------------|--------------|---------------|-------------| -| Code Llama 34B | OOM ❌ | 0.99x | 1.87x | 2.61x | 4.27x | 12.82x | -| code | [Code](https://colab.research.google.com/drive/1ykfz3BqrtC_AUFegCzUQjjfUNlxp6Otc?usp=sharing) | [Code](https://colab.research.google.com/drive/12ZypxQh7OC6kBXvWZI-5d05I4m-B_hoR?usp=sharing) | [Code](https://colab.research.google.com/drive/1gdHyAx8XJsz2yNV-DHvbHjR1iCef5Qmh?usp=sharing) | [Code](https://colab.research.google.com/drive/1fm7wqx9MJ0kRrwKOfmLkK1Rmw-pySahB?usp=sharing) | | -| seconds | 1953 | 1982 | 1043 | 748 | 458 | 152 | -| memory MB | 40000 | 33217 | 27413 | 22161 | | | -| % saved| | 16.96| 31.47 | 44.60 | | | | - -### 1 Tesla T4 - -| 1 T4 16GB | Hugging Face | Flash Attention | Unsloth Open | Unsloth Pro Equal | Unsloth Pro | Unsloth Max | -|--------------|-------------|-----------------|-----------------|---------------|---------------|-------------| -| Alpaca | 1x | 1.09x | 1.69x | 1.79x | 2.93x | **8.3x** | -| code | [Code](https://colab.research.google.com/drive/1XpLIV4s8Bj5uryB-X2gqM88oRGHEGdaB?usp=sharing) | [Code](https://colab.research.google.com/drive/1LyXu6CjuymQg6ddHX8g1dpUvrMa1nn4L?usp=sharing) | [Code](https://colab.research.google.com/drive/1gsv4LpY7C32otl1rgRo5wXTk4HIitXoM?usp=sharing) | [Code](https://colab.research.google.com/drive/1VtULwRQwhEnVdNryjm27zXfdSM1tNfFK?usp=sharing) | | | -| seconds | 1599 | 1468 | 942 | 894 | 545 | 193 | -| memory MB | 7199 | 7059 | 6459 | 5443 | | | -| % saved | | 1.94 | 10.28 | 24.39 | | | - -| 1 T4 16GB | Hugging Face | Flash Attention | Unsloth Open | Unsloth Pro Equal | Unsloth Pro | Unsloth Max | -|--------------|-------------|-----------------|-----------------|---------------|---------------|-------------| -| LAION Chip2 | 1x | 0.99x | 1.80x | 1.75x | 4.15x | **11.75x** | -| code | [Code](https://colab.research.google.com/drive/1EtdStADehE4FVJnU2Cu6O8p9jDYdqG2L?usp=sharing) | [Code](https://colab.research.google.com/drive/1Ik4jO68odUiQIJ_szZ3xok5fk58WpA5Q?usp=sharing) | [Code](https://colab.research.google.com/drive/1E2nR4V3bXIWBQIUE7uR39lYPr3UikzqH?usp=sharing) | [Code](https://colab.research.google.com/drive/13jbj8D8FOt9KyXwZt9Yf2MsYkD8CyCVR?usp=sharing) | | | -| seconds | 952 | 955 | 529 | 543 | 229 | 81 | -| memory MB | 6037 | 6033 | 5797 | 4855 | | | -| % saved | | 0.07 | 3.98 | 19.58 | | | - -| 1 T4 16GB | Hugging Face | Flash Attention | Unsloth Open | Unsloth Pro Equal | Unsloth Pro | Unsloth Max | -|--------------|-------------|-----------------|-----------------|---------------|---------------|-------------| -| OASST | 1x | 1.19x | 1.95x | 1.86x | 2.58x | **7.3x** | -| code | [Code](https://colab.research.google.com/drive/1aXzGgEM3yYB6SWy_XR81nQFWME40ksSy?usp=sharing) | [Code](https://colab.research.google.com/drive/1-5MdIOp0cM0scC-CdRZhh8OYhnGHqct4?usp=sharing) | [Code](https://colab.research.google.com/drive/1n-fgduZhRUsSjgpqNtVkXA3rSfE7iBdg?usp=sharing) | [Code](https://colab.research.google.com/drive/1z_GlHr2M_bB4lQrPhdWC7dseZv23cBIy?usp=sharing) | | | -| seconds | 2640 | 2222 | 1355 | 1421 | 1024 | 362 | -| memory MB | 14827 | 10391 | 8413 | 7031 | | | -| % saved | | 29.92 | 43.26 | 52.58 | | | - -| 1 T4 16GB | Hugging Face | Flash Attention | Unsloth Open | Unsloth Pro Equal | Unsloth Pro | Unsloth Max | -|--------------|-------------|-----------------|-----------------|---------------|---------------|-------------| -| Slim Orca | 1x | 1.21x | 1.77x | 1.85x | 2.71x | **7.67x** | -| code | [Code](https://colab.research.google.com/drive/15yLlJx9IE84kzx7ikky45pRcarPyUtEs?usp=sharing) | [Code](https://colab.research.google.com/drive/16IShIBmjKULWy87I-xURpj4nztTkAF13?usp=sharing) | [Code](https://colab.research.google.com/drive/1CJG3XLg_OQpCz71eB7Uqx7wuK_n2b-a8?usp=sharing) | [Code](https://colab.research.google.com/drive/1UmwuWHtlrC6MAfl9mX7A_TRfo5iSHDa-?usp=sharing) | | | -| seconds | 2735 | 2262 | 1545 | 1478 | 1009 | 356 | -| memory MB | 13933 | 10489 | 7661 | 6563 | | | -| % saved | | 24.72 | 45.02 | 52.90 | | | - -### 2 Tesla T4s via DDP - - | 2 T4 DDP | Hugging Face | Flash Attention | Unsloth Open | Unsloth Equal | Unsloth Pro | Unsloth Max | -|--------------|----------|-------------|-----------------|--------------|---------------|-------------| -| Alpaca | 1x | 0.99x | 4.95x | 4.44x | 7.28x | **20.61x** | -| code | [Code](https://www.kaggle.com/danielhanchen/hf-original-alpaca-t4-ddp) | [Code](https://www.kaggle.com/danielhanchen/hf-sdpa-alpaca-t4-ddp) | [Code](https://www.kaggle.com/danielhanchen/unsloth-alpaca-t4-ddp) | | | -| seconds | 9882 | 9946 | 1996 | 2227 | 1357 | 480 | -| memory MB| 9176 | 9128 | 6904 | 6782 | | | -| % saved | | 0.52 | 24.76 | 26.09 | | | | - - | 2 T4 DDP | Hugging Face | Flash Attention | Unsloth Open | Unsloth Equal | Unsloth Pro | Unsloth Max | -|--------------|----------|-------------|-----------------|--------------|---------------|-------------| -| LAION Chip2 | 1x | 1.12x | 5.28x | 4.21x | 10.01x | **28.32x** | -| code | [Code](https://www.kaggle.com/danielhanchen/hf-original-laion-t4-ddp) | [Code](https://www.kaggle.com/danielhanchen/hf-sdpa-laion-t4-ddp) | [Code](https://www.kaggle.com/danielhanchen/unsloth-laion-t4-ddp) | | | -| seconds | 5418 | 4854 | 1027 | 1286 | 541 | 191 | -| memory MB| 7316 | 7316 | 5732 | 5934 | | | -| % saved | | 0.00 | 21.65 | 18.89 | | | - - | 2 T4 DDP | Hugging Face | Flash Attention | Unsloth Open | Unsloth Equal | Unsloth Pro | Unsloth Max | -|--------------|----------|-------------|-----------------|--------------|---------------|-------------| -| OASST (bsz=1) | 1x | 1.14x | 5.56x | 5.09x | 5.64x | **15.97x** | -| code | [Code](https://www.kaggle.com/danielhanchen/hf-original-oasst-bsz1-t4-ddp) | [Code](https://www.kaggle.com/danielhanchen/hf-sdpa-oasst-bsz1-t4-ddp) | [Code](https://www.kaggle.com/danielhanchen/unsloth-oasst-bsz1-t4-ddp) | | | | -| seconds | 4503 | 3955 | 811 | 885 | 798 | 282 | -| memory MB | 11896 | 11628 | 6616 | 7105 | | | -| % saved | | 2.25 | 44.38 | 40.27 | | | - - | 2 T4 DDP | Hugging Face | Flash Attention | Unsloth Open | Unsloth Equal | Unsloth Pro | Unsloth Max | -|--------------|----------|-------------|-----------------|--------------|---------------|-------------| -| Slim Orca (bsz=1) | 1x | 0.97x | 5.54x | 4.68x | 6.88x | **19.46x** | -| code | [Code](https://www.kaggle.com/danielhanchen/hf-original-slimorca-bsz1-t4-ddp) | [Code](https://www.kaggle.com/danielhanchen/hf-sdpa-slimorca-bsz1-t4-ddp) | [Code](https://www.kaggle.com/danielhanchen/unsloth-slimorca-bsz1-t4-ddp) | | | -| seconds | 4042 | 4158 | 729 | 863 | 588 | 208 | -| memory MB| 11010 | 11042 | 6492 | 7410 | | | -| % saved | | -0.29| 41.04 | 32.70 | | | | - - | 2 T4 DDP | Hugging Face | Flash Attention | Unsloth Open | Unsloth Equal | Unsloth Pro | Unsloth Max | -|--------------|----------|-------------|-----------------|--------------|---------------|-------------| -| OASST (bsz=2) | OOM ❌ | OOM ❌ | ✓ | ✓ | ✓ | ✓ | -| code | [Code](https://www.kaggle.com/danielhanchen/hf-original-oasst-t4-ddp) | [Code](https://www.kaggle.com/danielhanchen/hf-sdpa-oasst-t4-ddp) | [Code](https://www.kaggle.com/danielhanchen/unsloth-oasst-t4-ddp) | | | | -| seconds | OOM | OOM | 2719 | 3391 | 2794 | 987 | -| memory MB| OOM | OOM | 8134 | 9600 | | | -| % saved | OOM | OOM | | | | | - - | 2 T4 DDP | Hugging Face | Flash Attention | Unsloth Open | Unsloth Equal | Unsloth Pro | Unsloth Max | -|--------------|----------|-------------|-----------------|--------------|---------------|-------------| -| Slim Orca (bsz=2) | OOM ❌ | OOM ❌ | ✓ | ✓ | ✓ |✓ | -| code | [Code](https://www.kaggle.com/danielhanchen/hf-original-slimorca-t4-ddp) | [Code](https://www.kaggle.com/danielhanchen/hf-sdpa-slimorca-t4-ddp) | [Code](https://www.kaggle.com/danielhanchen/unsloth-slimorca-t4-ddp) | | | -| seconds | OOM | OOM | 2990 | 3444 | 2351 | 831 | -| memory MB| OOM | OOM | 7594 | 8881 | | | -| % saved | OOM | OOM | | | | | - -# Credits +### Credits 1. [RandomInternetPreson](https://github.com/RandomInternetPreson) for confirming WSL support 2. [152334H](https://github.com/152334H) for experimental DPO support 3. [atgctg](https://github.com/atgctg) for syntax highlighting - diff --git a/images/buy me a coffee button.png b/images/buy me a coffee button.png new file mode 100644 index 0000000000..5eccb8e94b Binary files /dev/null and b/images/buy me a coffee button.png differ diff --git a/images/made with unsloth.png b/images/made with unsloth.png new file mode 100644 index 0000000000..6c5e90dbb0 Binary files /dev/null and b/images/made with unsloth.png differ diff --git a/images/start free finetune button.png b/images/start free finetune button.png new file mode 100644 index 0000000000..0f6c879fd4 Binary files /dev/null and b/images/start free finetune button.png differ diff --git a/images/unsloth end.png b/images/unsloth end.png new file mode 100644 index 0000000000..f8a774a4bc Binary files /dev/null and b/images/unsloth end.png differ