diff --git a/README.md b/README.md index 5e4add0a31..1f85647f94 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,14 @@ All notebooks are **beginner friendly**! Add your dataset, click "Run All", and - This [continued pretraining notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-CPT.ipynb) is for learning another language - Click [here](https://docs.unsloth.ai/) for detailed documentation for Unsloth. +## ⚡ Quickstart + +- **Install with pip (recommended)** for Linux devices: +``` +pip install unsloth +``` +For Windows install instructions, see [here](https://github.com/unslothai/unsloth/edit/main/README.md#windows-installation). + ## 🦥 Unsloth.ai News - 📣 NEW! Introducing Long-context [Reasoning (GRPO)](https://unsloth.ai/blog/grpo) in Unsloth. You can now reproduce DeepSeek-R1's "aha" moment with just 5GB VRAM. Transform Llama, Phi, Mistral etc. into reasoning LLMs! - 📣 NEW! [DeepSeek-R1](https://unsloth.ai/blog/deepseek-r1) - the most powerful open reasoning models with Llama & Qwen distillations. Run or fine-tune them now! More details: [unsloth.ai/blog/deepseek-r1](https://unsloth.ai/blog/deepseek-r1). All model uploads: [here](https://huggingface.co/collections/unsloth/deepseek-r1-all-versions-678e1c48f5d2fce87892ace5). @@ -65,7 +73,7 @@ All notebooks are **beginner friendly**! Add your dataset, click "Run All", and | ------------------------------- | --------------------------------------- | | 📚 **Documentation & Wiki** | [Read Our Docs](https://docs.unsloth.ai) | |   **Twitter (aka X)** | [Follow us on X](https://twitter.com/unslothai)| -| 💾 **Installation** | [Pip install](https://github.com/unslothai/unsloth/edit/main/README.md#-install-unsloth)| +| 💾 **Installation** | [Pip install](https://docs.unsloth.ai/get-started/installing-+-updating)| | 🔮 **Our Models** | [Unsloth Releases](https://docs.unsloth.ai/get-started/all-our-models)| | ✍️ **Blog** | [Read our Blogs](https://unsloth.ai/blog)| |   **Reddit** | [Join our Reddit page](https://reddit.com/r/unsloth)| @@ -74,17 +82,63 @@ All notebooks are **beginner friendly**! Add your dataset, click "Run All", and - All kernels written in [OpenAI's Triton](https://openai.com/index/triton/) language. **Manual backprop engine**. - **0% loss in accuracy** - no approximation methods - all exact. - No change of hardware. Supports NVIDIA GPUs since 2018+. Minimum CUDA Capability 7.0 (V100, T4, Titan V, RTX 20, 30, 40x, A100, H100, L40 etc) [Check your GPU!](https://developer.nvidia.com/cuda-gpus) GTX 1070, 1080 works, but is slow. -- Works on **Linux** and **Windows** via WSL. +- Works on **Linux** and **Windows** - Supports 4bit and 16bit QLoRA / LoRA finetuning via [bitsandbytes](https://github.com/TimDettmers/bitsandbytes). - If you trained a model with 🦥Unsloth, you can use this cool sticker!   ## 💾 Install Unsloth +You can also see our documentation for more detailed installation and updating instructions [here](https://docs.unsloth.ai/get-started/installing-+-updating). -- **Install with pip (recommended)** for Linux devices: +### Pip Installation +**Install with pip (recommended) for Linux devices:** ``` pip install unsloth ``` -See below for Windows install instructions: +See [here](https://github.com/unslothai/unsloth/edit/main/README.md#advanced-pip-installation) for advanced pip install instructions. +### Windows Installation +> [!warning] +> Python 3.13 does not support Unsloth. Use 3.12, 3.11 or 3.10 + +1. **Install NVIDIA Video Driver:** + You should install the latest version of your GPUs driver. Download drivers here: [NVIDIA GPU Drive](https://www.nvidia.com/Download/index.aspx). + +3. **Install Visual Studio C++:** + You will need Visual Studio, with C++ installed. By default, C++ is not installed with [Visual Studio](https://visualstudio.microsoft.com/vs/community/), so make sure you select all of the C++ options. Also select options for Windows 10/11 SDK. For more detailed instructions, see [here](https://docs.unsloth.ai/get-started/installing-+-updating). + +5. **Install CUDA Toolkit:** + Follow the instructions to install [CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit-archive). + +6. **Install PyTorch:** + You will need the correct version of PyTorch that is compatibile with your CUDA drivers, so make sure to select them carefully. + [Install PyTorch](https://pytorch.org/get-started/locally/). + +7. **Install Unsloth:** + +```python +pip install "unsloth[windows] @ git+https://github.com/unslothai/unsloth.git" +``` + +#### Notes +To run Unsloth directly on Windows: +- Install Triton from this Windows fork and follow the instructions [here](https://github.com/woct0rdho/triton-windows) (be aware that the Windows fork requires PyTorch >= 2.4 and CUDA 12) +- In the SFTTrainer, set `dataset_num_proc=1` to avoid a crashing issue: +```python +trainer = SFTTrainer( + dataset_num_proc=1, + ... +) +``` + +#### Advanced/Troubleshooting + +For **advanced installation instructions** or if you see weird errors during installations: + +1. Install `torch` and `triton`. Go to https://pytorch.org to install it. For example `pip install torch torchvision torchaudio triton` +2. Confirm if CUDA is installated correctly. Try `nvcc`. If that fails, you need to install `cudatoolkit` or CUDA drivers. +3. Install `xformers` manually. You can try installing `vllm` and seeing if `vllm` succeeds. Check if `xformers` succeeded with `python -m xformers.info` Go to https://github.com/facebookresearch/xformers. Another option is to install `flash-attn` for Ampere GPUs. +4. Double check that your versions of Python, CUDA, CUDNN, `torch`, `triton`, and `xformers` are compatible with one another. The [PyTorch Compatibility Matrix](https://github.com/pytorch/pytorch/blob/main/RELEASE.md#release-compatibility-matrix) may be useful. +5. Finally, install `bitsandbytes` and check it with `python -m bitsandbytes` + ### Conda Installation (Optional) `⚠️Only use Conda if you have it. If not, use Pip`. Select either `pytorch-cuda=11.8,12.1` for CUDA 11.8 or CUDA 12.1. We support `python=3.10,3.11,3.12`. ```bash @@ -111,7 +165,7 @@ pip install unsloth ``` -### Pip Installation +### Advanced Pip Installation `⚠️Do **NOT** use this if you have Conda.` Pip is a bit more complex since there are dependency issues. The pip command is different for `torch 2.2,2.3,2.4,2.5` and CUDA versions. For other torch versions, we support `torch211`, `torch212`, `torch220`, `torch230`, `torch240` and for CUDA versions, we support `cu118` and `cu121` and `cu124`. For Ampere devices (A100, H100, RTX3090) and above, use `cu118-ampere` or `cu121-ampere` or `cu124-ampere`. @@ -168,71 +222,7 @@ x = x.format(cuda.replace(".", ""), "-ampere" if is_ampere else "") print(f'pip install --upgrade pip && pip install "unsloth[{x}] @ git+https://github.com/unslothai/unsloth.git"') ``` -## Windows Installation -> [!warning] -> Python 3.13 does not support Unsloth. Use 3.12, 3.11 or 3.10 -### Step 1: NVIDIA Video Driver - -You should install the latest version of your GPUs driver. You can download drivers here: - - [NVIDIA GPU Drive Download](https://www.nvidia.com/Download/index.aspx) - -### Step 2: Visual Studio C++ -You will need Visual Studio, with C++ installed. By default, C++ is not installed with Visual Studio, so make sure you select all of the C++ options. Also select options for Windows 10/11 SDK. - - [Visual Studio Community Edition](https://visualstudio.microsoft.com/vs/community/) - - - - - -
- VSCode C++ Ref Image - -
-

Steps to configure VS C++

-
-
    -
  1. Launch the Installer downloaded from the link above.
    2. -
  2. In the installer, navigate to Individual components and select all the options mentioned in the image.
    3. -
  3. Click on install now.
    4. -
-
- -### Step 3: CUDA Toolkit - - - [Download CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit-archive) - -### Step 4: Install PyTorch - -You will need the correct version of PyTorch that is compatibile with your CUDA drivers, so make sure to select them carefully - - [Install PyTorch](https://pytorch.org/get-started/locally/) - -### Step 5: Install Unsloth -```python -pip install "unsloth[windows] @ git+https://github.com/unslothai/unsloth.git" -``` - -### Side note -To run Unsloth directly on Windows: -- Install Triton from this Windows fork and follow the instructions: https://github.com/woct0rdho/triton-windows (be aware that the Windows fork requires PyTorch >= 2.4 and CUDA 12) -- In the SFTTrainer, set `dataset_num_proc=1` to avoid a crashing issue: -```python -trainer = SFTTrainer( - dataset_num_proc=1, - ... -) -``` - -### Advanced/Troubleshooting - -For **advanced installation instructions** or if you see weird errors during installations: - -1. Install `torch` and `triton`. Go to https://pytorch.org to install it. For example `pip install torch torchvision torchaudio triton` -2. Confirm if CUDA is installated correctly. Try `nvcc`. If that fails, you need to install `cudatoolkit` or CUDA drivers. -3. Install `xformers` manually. You can try installing `vllm` and seeing if `vllm` succeeds. Check if `xformers` succeeded with `python -m xformers.info` Go to https://github.com/facebookresearch/xformers. Another option is to install `flash-attn` for Ampere GPUs. -4. Double check that your versions of Python, CUDA, CUDNN, `torch`, `triton`, and `xformers` are compatible with one another. The [PyTorch Compatibility Matrix](https://github.com/pytorch/pytorch/blob/main/RELEASE.md#release-compatibility-matrix) may be useful. -5. Finally, install `bitsandbytes` and check it with `python -m bitsandbytes` - -## 📜 [Documentation](https://docs.unsloth.ai) +## 📜 Documentation - Go to our official [Documentation](https://docs.unsloth.ai) for saving to GGUF, checkpointing, evaluation and more! - We support Huggingface's TRL, Trainer, Seq2SeqTrainer or even Pytorch code! - We're in 🤗Hugging Face's official docs! Check out the [SFT docs](https://huggingface.co/docs/trl/main/en/sft_trainer#accelerate-fine-tuning-2x-using-unsloth) and [DPO docs](https://huggingface.co/docs/trl/main/en/dpo_trainer#accelerate-dpo-fine-tuning-using-unsloth)! @@ -437,8 +427,8 @@ You can cite the Unsloth repo as follows: ``` ### Thank You to +- Hugging Face's [TRL library](https://github.com/huggingface/trl) which serves as the basis foundation for Unsloth - [Erik](https://github.com/erikwijmans) for his help adding [Apple's ML Cross Entropy](https://github.com/apple/ml-cross-entropy) in Unsloth - [HuyNguyen-hust](https://github.com/HuyNguyen-hust) for making [RoPE Embeddings 28% faster](https://github.com/unslothai/unsloth/pull/238) - [RandomInternetPreson](https://github.com/RandomInternetPreson) for confirming WSL support - [152334H](https://github.com/152334H) for experimental DPO support -- [atgctg](https://github.com/atgctg) for syntax highlighting diff --git a/unsloth/kernels/utils.py b/unsloth/kernels/utils.py index 5eb9b8f5ce..8da152bcb3 100644 --- a/unsloth/kernels/utils.py +++ b/unsloth/kernels/utils.py @@ -138,7 +138,7 @@ def get_lora_parameters_bias(proj): # if not hasattr(proj, "disable_adapters") or proj.disable_adapters or proj.merged: if getattr(proj, "disable_adapters", True) or proj.merged: - return W, getattr(W, "quant_state", None), None, None, None, bias + return W, getattr(W, "quant_state", None), None, None, None, base_layer.bias pass adapter = getattr(proj, "active_adapters", None) diff --git a/unsloth/models/rl_replacements.py b/unsloth/models/rl_replacements.py index fe7f4accee..5ea61cb9b3 100644 --- a/unsloth/models/rl_replacements.py +++ b/unsloth/models/rl_replacements.py @@ -90,9 +90,11 @@ def sft_trainer_prepare_dataset(function_name, function): "if getattr(tokenizer, 'bos_token', None) is not None else False\n"\ "if 'add_special_tokens' not in locals() and has_bos_token_already:\n"\ " from functools import partial\n"\ - " tokenizer = partial(tokenizer, add_special_tokens = False)\n"\ + " tokenizer_call = tokenizer.__call__\n"\ + " tokenizer.__call__ = partial(tokenizer_call, add_special_tokens = False)\n"\ " processing_class = tokenizer\n"\ "else:\n"\ + " tokenizer_call = None\n"\ " add_special_tokens = False if has_bos_token_already else locals().get('add_special_tokens', False)\n" check_text = check_text.split("\n") @@ -109,6 +111,14 @@ def sft_trainer_prepare_dataset(function_name, function): replacer = replacer[0] function = function.replace(replacer, replacer + check_text) pass + + # Return tokenizer's original state + return_state = "if tokenizer_call is not None: tokenizer.__call__ = tokenizer_call\n" + function = re.sub( + r"\n([ ]{4,})(return .*?[\s]{0,})$", + rf"\1{return_state}\1\2", + function, + ) return function pass RL_FUNCTIONS["sft_trainer"].append(sft_trainer_prepare_dataset) diff --git a/unsloth/tokenizer_utils.py b/unsloth/tokenizer_utils.py index 9c5f825a0c..91bb0202ff 100644 --- a/unsloth/tokenizer_utils.py +++ b/unsloth/tokenizer_utils.py @@ -859,6 +859,7 @@ pass import inspect from inspect import getsource +import trl import trl.trainer.sft_trainer from trl.trainer.sft_trainer import * from transformers.trainer import *