Merge branch 'main' into nightly
This commit is contained in:
Daniel Han
commit
1dafd6af8c
4 changed files with 74 additions and 73 deletions
132
README.md
132
README.md
|
|
@ -39,6 +39,14 @@ All notebooks are **beginner friendly**! Add your dataset, click "Run All", and
|
|||
- This [continued pretraining notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-CPT.ipynb) is for learning another language
|
||||
- Click [here](https://docs.unsloth.ai/) for detailed documentation for Unsloth.
|
||||
|
||||
## ⚡ Quickstart
|
||||
|
||||
- **Install with pip (recommended)** for Linux devices:
|
||||
```
|
||||
pip install unsloth
|
||||
```
|
||||
For Windows install instructions, see [here](https://github.com/unslothai/unsloth/edit/main/README.md#windows-installation).
|
||||
|
||||
## 🦥 Unsloth.ai News
|
||||
- 📣 NEW! Introducing Long-context [Reasoning (GRPO)](https://unsloth.ai/blog/grpo) in Unsloth. You can now reproduce DeepSeek-R1's "aha" moment with just 5GB VRAM. Transform Llama, Phi, Mistral etc. into reasoning LLMs!
|
||||
- 📣 NEW! [DeepSeek-R1](https://unsloth.ai/blog/deepseek-r1) - the most powerful open reasoning models with Llama & Qwen distillations. Run or fine-tune them now! More details: [unsloth.ai/blog/deepseek-r1](https://unsloth.ai/blog/deepseek-r1). All model uploads: [here](https://huggingface.co/collections/unsloth/deepseek-r1-all-versions-678e1c48f5d2fce87892ace5).
|
||||
|
|
@ -65,7 +73,7 @@ All notebooks are **beginner friendly**! Add your dataset, click "Run All", and
|
|||
| ------------------------------- | --------------------------------------- |
|
||||
| 📚 **Documentation & Wiki** | [Read Our Docs](https://docs.unsloth.ai) |
|
||||
| <img height="14" src="https://upload.wikimedia.org/wikipedia/commons/6/6f/Logo_of_Twitter.svg" /> **Twitter (aka X)** | [Follow us on X](https://twitter.com/unslothai)|
|
||||
| 💾 **Installation** | [Pip install](https://github.com/unslothai/unsloth/edit/main/README.md#-install-unsloth)|
|
||||
| 💾 **Installation** | [Pip install](https://docs.unsloth.ai/get-started/installing-+-updating)|
|
||||
| 🔮 **Our Models** | [Unsloth Releases](https://docs.unsloth.ai/get-started/all-our-models)|
|
||||
| ✍️ **Blog** | [Read our Blogs](https://unsloth.ai/blog)|
|
||||
| <img height="14" src="https://redditinc.com/hs-fs/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png" /> **Reddit** | [Join our Reddit page](https://reddit.com/r/unsloth)|
|
||||
|
|
@ -74,17 +82,63 @@ All notebooks are **beginner friendly**! Add your dataset, click "Run All", and
|
|||
- All kernels written in [OpenAI's Triton](https://openai.com/index/triton/) language. **Manual backprop engine**.
|
||||
- **0% loss in accuracy** - no approximation methods - all exact.
|
||||
- No change of hardware. Supports NVIDIA GPUs since 2018+. Minimum CUDA Capability 7.0 (V100, T4, Titan V, RTX 20, 30, 40x, A100, H100, L40 etc) [Check your GPU!](https://developer.nvidia.com/cuda-gpus) GTX 1070, 1080 works, but is slow.
|
||||
- Works on **Linux** and **Windows** via WSL.
|
||||
- Works on **Linux** and **Windows**
|
||||
- Supports 4bit and 16bit QLoRA / LoRA finetuning via [bitsandbytes](https://github.com/TimDettmers/bitsandbytes).
|
||||
- If you trained a model with 🦥Unsloth, you can use this cool sticker! <img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/made with unsloth.png" height="50" align="center" />
|
||||
|
||||
## 💾 Install Unsloth
|
||||
You can also see our documentation for more detailed installation and updating instructions [here](https://docs.unsloth.ai/get-started/installing-+-updating).
|
||||
|
||||
- **Install with pip (recommended)** for Linux devices:
|
||||
### Pip Installation
|
||||
**Install with pip (recommended) for Linux devices:**
|
||||
```
|
||||
pip install unsloth
|
||||
```
|
||||
See below for Windows install instructions:
|
||||
See [here](https://github.com/unslothai/unsloth/edit/main/README.md#advanced-pip-installation) for advanced pip install instructions.
|
||||
### Windows Installation
|
||||
> [!warning]
|
||||
> Python 3.13 does not support Unsloth. Use 3.12, 3.11 or 3.10
|
||||
|
||||
1. **Install NVIDIA Video Driver:**
|
||||
You should install the latest version of your GPUs driver. Download drivers here: [NVIDIA GPU Drive](https://www.nvidia.com/Download/index.aspx).
|
||||
|
||||
3. **Install Visual Studio C++:**
|
||||
You will need Visual Studio, with C++ installed. By default, C++ is not installed with [Visual Studio](https://visualstudio.microsoft.com/vs/community/), so make sure you select all of the C++ options. Also select options for Windows 10/11 SDK. For more detailed instructions, see [here](https://docs.unsloth.ai/get-started/installing-+-updating).
|
||||
|
||||
5. **Install CUDA Toolkit:**
|
||||
Follow the instructions to install [CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit-archive).
|
||||
|
||||
6. **Install PyTorch:**
|
||||
You will need the correct version of PyTorch that is compatibile with your CUDA drivers, so make sure to select them carefully.
|
||||
[Install PyTorch](https://pytorch.org/get-started/locally/).
|
||||
|
||||
7. **Install Unsloth:**
|
||||
|
||||
```python
|
||||
pip install "unsloth[windows] @ git+https://github.com/unslothai/unsloth.git"
|
||||
```
|
||||
|
||||
#### Notes
|
||||
To run Unsloth directly on Windows:
|
||||
- Install Triton from this Windows fork and follow the instructions [here](https://github.com/woct0rdho/triton-windows) (be aware that the Windows fork requires PyTorch >= 2.4 and CUDA 12)
|
||||
- In the SFTTrainer, set `dataset_num_proc=1` to avoid a crashing issue:
|
||||
```python
|
||||
trainer = SFTTrainer(
|
||||
dataset_num_proc=1,
|
||||
...
|
||||
)
|
||||
```
|
||||
|
||||
#### Advanced/Troubleshooting
|
||||
|
||||
For **advanced installation instructions** or if you see weird errors during installations:
|
||||
|
||||
1. Install `torch` and `triton`. Go to https://pytorch.org to install it. For example `pip install torch torchvision torchaudio triton`
|
||||
2. Confirm if CUDA is installated correctly. Try `nvcc`. If that fails, you need to install `cudatoolkit` or CUDA drivers.
|
||||
3. Install `xformers` manually. You can try installing `vllm` and seeing if `vllm` succeeds. Check if `xformers` succeeded with `python -m xformers.info` Go to https://github.com/facebookresearch/xformers. Another option is to install `flash-attn` for Ampere GPUs.
|
||||
4. Double check that your versions of Python, CUDA, CUDNN, `torch`, `triton`, and `xformers` are compatible with one another. The [PyTorch Compatibility Matrix](https://github.com/pytorch/pytorch/blob/main/RELEASE.md#release-compatibility-matrix) may be useful.
|
||||
5. Finally, install `bitsandbytes` and check it with `python -m bitsandbytes`
|
||||
|
||||
### Conda Installation (Optional)
|
||||
`⚠️Only use Conda if you have it. If not, use Pip`. Select either `pytorch-cuda=11.8,12.1` for CUDA 11.8 or CUDA 12.1. We support `python=3.10,3.11,3.12`.
|
||||
```bash
|
||||
|
|
@ -111,7 +165,7 @@ pip install unsloth
|
|||
```
|
||||
</details>
|
||||
|
||||
### Pip Installation
|
||||
### Advanced Pip Installation
|
||||
`⚠️Do **NOT** use this if you have Conda.` Pip is a bit more complex since there are dependency issues. The pip command is different for `torch 2.2,2.3,2.4,2.5` and CUDA versions.
|
||||
|
||||
For other torch versions, we support `torch211`, `torch212`, `torch220`, `torch230`, `torch240` and for CUDA versions, we support `cu118` and `cu121` and `cu124`. For Ampere devices (A100, H100, RTX3090) and above, use `cu118-ampere` or `cu121-ampere` or `cu124-ampere`.
|
||||
|
|
@ -168,71 +222,7 @@ x = x.format(cuda.replace(".", ""), "-ampere" if is_ampere else "")
|
|||
print(f'pip install --upgrade pip && pip install "unsloth[{x}] @ git+https://github.com/unslothai/unsloth.git"')
|
||||
```
|
||||
|
||||
## Windows Installation
|
||||
> [!warning]
|
||||
> Python 3.13 does not support Unsloth. Use 3.12, 3.11 or 3.10
|
||||
### Step 1: NVIDIA Video Driver
|
||||
|
||||
You should install the latest version of your GPUs driver. You can download drivers here:
|
||||
- [NVIDIA GPU Drive Download](https://www.nvidia.com/Download/index.aspx)
|
||||
|
||||
### Step 2: Visual Studio C++
|
||||
You will need Visual Studio, with C++ installed. By default, C++ is not installed with Visual Studio, so make sure you select all of the C++ options. Also select options for Windows 10/11 SDK.
|
||||
- [Visual Studio Community Edition](https://visualstudio.microsoft.com/vs/community/)
|
||||
<table>
|
||||
<tr>
|
||||
<td>
|
||||
<img src="https://github.com/user-attachments/assets/d3e6ca95-85bb-442a-8c6f-81944300598e" alt="VSCode C++ Ref Image" width="400" height="350"/>
|
||||
</td>
|
||||
<td>
|
||||
<div align="center">
|
||||
<h1>Steps to configure VS C++</h1>
|
||||
</div>
|
||||
<ol>
|
||||
<li>Launch the Installer downloaded from the link above.</li>
|
||||
<li>In the installer, navigate to Individual components and select all the options mentioned in the image.</li>
|
||||
<li>Click on install now.</li>
|
||||
</ol>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
### Step 3: CUDA Toolkit
|
||||
|
||||
- [Download CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit-archive)
|
||||
|
||||
### Step 4: Install PyTorch
|
||||
|
||||
You will need the correct version of PyTorch that is compatibile with your CUDA drivers, so make sure to select them carefully
|
||||
- [Install PyTorch](https://pytorch.org/get-started/locally/)
|
||||
|
||||
### Step 5: Install Unsloth
|
||||
```python
|
||||
pip install "unsloth[windows] @ git+https://github.com/unslothai/unsloth.git"
|
||||
```
|
||||
|
||||
### Side note
|
||||
To run Unsloth directly on Windows:
|
||||
- Install Triton from this Windows fork and follow the instructions: https://github.com/woct0rdho/triton-windows (be aware that the Windows fork requires PyTorch >= 2.4 and CUDA 12)
|
||||
- In the SFTTrainer, set `dataset_num_proc=1` to avoid a crashing issue:
|
||||
```python
|
||||
trainer = SFTTrainer(
|
||||
dataset_num_proc=1,
|
||||
...
|
||||
)
|
||||
```
|
||||
|
||||
### Advanced/Troubleshooting
|
||||
|
||||
For **advanced installation instructions** or if you see weird errors during installations:
|
||||
|
||||
1. Install `torch` and `triton`. Go to https://pytorch.org to install it. For example `pip install torch torchvision torchaudio triton`
|
||||
2. Confirm if CUDA is installated correctly. Try `nvcc`. If that fails, you need to install `cudatoolkit` or CUDA drivers.
|
||||
3. Install `xformers` manually. You can try installing `vllm` and seeing if `vllm` succeeds. Check if `xformers` succeeded with `python -m xformers.info` Go to https://github.com/facebookresearch/xformers. Another option is to install `flash-attn` for Ampere GPUs.
|
||||
4. Double check that your versions of Python, CUDA, CUDNN, `torch`, `triton`, and `xformers` are compatible with one another. The [PyTorch Compatibility Matrix](https://github.com/pytorch/pytorch/blob/main/RELEASE.md#release-compatibility-matrix) may be useful.
|
||||
5. Finally, install `bitsandbytes` and check it with `python -m bitsandbytes`
|
||||
|
||||
## 📜 [Documentation](https://docs.unsloth.ai)
|
||||
## 📜 Documentation
|
||||
- Go to our official [Documentation](https://docs.unsloth.ai) for saving to GGUF, checkpointing, evaluation and more!
|
||||
- We support Huggingface's TRL, Trainer, Seq2SeqTrainer or even Pytorch code!
|
||||
- We're in 🤗Hugging Face's official docs! Check out the [SFT docs](https://huggingface.co/docs/trl/main/en/sft_trainer#accelerate-fine-tuning-2x-using-unsloth) and [DPO docs](https://huggingface.co/docs/trl/main/en/dpo_trainer#accelerate-dpo-fine-tuning-using-unsloth)!
|
||||
|
|
@ -437,8 +427,8 @@ You can cite the Unsloth repo as follows:
|
|||
```
|
||||
|
||||
### Thank You to
|
||||
- Hugging Face's [TRL library](https://github.com/huggingface/trl) which serves as the basis foundation for Unsloth
|
||||
- [Erik](https://github.com/erikwijmans) for his help adding [Apple's ML Cross Entropy](https://github.com/apple/ml-cross-entropy) in Unsloth
|
||||
- [HuyNguyen-hust](https://github.com/HuyNguyen-hust) for making [RoPE Embeddings 28% faster](https://github.com/unslothai/unsloth/pull/238)
|
||||
- [RandomInternetPreson](https://github.com/RandomInternetPreson) for confirming WSL support
|
||||
- [152334H](https://github.com/152334H) for experimental DPO support
|
||||
- [atgctg](https://github.com/atgctg) for syntax highlighting
|
||||
|
|
|
|||
|
|
@ -138,7 +138,7 @@ def get_lora_parameters_bias(proj):
|
|||
|
||||
# if not hasattr(proj, "disable_adapters") or proj.disable_adapters or proj.merged:
|
||||
if getattr(proj, "disable_adapters", True) or proj.merged:
|
||||
return W, getattr(W, "quant_state", None), None, None, None, bias
|
||||
return W, getattr(W, "quant_state", None), None, None, None, base_layer.bias
|
||||
pass
|
||||
|
||||
adapter = getattr(proj, "active_adapters", None)
|
||||
|
|
|
|||
|
|
@ -90,9 +90,11 @@ def sft_trainer_prepare_dataset(function_name, function):
|
|||
"if getattr(tokenizer, 'bos_token', None) is not None else False\n"\
|
||||
"if 'add_special_tokens' not in locals() and has_bos_token_already:\n"\
|
||||
" from functools import partial\n"\
|
||||
" tokenizer = partial(tokenizer, add_special_tokens = False)\n"\
|
||||
" tokenizer_call = tokenizer.__call__\n"\
|
||||
" tokenizer.__call__ = partial(tokenizer_call, add_special_tokens = False)\n"\
|
||||
" processing_class = tokenizer\n"\
|
||||
"else:\n"\
|
||||
" tokenizer_call = None\n"\
|
||||
" add_special_tokens = False if has_bos_token_already else locals().get('add_special_tokens', False)\n"
|
||||
|
||||
check_text = check_text.split("\n")
|
||||
|
|
@ -109,6 +111,14 @@ def sft_trainer_prepare_dataset(function_name, function):
|
|||
replacer = replacer[0]
|
||||
function = function.replace(replacer, replacer + check_text)
|
||||
pass
|
||||
|
||||
# Return tokenizer's original state
|
||||
return_state = "if tokenizer_call is not None: tokenizer.__call__ = tokenizer_call\n"
|
||||
function = re.sub(
|
||||
r"\n([ ]{4,})(return .*?[\s]{0,})$",
|
||||
rf"\1{return_state}\1\2",
|
||||
function,
|
||||
)
|
||||
return function
|
||||
pass
|
||||
RL_FUNCTIONS["sft_trainer"].append(sft_trainer_prepare_dataset)
|
||||
|
|
|
|||
|
|
@ -859,6 +859,7 @@ pass
|
|||
|
||||
import inspect
|
||||
from inspect import getsource
|
||||
import trl
|
||||
import trl.trainer.sft_trainer
|
||||
from trl.trainer.sft_trainer import *
|
||||
from transformers.trainer import *
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue