diff --git a/README.md b/README.md index cf1568f124..69c8ddc67b 100644 --- a/README.md +++ b/README.md @@ -1,26 +1,27 @@
- +
## 2-5x faster 60% less memory local QLoRA finetuning | Llama 7b | Mistral 7b | CodeLlama 34b | Llama 7b Kaggle 2x T4 | |-----------------------------|-----------------------------|-------------------------|------------------------| | **2.2x faster, -43% VRAM** | **2.2x faster, -62% VRAM** | **1.9x faster, -27% VRAM** | **5.5x faster, -44% VRAM** | -| [Colab Alpaca example + inference, saving](https://colab.research.google.com/drive/1lBzz5KeZJKXjvivbYvmGarix9Ao6Wxe5?usp=sharing) | [Colab T4 example + inference, saving](https://colab.research.google.com/drive/1Dyauq4kTZoLewQ1cApceUQVNcnnNTzg_?usp=sharing) | [A100 example](https://colab.research.google.com/drive/1gdHyAx8XJsz2yNV-DHvbHjR1iCef5Qmh?usp=sharing) | [Kaggle Alpaca example](https://www.kaggle.com/danielhanchen/unsloth-alpaca-t4-ddp) | -| [Colab A100 example](https://colab.research.google.com/drive/1YIPY_18xm-K0iJDgvNkRoJsgkPMPAO3G?usp=sharing) | [Colab A100 example](https://colab.research.google.com/drive/1SKrKGV-BZoU4kv5q3g0jtE_OhRgPtrrQ?usp=sharing) | (59 more examples if you scroll down) | [Kaggle Slim Orca](https://www.kaggle.com/danielhanchen/unsloth-slimorca-t4-ddp) | +| [Free Colab Alpaca dataset example](https://colab.research.google.com/drive/1lBzz5KeZJKXjvivbYvmGarix9Ao6Wxe5?usp=sharing) | [Free Colab Alpaca dataset example](https://colab.research.google.com/drive/1Dyauq4kTZoLewQ1cApceUQVNcnnNTzg_?usp=sharing) | [Colab A100 example](https://colab.research.google.com/drive/1y7A0AxE3y8gdj4AVkl2aZX47Xu3P1wJT?usp=sharing) | [Kaggle Alpaca example](https://www.kaggle.com/danielhanchen/unsloth-alpaca-t4-ddp) | +| [Colab A100 example](https://colab.research.google.com/drive/1YIPY_18xm-K0iJDgvNkRoJsgkPMPAO3G?usp=sharing) | [Colab A100 example](https://colab.research.google.com/drive/1SKrKGV-BZoU4kv5q3g0jtE_OhRgPtrrQ?usp=sharing) | (59 more examples if you scroll down) | [Kaggle Slim Orca example](https://www.kaggle.com/danielhanchen/unsloth-slimorca-t4-ddp) | -* Supports Llama (7, 13, 70b), Yi (6, 34b), Mistral (7b), Tinyllama, CodeLlama (7, 13, 34b), and all Llama / Mistral derived architectures! -* All kernels written in [OpenAI's Triton](https://openai.com/research/triton) language. +* Supports Llama, Yi, Mistral, CodeLlama, and their derived models (Open Hermes etc). +* All kernels written in [OpenAI's Triton](https://openai.com/research/triton) language. **Manual backpropagation engine**. * **0% loss in accuracy** - no approximation methods - all exact. * No change of hardware necessary. Supports NVIDIA GPUs since 2018+. Minimum CUDA Compute Capability 7.0 (V100, T4, Titan V, RTX 20, 30, 40x, A100, H100, L40 etc) [Check your GPU](https://developer.nvidia.com/cuda-gpus) * **NEW!** Works on **Linux** and **Windows** via WSL. -* **NEW!** Experimental support for [DPO (Direct Preference Optimization)](https://arxiv.org/abs/2305.18290)! +* **NEW!** Support for [DPO (Direct Preference Optimization)](https://arxiv.org/abs/2305.18290), PPO and Reward Modelling via [TRL](https://huggingface.co/docs/trl/dpo_trainer). +* **NEW!** Download 4 bit models 4x faster directly from Huggingface! * Supports 4bit and 16bit QLoRA / LoRA finetuning via [bitsandbytes](https://github.com/TimDettmers/bitsandbytes). -* Open source version trains 5x faster or you can check out [Unsloth Pro and Max](https://unsloth.ai/) codepaths for **30x faster training**! +* Open source version trains 5x faster - check out [Unsloth Max](https://unsloth.ai/) for **30x faster training**! -| 1 A100 40GB | Hugging Face | Flash Attention 2 | Unsloth Open | Unsloth Equal | Unsloth Pro | Unsloth Max | +| 1 A100 40GB | Huggingface | Flash Attention | Unsloth Open | Unsloth Equal | Unsloth Pro | Unsloth Max | |--------------|-------------|-------------|-----------------|--------------|---------------|-------------| | Alpaca | 1x | 1.04x | 1.98x | 2.48x | 5.32x | **15.64x** | | LAION Chip2 | 1x | 0.92x | 1.61x | 1.84x | 7.05x | **20.73x** | @@ -28,11 +29,11 @@ | Slim Orca | 1x | 1.18x | 2.22x | 2.64x | 5.04x | **14.82x** | Join our [Discord](https://discord.gg/nsS4V5Z6ge)! -If you trained a model with Unsloth, we made a cool sticker!! +If you trained a model with Unsloth, we made a cool sticker if you want to use it! # Installation Instructions - Conda -Unsloth currently only supports Linux distros and Pytorch == 2.1. +Select either `pytorch-cuda=11.8` for CUDA 11.8 or `pytorch-cuda=12.1` for CUDA 12.1. ```bash conda install cudatoolkit xformers bitsandbytes pytorch pytorch-cuda=12.1 \ -c pytorch -c nvidia -c xformers -c conda-forge -y @@ -40,25 +41,36 @@ pip install "unsloth[conda] @ git+https://github.com/unslothai/unsloth.git" ``` # Installation Instructions - Pip +Do **NOT** use this if you have Anaconda. You must use the Conda install method, or else stuff will BREAK. + 1. Find your CUDA version via ```python import torch; torch.version.cuda ``` -2. We only support Pytorch 2.1 (2.1.1 bugs out for now): You can update Pytorch via Pip (interchange cu121 / cu118) +2. For Pytorch 2.1.0: You can update Pytorch via Pip (interchange `cu121` / `cu118`). Go to https://pytorch.org/ to learn more. Select either `cu118` for CUDA 11.8 or `cu121` for CUDA 12.1. If you have a RTX 3060 or higher (A100, H100 etc), use the `"ampere"` path. ```bash pip install --upgrade --force-reinstall --no-cache-dir torch==2.1.0 triton \ --index-url https://download.pytorch.org/whl/cu121 ``` -2. Select either cu118 for CUDA 11.8 or cu121 for CUDA 12.1. If you have a RTX 3060 or higher (A100, H100 etc), use the "ampere" path. ```bash pip install "unsloth[cu118] @ git+https://github.com/unslothai/unsloth.git" pip install "unsloth[cu121] @ git+https://github.com/unslothai/unsloth.git" pip install "unsloth[cu118_ampere] @ git+https://github.com/unslothai/unsloth.git" pip install "unsloth[cu121_ampere] @ git+https://github.com/unslothai/unsloth.git" ``` -Change `cu121` to `cu118` for CUDA version 11.8 or 12.1. Go to https://pytorch.org/ to learn more. - -4. If you get errors, try the below first, then go back to step 1: +3. For Pytorch 2.1.1: Use the `"ampere"` path for newer RTX 30xx GPUs or higher. +```bash +pip install --upgrade --force-reinstall --no-cache-dir torch==2.1.1 triton \ + --index-url https://download.pytorch.org/whl/cu121 +``` +```bash +pip install "unsloth[cu118_torch211] @ git+https://github.com/unslothai/unsloth.git" +pip install "unsloth[cu121_torch211] @ git+https://github.com/unslothai/unsloth.git" +pip install "unsloth[cu118_ampere_torch211] @ git+https://github.com/unslothai/unsloth.git" +pip install "unsloth[cu121_ampere_torch211] @ git+https://github.com/unslothai/unsloth.git" +``` +4. We're working on Pytorch 2.1.2 support. +5. If you get errors, try the below first, then go back to step 1: ```bash pip install --upgrade pip ``` @@ -120,10 +132,8 @@ trainer = SFTTrainer( trainer.train() ``` -# DPO (Direct Preference Optimization) Experimental support -[152334H](https://github.com/152334H) hacked Unsloth to work with DPO via TRL! -1. Hack the model's `config.json` to be llama model. [Example gist](https://gist.github.com/152334H/d8a68b51b83bac008a02e69ecc81d5c1). -2. Use Unsloth for DPO for both base and reference models. [Example gist](https://gist.github.com/152334H/4847f3a8cca12894877e6b30698b0b64). +# DPO (Direct Preference Optimization) Support +DPO, PPO, Reward Modelling all seem to work as per 3rd party independent testing from [Llama-Factory](https://github.com/hiyouga/LLaMA-Factory). # Future Milestones and limitations 1. Support Mixtral. @@ -173,6 +183,40 @@ Two Tesla T4s on Kaggle * Slim Orca `bsz=1` for all benchmarks since `bsz=2` OOMs. We can handle `bsz=2`, but we benchmark it with `bsz=1` for consistency. +# Llama-Factory 3rd party benchmarking + +| Method | Bits | TGS | GRAM | Speed | +| --- | --- | --- | --- | --- | +| HF | 16 | 2392 | 18GB | 100% | +| HF+FA2 | 16 | 2954 | 17GB | 123% | +| Unsloth+FA2 | 16 | 4007 | 16GB | **168%** | +| HF | 4 | 2415 | 9GB | 101% | +| Unsloth+FA2 | 4 | 3726 | 7GB | **160%** | + +[Link](https://github.com/hiyouga/LLaMA-Factory/wiki/Performance-Comparison) to performance table. TGS: tokens per GPU per second. Model: LLaMA2-7B. GPU: NVIDIA A100 * 1. Batch size: 4. Gradient accumulation: 2. LoRA rank: 8. Max length: 1024. + +# How did we make it faster? +Manual autograd, Triton kernels etc. See our [Benchmark Breakdown](https://unsloth.ai/blog/mistral-benchmark) for more info! + +$$ +\begin{align} +y &= \frac{x_i}{\sqrt{\frac{1}{n}\sum{x_i^2}+\epsilon}} \cdot w \\ +r &= \frac{1}{\sqrt{\frac{1}{n}\sum{x_i^2}+\epsilon}} \\ +\frac{dC}{dX} &= \frac{1}{n} r \bigg( n (dY \cdot w) - \bigg( x_i \cdot r \cdot \sum{dY \cdot y_i } \bigg) \bigg) +\end{align} +$$ + + +# Troubleshooting +1. Sometimes `bitsandbytes` or `xformers` does not link properly. Try running: +```bash +!ldconfig /usr/lib64-nvidia +``` +2. Windows is not supported as of yet - we rely on Xformers and Triton support, so until both packages support Windows officially, Unsloth will then support Windows. + +3. If it doesn't install - maybe try updating `pip`. + + # Full benchmarking tables Click "Code" for a fully reproducible example. "Unsloth Equal" is a preview of our PRO version, with code stripped out. All settings and the loss curve remains identical. @@ -312,27 +356,6 @@ Click "Code" for a fully reproducible example. | memory MB| OOM | OOM | 7594 | 8881 | | | | % saved | OOM | OOM | | | | | -# How did we make it faster? -Manual autograd, Triton kernels etc. See our [Benchmark Breakdown](https://unsloth.ai/blog/mistral-benchmark) for more info! - -$$ -\begin{align} -y &= \frac{x_i}{\sqrt{\frac{1}{n}\sum{x_i^2}+\epsilon}} \cdot w \\ -r &= \frac{1}{\sqrt{\frac{1}{n}\sum{x_i^2}+\epsilon}} \\ -\frac{dC}{dX} &= \frac{1}{n} r \bigg( n (dY \cdot w) - \bigg( x_i \cdot r \cdot \sum{dY \cdot y_i } \bigg) \bigg) -\end{align} -$$ - - -# Troubleshooting -1. Sometimes `bitsandbytes` or `xformers` does not link properly. Try running: -```bash -!ldconfig /usr/lib64-nvidia -``` -2. Windows is not supported as of yet - we rely on Xformers and Triton support, so until both packages support Windows officially, Unsloth will then support Windows. - -3. If it doesn't install - maybe try updating `pip`. - # Credits 1. [RandomInternetPreson](https://github.com/RandomInternetPreson) for confirming WSL support 2. [152334H](https://github.com/152334H) for experimental DPO support diff --git a/pyproject.toml b/pyproject.toml index ab710de542..2bceca566f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,21 +5,21 @@ build-backend = "setuptools.build_meta" [project] name = "unsloth" dynamic = ["version"] -description = "2X faster LLM finetuning" +description = "2-5X faster LLM finetuning" readme = "README.md" requires-python = ">=3.9" license = {file = "LICENSE"} keywords = ["ai", "llm",] authors = [ - {email = "info@unsloth.ai"}, - {name = "Unsloth AI team"}, + {email = "info@unsloth.ai"}, + {name = "Unsloth AI team"}, ] maintainers = [ - {name = "Daniel Han", email = "danielhanchen@gmail.com"}, - {name = "Michael Han", email = "info@unsloth.ai"}, + {name = "Daniel Han", email = "danielhanchen@gmail.com"}, + {name = "Michael Han", email = "info@unsloth.ai"}, ] classifiers = [ - "Programming Language :: Python", + "Programming Language :: Python", ] [tool.setuptools.dynamic] @@ -40,54 +40,84 @@ huggingface = [ "trl", "peft", "packaging", + "ninja", ] cu118only = [ - "xformers @ https://download.pytorch.org/whl/cu118/xformers-0.0.22.post7%2Bcu118-cp39-cp39-manylinux2014_x86_64.whl ; python_version=='3.9'", - "xformers @ https://download.pytorch.org/whl/cu118/xformers-0.0.22.post7%2Bcu118-cp310-cp310-manylinux2014_x86_64.whl ; python_version=='3.10'", - "xformers @ https://download.pytorch.org/whl/cu118/xformers-0.0.22.post7%2Bcu118-cp311-cp311-manylinux2014_x86_64.whl ; python_version=='3.11'", + "xformers @ https://download.pytorch.org/whl/cu118/xformers-0.0.22.post7%2Bcu118-cp39-cp39-manylinux2014_x86_64.whl ; python_version=='3.9'", + "xformers @ https://download.pytorch.org/whl/cu118/xformers-0.0.22.post7%2Bcu118-cp310-cp310-manylinux2014_x86_64.whl ; python_version=='3.10'", + "xformers @ https://download.pytorch.org/whl/cu118/xformers-0.0.22.post7%2Bcu118-cp311-cp311-manylinux2014_x86_64.whl ; python_version=='3.11'", ] cu121only = [ - "xformers @ https://download.pytorch.org/whl/cu121/xformers-0.0.22.post7-cp39-cp39-manylinux2014_x86_64.whl ; python_version=='3.9'", - "xformers @ https://download.pytorch.org/whl/cu121/xformers-0.0.22.post7-cp310-cp310-manylinux2014_x86_64.whl ; python_version=='3.10'", - "xformers @ https://download.pytorch.org/whl/cu121/xformers-0.0.22.post7-cp311-cp311-manylinux2014_x86_64.whl ; python_version=='3.11'", + "xformers @ https://download.pytorch.org/whl/cu121/xformers-0.0.22.post7-cp39-cp39-manylinux2014_x86_64.whl ; python_version=='3.9'", + "xformers @ https://download.pytorch.org/whl/cu121/xformers-0.0.22.post7-cp310-cp310-manylinux2014_x86_64.whl ; python_version=='3.10'", + "xformers @ https://download.pytorch.org/whl/cu121/xformers-0.0.22.post7-cp311-cp311-manylinux2014_x86_64.whl ; python_version=='3.11'", +] +cu118only_torch211 = [ + "xformers @ https://download.pytorch.org/whl/cu118/xformers-0.0.23%2Bcu118-cp39-cp39-manylinux2014_x86_64.whl ; python_version=='3.9'", + "xformers @ https://download.pytorch.org/whl/cu118/xformers-0.0.23%2Bcu118-cp310-cp310-manylinux2014_x86_64.whl ; python_version=='3.10'", + "xformers @ https://download.pytorch.org/whl/cu118/xformers-0.0.23%2Bcu118-cp311-cp311-manylinux2014_x86_64.whl ; python_version=='3.11'", +] +cu121only_torch211 = [ + "xformers @ https://download.pytorch.org/whl/cu121/xformers-0.0.23-cp39-cp39-manylinux2014_x86_64.whl ; python_version=='3.9'", + "xformers @ https://download.pytorch.org/whl/cu121/xformers-0.0.23-cp310-cp310-manylinux2014_x86_64.whl ; python_version=='3.10'", + "xformers @ https://download.pytorch.org/whl/cu121/xformers-0.0.23-cp311-cp311-manylinux2014_x86_64.whl ; python_version=='3.11'", ] cu118 = [ - "unsloth[huggingface]", - "bitsandbytes", - "unsloth[cu118only]", + "unsloth[huggingface]", + "bitsandbytes", + "unsloth[cu118only]", ] cu121 = [ - "unsloth[huggingface]", - "bitsandbytes", - "unsloth[cu121only]", + "unsloth[huggingface]", + "bitsandbytes", + "unsloth[cu121only]", +] +cu118_torch211 = [ + "unsloth[huggingface]", + "bitsandbytes", + "unsloth[cu118only_torch211]", +] +cu121_torch211 = [ + "unsloth[huggingface]", + "bitsandbytes", + "unsloth[cu121only_torch211]", ] kaggle = [ - "unsloth[huggingface]", + "unsloth[huggingface]", ] conda = [ - "unsloth[huggingface]", + "unsloth[huggingface]", ] colab = [ - "unsloth[cu121]", -] -cu118_ampere = [ - "unsloth[huggingface]", - "bitsandbytes", - "unsloth[cu118only]", - "ninja", - "flash-attn", -] -cu121_ampere = [ - "unsloth[huggingface]", - "bitsandbytes", - "unsloth[cu121only]", - "ninja", - "flash-attn", + "unsloth[cu121]", ] colab_ampere = [ - "unsloth[cu121]", - "ninja", - "flash-attn", + "unsloth[cu121]", + "flash-attn", +] +cu118_ampere = [ + "unsloth[huggingface]", + "bitsandbytes", + "unsloth[cu118only]", + "flash-attn", +] +cu121_ampere = [ + "unsloth[huggingface]", + "bitsandbytes", + "unsloth[cu121only]", + "flash-attn", +] +cu118_ampere_torch211 = [ + "unsloth[huggingface]", + "bitsandbytes", + "unsloth[cu118only_torch211]", + "flash-attn", +] +cu121_ampere_torch211 = [ + "unsloth[huggingface]", + "bitsandbytes", + "unsloth[cu121only_torch211]", + "flash-attn", ] [project.urls] diff --git a/unsloth/__init__.py b/unsloth/__init__.py index 5b0b8d325d..28c7affe94 100644 --- a/unsloth/__init__.py +++ b/unsloth/__init__.py @@ -11,7 +11,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -__version__ = "2023.12" +__version__ = "2024.1" import os import warnings import importlib diff --git a/unsloth/models/llama.py b/unsloth/models/llama.py index cbbb6b7c9e..f6cc078356 100644 --- a/unsloth/models/llama.py +++ b/unsloth/models/llama.py @@ -369,7 +369,8 @@ def LlamaModel_fast_forward( raise ValueError("Unsloth: You have to specify either decoder_input_ids or decoder_inputs_embeds") seq_length_with_past = seq_length - assert(seq_length <= self.max_seq_length) + if hasattr(self, "max_seq_length"): + assert(seq_length <= self.max_seq_length) past_key_values_length = 0 if past_key_values is not None: @@ -690,7 +691,14 @@ class FastLlamaModel: layer.self_attn.apply_o = original_apply_o pass + # Save max_seq_length model.max_seq_length = max_position_embeddings + internal_model = model + while hasattr(internal_model, "model"): + internal_model.max_seq_length = max_position_embeddings + internal_model = internal_model.model + pass + internal_model.max_seq_length = max_position_embeddings return model, tokenizer pass @@ -757,9 +765,9 @@ class FastLlamaModel: assert(max_seq_length <= model.max_seq_length) if lora_dropout != 0: - raise TypeError("Unsloth: Fast Llama patching only works with dropout = 0.") + raise TypeError("Unsloth: Fast model patching only works with dropout = 0.") if bias != "none": - raise TypeError("Unsloth: Fast Llama patching only works with bias = 'none'.") + raise TypeError("Unsloth: Fast model patching only works with bias = 'none'.") transformers_set_seed(random_state) diff --git a/unsloth/models/loader.py b/unsloth/models/loader.py index d458626386..421b743299 100644 --- a/unsloth/models/loader.py +++ b/unsloth/models/loader.py @@ -15,6 +15,21 @@ from .llama import FastLlamaModel, logger from .mistral import FastMistralModel from transformers import AutoConfig +from transformers import __version__ as transformers_version + +FOURBIT_MAPPER = \ +{ + "unsloth/mistral-7b-bnb-4bit" : "unsloth/mistral-7b", + "unsloth/llama-2-7b-bnb-4bit" : "unsloth/llama-2-7b", + "unsloth/llama-2-13b-bnb-4bit" : "unsloth/llama-13-7b", + "unsloth/codellama-34b-bnb-4bit" : "codellama/CodeLlama-34b-hf", +} + +# https://github.com/huggingface/transformers/pull/26037 allows 4 bit loading! +major, minor = transformers_version.split(".")[:2] +major, minor = int(major), int(minor) +SUPPORTS_FOURBIT = (major > 4) or (major == 4 and minor >= 37) +del major, minor class FastLanguageModel(FastLlamaModel): @@ -29,36 +44,37 @@ class FastLanguageModel(FastLlamaModel): rope_scaling = None, *args, **kwargs, ): + if not SUPPORTS_FOURBIT and model_name in FOURBIT_MAPPER: + model_name = FOURBIT_MAPPER[model_name] + logger.warning_once( + f"Unsloth: Your transformers version of {transformers_version} does not support native "\ + f"4bit loading.\nThe minimum required version is 4.37.\n"\ + f'Try `pip install "git+https://github.com/huggingface/transformers.git"`\n'\ + f"to obtain the latest transformers build, then restart this session.\n"\ + f"For now, we shall load `{model_name}` instead (still 4bit, just slower downloading)." + ) + pass + model_config = AutoConfig.from_pretrained(model_name) model_type = model_config.model_type - if model_type == "llama": - return FastLlamaModel.from_pretrained( - model_name = model_name, - max_seq_length = max_seq_length, - dtype = dtype, - load_in_4bit = load_in_4bit, - token = token, - device_map = device_map, - rope_scaling = rope_scaling, - *args, **kwargs, - ) - elif model_type == "mistral": - if rope_scaling is not None: - logger.warning_once("Unsloth: Mistral models do not support RoPE scaling.") - return FastMistralModel.from_pretrained( - model_name = model_name, - max_seq_length = max_seq_length, - dtype = dtype, - load_in_4bit = load_in_4bit, - token = token, - device_map = device_map, - *args, **kwargs, - ) + if model_type == "llama": dispatch_model = FastLlamaModel + elif model_type == "mistral": dispatch_model = FastMistralModel else: raise NotImplementedError( f"Unsloth: {model_name} not supported yet!\n"\ "Make an issue to https://github.com/unslothai/unsloth!", ) + + return dispatch_model.from_pretrained( + model_name = model_name, + max_seq_length = max_seq_length, + dtype = dtype, + load_in_4bit = load_in_4bit, + token = token, + device_map = device_map, + rope_scaling = rope_scaling, + *args, **kwargs, + ) pass pass diff --git a/unsloth/models/mistral.py b/unsloth/models/mistral.py index 826ec47789..e4bac44694 100644 --- a/unsloth/models/mistral.py +++ b/unsloth/models/mistral.py @@ -243,7 +243,7 @@ class FastMistralModel(FastLlamaModel): MistralDecoderLayer .forward = LlamaDecoderLayer_fast_forward MistralModel .forward = LlamaModel_fast_forward MistralForCausalLM .forward = MistralForCausalLM_fast_forward - PeftModelForCausalLM.forward = PeftModelForCausalLM_fast_forward + PeftModelForCausalLM .forward = PeftModelForCausalLM_fast_forward return pass @@ -256,8 +256,11 @@ class FastMistralModel(FastLlamaModel): load_in_4bit = True, token = None, device_map = "sequential", - # rope_scaling = None, Mistral does not support RoPE scaling - ): + rope_scaling = None, # Mistral does not support RoPE scaling + ): + if rope_scaling is not None: + logger.warning_once("Unsloth: Mistral models do not support RoPE scaling.") + SUPPORTS_BFLOAT16 = torch.cuda.is_bf16_supported() gpu_stats = torch.cuda.get_device_properties(0) max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3) @@ -312,7 +315,15 @@ class FastMistralModel(FastLlamaModel): layer.self_attn.apply_o = original_apply_o pass - model.max_seq_length = max(max_seq_length, model.config.max_position_embeddings) + # Save max_seq_length + max_position_embeddings = max(max_seq_length, model.config.max_position_embeddings) + model.max_seq_length = max_position_embeddings + internal_model = model + while hasattr(internal_model, "model"): + internal_model.max_seq_length = max_position_embeddings + internal_model = internal_model.model + pass + internal_model.max_seq_length = max_position_embeddings return model, tokenizer pass pass