Merge branch 'main' into nightly
This commit is contained in:
Daniel Han
commit
bdbe6ecb3d
14 changed files with 750 additions and 108 deletions
2
.github/workflows/stale.yml
vendored
2
.github/workflows/stale.yml
vendored
|
|
@ -11,7 +11,7 @@ jobs:
|
|||
issues: write
|
||||
|
||||
steps:
|
||||
- uses: actions/stale@v9
|
||||
- uses: actions/stale@v10
|
||||
with:
|
||||
# The message to post on stale issues.
|
||||
# This message will ping the issue author.
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
repos:
|
||||
- repo: https://github.com/astral-sh/ruff-pre-commit
|
||||
rev: v0.14.13
|
||||
rev: v0.14.14
|
||||
hooks:
|
||||
- id: ruff
|
||||
args:
|
||||
|
|
|
|||
32
README.md
32
README.md
|
|
@ -23,18 +23,18 @@ Notebooks are beginner friendly. Read our [guide](https://unsloth.ai/docs/get-st
|
|||
| Model | Free Notebooks | Performance | Memory use |
|
||||
|-----------|---------|--------|----------|
|
||||
| **gpt-oss (20B)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-Fine-tuning.ipynb) | 1.5x faster | 70% less |
|
||||
| **Mistral Ministral 3 (3B)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_VL_(3B)_Vision.ipynb) | 1.5x faster | 60% less |
|
||||
| **gpt-oss (20B): GRPO** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-GRPO.ipynb) | 2x faster | 80% less |
|
||||
| **Qwen3: Advanced GRPO** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb) | 2x faster | 50% less |
|
||||
| **Qwen3-VL (8B): GSPO** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision-GRPO.ipynb) | 1.5x faster | 80% less |
|
||||
| **Gemma 3 (270M)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(270M).ipynb) | 1.7x faster | 60% less |
|
||||
| **Gemma 3n (4B)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Conversational.ipynb) | 1.5x faster | 50% less |
|
||||
| **DeepSeek-OCR (3B)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Deepseek_OCR_(3B).ipynb) | 1.5x faster | 30% less |
|
||||
| **Gemma 3 (4B) Vision** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B)-Vision.ipynb) | 1.7x faster | 60% less |
|
||||
| **Gemma 3n (e4B)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Conversational.ipynb) | 1.5x faster | 50% less |
|
||||
| **embeddinggemma (300M)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/EmbeddingGemma_(300M).ipynb) | 2x faster | 20% less |
|
||||
| **Mistral Ministral 3 (3B)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_VL_(3B)_Vision.ipynb) | 1.5x faster | 60% less |
|
||||
| **Llama 3.1 (8B) Alpaca** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-Alpaca.ipynb) | 2x faster | 70% less |
|
||||
| **Llama 3.2 Conversational** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(1B_and_3B)-Conversational.ipynb) | 2x faster | 70% less |
|
||||
| **Orpheus-TTS (3B)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Orpheus_(3B)-TTS.ipynb) | 1.5x faster | 50% less |
|
||||
|
||||
- See all our notebooks for: [Kaggle](https://github.com/unslothai/notebooks?tab=readme-ov-file#-kaggle-notebooks), [GRPO](https://unsloth.ai/docs/get-started/unsloth-notebooks#grpo-reasoning-rl-notebooks), [TTS](https://unsloth.ai/docs/get-started/unsloth-notebooks#text-to-speech-tts-notebooks) & [Vision](https://unsloth.ai/docs/get-started/unsloth-notebooks#vision-multimodal-notebooks)
|
||||
- See all our notebooks for: [Kaggle](https://github.com/unslothai/notebooks?tab=readme-ov-file#-kaggle-notebooks), [GRPO](https://unsloth.ai/docs/get-started/unsloth-notebooks#grpo-reasoning-rl-notebooks), [TTS](https://unsloth.ai/docs/get-started/unsloth-notebooks#text-to-speech-tts-notebooks), [embedding](https://unsloth.ai/docs/new/embedding-finetuning) & [Vision](https://unsloth.ai/docs/get-started/unsloth-notebooks#vision-multimodal-notebooks)
|
||||
- See [all our models](https://unsloth.ai/docs/get-started/unsloth-model-catalog) and [all our notebooks](https://unsloth.ai/docs/get-started/unsloth-notebooks)
|
||||
- See detailed documentation for Unsloth [here](https://unsloth.ai/docs)
|
||||
|
||||
|
|
@ -53,22 +53,22 @@ Use our official [Unsloth Docker image](https://hub.docker.com/r/unsloth/unsloth
|
|||
For RTX 50x, B200, 6000 GPUs: `pip install unsloth`. Read our [Blackwell Guide](https://unsloth.ai/docs/basics/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth) and [DGX Spark Guide](https://unsloth.ai/docs/basics/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth) for more details.
|
||||
|
||||
## 🦥 Unsloth News
|
||||
- New 7x longer context reinforcement learning vs. all other setups, via our new batching algorithms. [Blog](https://unsloth.ai/docs/new/grpo-long-context)
|
||||
- **Embedding models**: Unsloth now supports ~1.8-3.3x faster embedding fine-tuning. [Blog](https://unsloth.ai/docs/new/embedding-finetuning) • [Notebooks](https://unsloth.ai/docs/get-started/unsloth-notebooks#embedding-models)
|
||||
- New **7x longer context RL** vs. all other setups, via our new batching algorithms. [Blog](https://unsloth.ai/docs/new/grpo-long-context)
|
||||
- New RoPE & MLP **Triton Kernels** & **Padding Free + Packing**: 3x faster training & 30% less VRAM. [Blog](https://unsloth.ai/docs/new/3x-faster-training-packing)
|
||||
- **Mistral 3**: Run Ministral 3 or Devstral 2 and fine-tune with vision/RL sodoku notebooks. [Guide](https://unsloth.ai/docs/models/ministral-3) • [Notebooks](https://unsloth.ai/docs/models/ministral-3#fine-tuning-ministral-3)
|
||||
- **500K Context**: Training a 20B model with >500K context is now possible on an 80GB GPU. [Blog](https://unsloth.ai/docs/new/500k-context-length-fine-tuning)
|
||||
- **FP8 Reinforcement Learning**: You can now do FP8 GRPO on consumer GPUs. [Blog](https://unsloth.ai/docs/new/fp8-reinforcement-learning) • [Notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_8B_FP8_GRPO.ipynb)
|
||||
- **DeepSeek-OCR**: Fine-tune to improve language understanding by 89%. [Guide](https://unsloth.ai/docs/models/deepseek-ocr-how-to-run-and-fine-tune) • [Notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Deepseek_OCR_(3B).ipynb)
|
||||
- **Docker**: Use Unsloth with no setup & environment issues with our new image. [Guide](https://unsloth.ai/docs/new/how-to-fine-tune-llms-with-unsloth-and-docker) • [Docker image](https://hub.docker.com/r/unsloth/unsloth)
|
||||
- **gpt-oss RL**: Introducing the fastest possible inference for gpt-oss RL! [Read blog](https://unsloth.ai/docs/models/gpt-oss-how-to-run-and-fine-tune/gpt-oss-reinforcement-learning)
|
||||
- **Vision RL**: You can now train VLMs with GRPO or GSPO in Unsloth! [Read guide](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/vision-reinforcement-learning-vlm-rl)
|
||||
- **gpt-oss** by OpenAI: Read our [Unsloth Flex Attention](https://unsloth.ai/docs/models/gpt-oss-how-to-run-and-fine-tune/long-context-gpt-oss-training) blog and [gpt-oss Guide](https://unsloth.ai/docs/models/gpt-oss-how-to-run-and-fine-tune). 20B works on 14GB VRAM. 120B on 65GB.
|
||||
- **gpt-oss** by OpenAI: Read our [RL blog](https://unsloth.ai/docs/models/gpt-oss-how-to-run-and-fine-tune/gpt-oss-reinforcement-learning), [Flex Attention](https://unsloth.ai/docs/models/gpt-oss-how-to-run-and-fine-tune/long-context-gpt-oss-training) blog and [gpt-oss Guide](https://unsloth.ai/docs/models/gpt-oss-how-to-run-and-fine-tune). 20B works on 14GB VRAM. 120B on 65GB.
|
||||
|
||||
<details>
|
||||
<summary>Click for more news</summary>
|
||||
|
||||
- **Quantization-Aware Training**: We collabed with Pytorch, recovering ~70% accuracy. [Read blog](https://unsloth.ai/docs/basics/quantization-aware-training-qat)
|
||||
- **Memory-efficient RL**: We're introducing even better RL. Our new kernels & algos allows faster RL with 50% less VRAM & 10× more context. [Read blog](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/memory-efficient-rl)
|
||||
- **Mistral 3**: Run Ministral 3 or Devstral 2 and fine-tune with vision/RL sodoku notebooks. [Guide](https://unsloth.ai/docs/models/ministral-3) • [Notebooks](https://unsloth.ai/docs/models/ministral-3#fine-tuning-ministral-3)
|
||||
- **Gemma 3n** by Google: [Read Blog](https://unsloth.ai/docs/models/gemma-3-how-to-run-and-fine-tune/gemma-3n-how-to-run-and-fine-tune). We [uploaded GGUFs, 4-bit models](https://huggingface.co/collections/unsloth/gemma-3n-685d3874830e49e1c93f9339).
|
||||
- **[Text-to-Speech (TTS)](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning)** is now supported, including `sesame/csm-1b` and STT `openai/whisper-large-v3`.
|
||||
- **[Qwen3](https://unsloth.ai/docs/models/qwen3-how-to-run-and-fine-tune)** is now supported. Qwen3-30B-A3B fits on 17.5GB VRAM.
|
||||
|
|
@ -99,7 +99,7 @@ For RTX 50x, B200, 6000 GPUs: `pip install unsloth`. Read our [Blackwell Guide](
|
|||
## ⭐ Key Features
|
||||
|
||||
* Supports **full-finetuning**, pretraining, 4b-bit, 16-bit and **FP8** training
|
||||
* Supports **all models** including [TTS](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning), multimodal, [BERT](https://unsloth.ai/docs/get-started/unsloth-notebooks#other-important-notebooks) and more! Any model that works in transformers, works in Unsloth.
|
||||
* Supports **all models** including [TTS](https://unsloth.ai/docs/basics/text-to-speech-tts-fine-tuning), multimodal, [embedding](https://unsloth.ai/docs/new/embedding-finetuning) and more! Any model that works in transformers, works in Unsloth.
|
||||
* The most efficient library for [Reinforcement Learning (RL)](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide), using 80% less VRAM. Supports GRPO, GSPO, DrGRPO, DAPO etc.
|
||||
* **0% loss in accuracy** - no approximation methods - all exact.
|
||||
* Export and [deploy your model](https://unsloth.ai/docs/basics/inference-and-deployment) to GGUF, llama.cpp, vLLM, SGLang and Hugging Face.
|
||||
|
|
@ -194,9 +194,9 @@ pip install unsloth
|
|||
</details>
|
||||
|
||||
### Advanced Pip Installation
|
||||
`⚠️Do **NOT** use this if you have Conda.` Pip is a bit more complex since there are dependency issues. The pip command is different for `torch 2.2,2.3,2.4,2.5,2.6,2.7,2.8,2.9` and CUDA versions.
|
||||
`⚠️Do **NOT** use this if you have Conda.` Pip is a bit more complex since there are dependency issues. The pip command is different for `torch 2.2,2.3,2.4,2.5,2.6,2.7,2.8,2.9,2.10` and CUDA versions.
|
||||
|
||||
For other torch versions, we support `torch211`, `torch212`, `torch220`, `torch230`, `torch240`, `torch250`, `torch260`, `torch270`, `torch280`, `torch290` and for CUDA versions, we support `cu118` and `cu121` and `cu124`. For Ampere devices (A100, H100, RTX3090) and above, use `cu118-ampere` or `cu121-ampere` or `cu124-ampere`.
|
||||
For other torch versions, we support `torch211`, `torch212`, `torch220`, `torch230`, `torch240`, `torch250`, `torch260`, `torch270`, `torch280`, `torch290`, `torch2100` and for CUDA versions, we support `cu118` and `cu121` and `cu124`. For Ampere devices (A100, H100, RTX3090) and above, use `cu118-ampere` or `cu121-ampere` or `cu124-ampere`. Note: torch 2.10 only supports CUDA 12.6, 12.8, and 13.0.
|
||||
|
||||
For example, if you have `torch 2.4` and `CUDA 12.1`, use:
|
||||
```bash
|
||||
|
|
@ -210,6 +210,12 @@ pip install --upgrade pip
|
|||
pip install "unsloth[cu130-torch290] @ git+https://github.com/unslothai/unsloth.git"
|
||||
```
|
||||
|
||||
Another example, if you have `torch 2.10` and `CUDA 12.6`, use:
|
||||
```bash
|
||||
pip install --upgrade pip
|
||||
pip install "unsloth[cu126-torch2100] @ git+https://github.com/unslothai/unsloth.git"
|
||||
```
|
||||
|
||||
And other examples:
|
||||
```bash
|
||||
pip install "unsloth[cu121-ampere-torch240] @ git+https://github.com/unslothai/unsloth.git"
|
||||
|
|
@ -254,8 +260,10 @@ elif v < V('2.8.0'): x = 'cu{}{}-torch271'
|
|||
elif v < V('2.8.9'): x = 'cu{}{}-torch280'
|
||||
elif v < V('2.9.1'): x = 'cu{}{}-torch290'
|
||||
elif v < V('2.9.2'): x = 'cu{}{}-torch291'
|
||||
elif v < V('2.10.1'): x = 'cu{}{}-torch2100'
|
||||
else: raise RuntimeError(f"Torch = {v} too new!")
|
||||
if v > V('2.6.9') and cuda not in ("11.8", "12.6", "12.8", "13.0"): raise RuntimeError(f"CUDA = {cuda} not supported!")
|
||||
if v >= V('2.10.0') and cuda not in ("12.6", "12.8", "13.0"): raise RuntimeError(f"Torch 2.10 requires CUDA 12.6, 12.8, or 13.0! Got CUDA = {cuda}")
|
||||
x = x.format(cuda.replace(".", ""), "-ampere" if False else "") # is_ampere is broken due to flash-attn
|
||||
print(f'pip install --upgrade pip && pip install --no-deps git+https://github.com/unslothai/unsloth-zoo.git && pip install "unsloth[{x}] @ git+https://github.com/unslothai/unsloth.git" --no-build-isolation')
|
||||
```
|
||||
|
|
|
|||
|
|
@ -57,6 +57,7 @@ huggingfacenotorch = [
|
|||
"diffusers",
|
||||
"transformers>=4.51.3,!=4.52.0,!=4.52.1,!=4.52.2,!=4.52.3,!=4.53.0,!=4.54.0,!=4.55.0,!=4.55.1,!=4.57.0,!=4.57.4,!=4.57.5,<=4.57.6",
|
||||
"trl>=0.18.2,!=0.19.0,<=0.24.0",
|
||||
"sentence-transformers",
|
||||
]
|
||||
huggingface = [
|
||||
"unsloth[huggingfacenotorch]",
|
||||
|
|
@ -296,6 +297,18 @@ cu130onlytorch291 = [
|
|||
"xformers @ https://download.pytorch.org/whl/cu130/xformers-0.0.33.post2-cp39-abi3-manylinux_2_28_x86_64.whl ; ('linux' in sys_platform)",
|
||||
"xformers @ https://download.pytorch.org/whl/cu130/xformers-0.0.33.post2-cp39-abi3-win_amd64.whl ; (sys_platform == 'win32')",
|
||||
]
|
||||
cu126onlytorch2100 = [
|
||||
"xformers @ https://download.pytorch.org/whl/cu126/xformers-0.0.34-cp39-abi3-manylinux_2_28_x86_64.whl ; ('linux' in sys_platform)",
|
||||
"xformers @ https://download.pytorch.org/whl/cu126/xformers-0.0.34-cp39-abi3-win_amd64.whl ; (sys_platform == 'win32')",
|
||||
]
|
||||
cu128onlytorch2100 = [
|
||||
"xformers @ https://download.pytorch.org/whl/cu128/xformers-0.0.34-cp39-abi3-manylinux_2_28_x86_64.whl ; ('linux' in sys_platform)",
|
||||
"xformers @ https://download.pytorch.org/whl/cu128/xformers-0.0.34-cp39-abi3-win_amd64.whl ; (sys_platform == 'win32')",
|
||||
]
|
||||
cu130onlytorch2100 = [
|
||||
"xformers @ https://download.pytorch.org/whl/cu130/xformers-0.0.34-cp39-abi3-manylinux_2_28_x86_64.whl ; ('linux' in sys_platform)",
|
||||
"xformers @ https://download.pytorch.org/whl/cu130/xformers-0.0.34-cp39-abi3-win_amd64.whl ; (sys_platform == 'win32')",
|
||||
]
|
||||
cu118 = [
|
||||
"unsloth[huggingface]",
|
||||
"bitsandbytes>=0.45.5,!=0.46.0,!=0.48.0",
|
||||
|
|
@ -486,6 +499,21 @@ cu130-torch291 = [
|
|||
"bitsandbytes>=0.45.5,!=0.46.0,!=0.48.0",
|
||||
"unsloth[cu130onlytorch291]",
|
||||
]
|
||||
cu126-torch2100 = [
|
||||
"unsloth[huggingface]",
|
||||
"bitsandbytes>=0.45.5,!=0.46.0,!=0.48.0",
|
||||
"unsloth[cu126onlytorch2100]",
|
||||
]
|
||||
cu128-torch2100 = [
|
||||
"unsloth[huggingface]",
|
||||
"bitsandbytes>=0.45.5,!=0.46.0,!=0.48.0",
|
||||
"unsloth[cu128onlytorch2100]",
|
||||
]
|
||||
cu130-torch2100 = [
|
||||
"unsloth[huggingface]",
|
||||
"bitsandbytes>=0.45.5,!=0.46.0,!=0.48.0",
|
||||
"unsloth[cu130onlytorch2100]",
|
||||
]
|
||||
kaggle = [
|
||||
"unsloth[huggingface]",
|
||||
]
|
||||
|
|
@ -538,6 +566,7 @@ colab-new = [
|
|||
"hf_transfer",
|
||||
"bitsandbytes>=0.45.5,!=0.46.0,!=0.48.0",
|
||||
"unsloth[triton]",
|
||||
"sentence-transformers",
|
||||
]
|
||||
colab-no-deps = [
|
||||
"accelerate>=0.34.1",
|
||||
|
|
@ -769,6 +798,21 @@ cu130-ampere-torch291 = [
|
|||
"bitsandbytes>=0.45.5,!=0.46.0,!=0.48.0",
|
||||
"unsloth[cu130onlytorch291]",
|
||||
]
|
||||
cu126-ampere-torch2100 = [
|
||||
"unsloth[huggingface]",
|
||||
"bitsandbytes>=0.45.5,!=0.46.0,!=0.48.0",
|
||||
"unsloth[cu126onlytorch2100]",
|
||||
]
|
||||
cu128-ampere-torch2100 = [
|
||||
"unsloth[huggingface]",
|
||||
"bitsandbytes>=0.45.5,!=0.46.0,!=0.48.0",
|
||||
"unsloth[cu128onlytorch2100]",
|
||||
]
|
||||
cu130-ampere-torch2100 = [
|
||||
"unsloth[huggingface]",
|
||||
"bitsandbytes>=0.45.5,!=0.46.0,!=0.48.0",
|
||||
"unsloth[cu130onlytorch2100]",
|
||||
]
|
||||
flashattentiontorch260abiFALSEcu12x = [
|
||||
"flash-attn @ https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.4.post1/flash_attn-2.7.4.post1+cu12torch2.6cxx11abiFALSE-cp39-cp39-linux_x86_64.whl ; ('linux' in sys_platform) and python_version == '3.9'",
|
||||
"flash-attn @ https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.4.post1/flash_attn-2.7.4.post1+cu12torch2.6cxx11abiFALSE-cp310-cp310-linux_x86_64.whl ; ('linux' in sys_platform) and python_version == '3.10'",
|
||||
|
|
@ -861,7 +905,6 @@ intelgputorch270 = [
|
|||
"torch @ https://download.pytorch.org/whl/xpu/torch-2.7.0%2Bxpu-cp311-cp311-win_amd64.whl#sha256=356ba66cee127e7e2c942880bd50e03768306a4ea08d358a0f29c6eebfc4bc81 ; sys_platform == 'win32' and python_version == '3.11' and (platform_machine == 'AMD64' or platform_machine == 'x86_64')",
|
||||
"torch @ https://download.pytorch.org/whl/xpu/torch-2.7.0%2Bxpu-cp312-cp312-win_amd64.whl#sha256=94739e665d9b4d5cd7af5f517cb6103f6f9fb421c095184609653a24524040f5 ; sys_platform == 'win32' and python_version == '3.12' and (platform_machine == 'AMD64' or platform_machine == 'x86_64')",
|
||||
"torch @ https://download.pytorch.org/whl/xpu/torch-2.7.0%2Bxpu-cp313-cp313-win_amd64.whl#sha256=31df3cb674918e89bc8c532baa331dc84f4430e1f9c0ec379232db44cba78355 ; sys_platform == 'win32' and python_version == '3.13' and (platform_machine == 'AMD64' or platform_machine == 'x86_64')",
|
||||
|
||||
]
|
||||
intel-gpu-torch270 = [
|
||||
"unsloth[intelgputorch270]"
|
||||
|
|
@ -905,7 +948,6 @@ intelgputorch280 = [
|
|||
"torchvision @ https://download.pytorch.org/whl/xpu/torchvision-0.23.0%2Bxpu-cp311-cp311-win_amd64.whl#sha256=c6d030f5361461550c0ff1339b5bca8585fc1e84fda2e64b6184e65a581e4f98 ; sys_platform == 'win32' and python_version == '3.11' and (platform_machine == 'AMD64' or platform_machine == 'x86_64')",
|
||||
"torchvision @ https://download.pytorch.org/whl/xpu/torchvision-0.23.0%2Bxpu-cp312-cp312-win_amd64.whl#sha256=91aafd61864cdce27461cbec13ddbf28c1bc6494265a1e4b80131c64a3b7d18f ; sys_platform == 'win32' and python_version == '3.12' and (platform_machine == 'AMD64' or platform_machine == 'x86_64')",
|
||||
"torchvision @ https://download.pytorch.org/whl/xpu/torchvision-0.23.0%2Bxpu-cp313-cp313-win_amd64.whl#sha256=71dc4a6421742ed1e7f585b04a100ad53615c341fbccfbc255aefb38ea9091da ; sys_platform == 'win32' and python_version == '3.13' and (platform_machine == 'AMD64' or platform_machine == 'x86_64')",
|
||||
|
||||
]
|
||||
intel-gpu-torch280 = [
|
||||
"unsloth[intelgputorch280]"
|
||||
|
|
@ -943,8 +985,6 @@ intelgputorch290 = [
|
|||
"torchvision @ https://download.pytorch.org/whl/xpu/torchvision-0.24.0%2Bxpu-cp311-cp311-win_amd64.whl#sha256=9bb0d1421c544ac8e2eca5b47daacaf54706dc9139c003aa5e77ee5f355c5931 ; sys_platform == 'win32' and python_version == '3.11' and (platform_machine == 'AMD64' or platform_machine == 'x86_64')",
|
||||
"torchvision @ https://download.pytorch.org/whl/xpu/torchvision-0.24.0%2Bxpu-cp312-cp312-win_amd64.whl#sha256=6a5194bc736089606342d48a3f6822829b167617e9495d91d753dd1bd46fda18 ; sys_platform == 'win32' and python_version == '3.12' and (platform_machine == 'AMD64' or platform_machine == 'x86_64')",
|
||||
"torchvision @ https://download.pytorch.org/whl/xpu/torchvision-0.24.0%2Bxpu-cp313-cp313-win_amd64.whl#sha256=da47a3ce2bb7f0301a31124668b5908f9b9e92d6241443de15a310ef9632fd83 ; sys_platform == 'win32' and python_version == '3.13' and (platform_machine == 'AMD64' or platform_machine == 'x86_64')",
|
||||
|
||||
|
||||
]
|
||||
intel-gpu-torch290 = [
|
||||
"unsloth[intelgputorch290]"
|
||||
|
|
|
|||
|
|
@ -125,43 +125,56 @@ from unsloth_zoo.device_type import (
|
|||
from .import_fixes import (
|
||||
fix_xformers_performance_issue,
|
||||
fix_vllm_aimv2_issue,
|
||||
check_vllm_torch_sm100_compatibility,
|
||||
fix_vllm_guided_decoding_params,
|
||||
fix_vllm_pdl_blackwell,
|
||||
fix_rocm_triton_key_error,
|
||||
ignore_logger_messages,
|
||||
patch_ipykernel_hf_xet,
|
||||
patch_trackio,
|
||||
patch_datasets,
|
||||
patch_enable_input_require_grads,
|
||||
fix_openenv_no_vllm,
|
||||
patch_openspiel_env_async,
|
||||
fix_executorch,
|
||||
patch_vllm_for_notebooks,
|
||||
patch_torchcodec_audio_decoder,
|
||||
)
|
||||
|
||||
fix_xformers_performance_issue()
|
||||
fix_vllm_aimv2_issue()
|
||||
# Check vLLM + torch < 2.9.0 + SM100 compatibility BEFORE importing vLLM
|
||||
check_vllm_torch_sm100_compatibility()
|
||||
fix_vllm_guided_decoding_params()
|
||||
fix_vllm_pdl_blackwell()
|
||||
fix_rocm_triton_key_error()
|
||||
ignore_logger_messages()
|
||||
patch_ipykernel_hf_xet()
|
||||
patch_trackio()
|
||||
patch_datasets()
|
||||
patch_enable_input_require_grads()
|
||||
fix_openenv_no_vllm()
|
||||
patch_openspiel_env_async()
|
||||
fix_executorch()
|
||||
patch_vllm_for_notebooks()
|
||||
patch_torchcodec_audio_decoder()
|
||||
|
||||
del fix_xformers_performance_issue
|
||||
del fix_vllm_aimv2_issue
|
||||
del check_vllm_torch_sm100_compatibility
|
||||
del fix_vllm_guided_decoding_params
|
||||
del fix_vllm_pdl_blackwell
|
||||
del fix_rocm_triton_key_error
|
||||
del ignore_logger_messages
|
||||
del patch_ipykernel_hf_xet
|
||||
del patch_trackio
|
||||
del patch_datasets
|
||||
del patch_enable_input_require_grads
|
||||
del fix_openenv_no_vllm
|
||||
del patch_openspiel_env_async
|
||||
del fix_executorch
|
||||
del patch_vllm_for_notebooks
|
||||
del patch_torchcodec_audio_decoder
|
||||
|
||||
# Torch 2.4 has including_emulation
|
||||
if DEVICE_TYPE == "cuda":
|
||||
|
|
|
|||
|
|
@ -35,7 +35,9 @@ elif v < V('2.8.0'): x = 'cu{}{}-torch271'
|
|||
elif v < V('2.8.9'): x = 'cu{}{}-torch280'
|
||||
elif v < V('2.9.1'): x = 'cu{}{}-torch290'
|
||||
elif v < V('2.9.2'): x = 'cu{}{}-torch291'
|
||||
elif v < V('2.10.1'): x = 'cu{}{}-torch2100'
|
||||
else: raise RuntimeError(f"Torch = {v} too new!")
|
||||
if v > V('2.6.9') and cuda not in ("11.8", "12.6", "12.8", "13.0"): raise RuntimeError(f"CUDA = {cuda} not supported!")
|
||||
if v >= V('2.10.0') and cuda not in ("12.6", "12.8", "13.0"): raise RuntimeError(f"Torch 2.10 requires CUDA 12.6, 12.8, or 13.0! Got CUDA = {cuda}")
|
||||
x = x.format(cuda.replace(".", ""), "-ampere" if False else "") # is_ampere is broken due to flash-attn
|
||||
print(f'pip install --upgrade pip && pip install --no-deps git+https://github.com/unslothai/unsloth-zoo.git && pip install "unsloth[{x}] @ git+https://github.com/unslothai/unsloth.git" --no-build-isolation')
|
||||
|
|
@ -123,6 +123,47 @@ if os.environ.get("UNSLOTH_ENABLE_LOGGING", "0") != "1":
|
|||
warnings.filterwarnings("ignore", message = "`int4_weight_only` is deprecated")
|
||||
warnings.filterwarnings("ignore", message = "`int8_weight_only` is deprecated")
|
||||
|
||||
# TorchAO deprecated import paths (https://github.com/pytorch/ao/issues/2752)
|
||||
warnings.filterwarnings(
|
||||
"ignore",
|
||||
message = r"Importing.*from torchao\.dtypes.*is deprecated",
|
||||
category = DeprecationWarning,
|
||||
)
|
||||
warnings.filterwarnings(
|
||||
"ignore",
|
||||
message = r"Importing BlockSparseLayout from torchao\.dtypes is deprecated",
|
||||
category = DeprecationWarning,
|
||||
)
|
||||
|
||||
# SWIG builtin type warnings (from bitsandbytes/triton SWIG bindings)
|
||||
warnings.filterwarnings(
|
||||
"ignore",
|
||||
message = r"builtin type Swig.*has no __module__ attribute",
|
||||
category = DeprecationWarning,
|
||||
)
|
||||
|
||||
# Triton autotuner deprecation (https://github.com/triton-lang/triton/pull/4496)
|
||||
warnings.filterwarnings(
|
||||
"ignore",
|
||||
message = r"warmup, rep, and use_cuda_graph parameters are deprecated",
|
||||
category = DeprecationWarning,
|
||||
)
|
||||
|
||||
# Python 3.12+ multiprocessing fork warning in multi-threaded processes
|
||||
warnings.filterwarnings(
|
||||
"ignore",
|
||||
message = r".*multi-threaded.*use of fork\(\) may lead to deadlocks",
|
||||
category = DeprecationWarning,
|
||||
)
|
||||
|
||||
# Resource warnings from internal socket/file operations
|
||||
warnings.filterwarnings(
|
||||
"ignore", message = r"unclosed.*socket", category = ResourceWarning
|
||||
)
|
||||
warnings.filterwarnings(
|
||||
"ignore", message = r"unclosed file.*dev/null", category = ResourceWarning
|
||||
)
|
||||
|
||||
|
||||
# Fix up AttributeError: 'MessageFactory' object has no attribute 'GetPrototype'
|
||||
# MUST do this at the start primarily due to tensorflow causing issues
|
||||
|
|
@ -503,46 +544,138 @@ def patch_enable_input_require_grads():
|
|||
)
|
||||
|
||||
|
||||
def _is_custom_torch_build(raw_version_str):
|
||||
"""Check if a raw version string indicates a custom or source build.
|
||||
Must operate on the raw string from importlib_version(), not the parsed
|
||||
Version object, since our custom Version() strips local identifiers.
|
||||
|
||||
Standard PyTorch releases use: +cu124, +rocm6.3, +cpu, +xpu
|
||||
Source/custom builds use: +gitXXXXXXX, +HEXHASH, or other suffixes.
|
||||
"""
|
||||
if "+" not in raw_version_str:
|
||||
return False
|
||||
local = raw_version_str.split("+", 1)[1]
|
||||
if not local:
|
||||
return False
|
||||
# Use fullmatch so the entire local identifier must match, not just a prefix.
|
||||
# cu/rocm require a trailing digit (e.g. cu124, rocm6.3). cpu/xpu are exact.
|
||||
# Case-insensitive since some builds may use uppercase.
|
||||
return not re.fullmatch(r"cu\d[\d.]*|rocm\d[\d.]*|cpu|xpu", local, re.IGNORECASE)
|
||||
|
||||
|
||||
def _infer_required_torchvision(torch_major, torch_minor):
|
||||
"""Infer the minimum required torchvision minor version from torch version.
|
||||
|
||||
The torch -> torchvision minor version mapping follows a consistent formula:
|
||||
torch 1.x -> torchvision 0.(x + 1) (verified: torch 1.7 through 1.13)
|
||||
torch 2.x -> torchvision 0.(x + 15) (verified: torch 2.0 through 2.9)
|
||||
|
||||
Returns (tv_major, tv_minor) or None if the major version is unrecognized.
|
||||
"""
|
||||
if torch_major == 1 and torch_minor >= 7:
|
||||
return (0, torch_minor + 1)
|
||||
if torch_major == 2:
|
||||
return (0, torch_minor + 15)
|
||||
return None
|
||||
|
||||
|
||||
def torchvision_compatibility_check():
|
||||
# Allow skipping via environment variable for custom environments
|
||||
if os.environ.get("UNSLOTH_SKIP_TORCHVISION_CHECK", "0").lower() in ("1", "true"):
|
||||
return
|
||||
|
||||
if importlib.util.find_spec("torch") is None:
|
||||
raise ImportError("Unsloth: torch not found. Please install torch first.")
|
||||
if importlib.util.find_spec("torchvision") is None:
|
||||
return
|
||||
torch_version = importlib_version("torch")
|
||||
torchvision_version = importlib_version("torchvision")
|
||||
|
||||
# Torch version -> minimum required torchvision version
|
||||
# See https://pytorch.org/get-started/previous-versions/
|
||||
TORCH_TORCHVISION_COMPAT = [
|
||||
("2.9.0", "0.24.0"),
|
||||
("2.8.0", "0.23.0"),
|
||||
("2.7.0", "0.22.0"),
|
||||
("2.6.0", "0.21.0"),
|
||||
("2.5.0", "0.20.0"),
|
||||
("2.4.0", "0.19.0"),
|
||||
]
|
||||
|
||||
required_torchvision = None
|
||||
for min_torch, min_torchvision in TORCH_TORCHVISION_COMPAT:
|
||||
if Version(torch_version) >= Version(min_torch):
|
||||
required_torchvision = min_torchvision
|
||||
break
|
||||
|
||||
if required_torchvision is None:
|
||||
# Torch version not in compatibility table, skip check
|
||||
try:
|
||||
torch_version_raw = importlib_version("torch")
|
||||
torchvision_version_raw = importlib_version("torchvision")
|
||||
except Exception:
|
||||
return
|
||||
|
||||
if Version(torchvision_version) < Version(required_torchvision):
|
||||
raise ImportError(
|
||||
f"Unsloth: torch=={torch_version} requires torchvision>={required_torchvision}, "
|
||||
f"but found torchvision=={torchvision_version}. "
|
||||
f"Please refer to https://pytorch.org/get-started/previous-versions/ for more information."
|
||||
)
|
||||
try:
|
||||
torch_v = Version(torch_version_raw)
|
||||
tv_v = Version(torchvision_version_raw)
|
||||
except Exception:
|
||||
return
|
||||
|
||||
logger.info(
|
||||
f"Unsloth: torch=={torch_version} and torchvision=={torchvision_version} are compatible."
|
||||
# Known compatibility table (ground truth, takes precedence over formula).
|
||||
# See https://pytorch.org/get-started/previous-versions/
|
||||
TORCH_TORCHVISION_COMPAT = {
|
||||
(2, 9): (0, 24),
|
||||
(2, 8): (0, 23),
|
||||
(2, 7): (0, 22),
|
||||
(2, 6): (0, 21),
|
||||
(2, 5): (0, 20),
|
||||
(2, 4): (0, 19),
|
||||
}
|
||||
|
||||
# Extract major.minor from the parsed version
|
||||
torch_release = torch_v.release
|
||||
if len(torch_release) < 2:
|
||||
return
|
||||
torch_major, torch_minor = torch_release[0], torch_release[1]
|
||||
|
||||
# Try known table first, then fall back to formula for forward compatibility
|
||||
required = TORCH_TORCHVISION_COMPAT.get((torch_major, torch_minor))
|
||||
is_in_known_table = required is not None
|
||||
|
||||
if required is None:
|
||||
required = _infer_required_torchvision(torch_major, torch_minor)
|
||||
|
||||
if required is None:
|
||||
return
|
||||
|
||||
required_tv_str = f"{required[0]}.{required[1]}.0"
|
||||
|
||||
if tv_v >= Version(required_tv_str):
|
||||
logger.info(
|
||||
f"Unsloth: torch=={torch_version_raw} and "
|
||||
f"torchvision=={torchvision_version_raw} are compatible."
|
||||
)
|
||||
return
|
||||
|
||||
# Version mismatch detected
|
||||
message = (
|
||||
f"Unsloth: torch=={torch_version_raw} requires "
|
||||
f"torchvision>={required_tv_str}, "
|
||||
f"but found torchvision=={torchvision_version_raw}. "
|
||||
f"Please refer to https://pytorch.org/get-started/previous-versions/ "
|
||||
f"for more information."
|
||||
)
|
||||
|
||||
is_custom = _is_custom_torch_build(torch_version_raw) or _is_custom_torch_build(
|
||||
torchvision_version_raw
|
||||
)
|
||||
|
||||
# Detect nightly/dev/alpha/beta/rc builds from the raw version string.
|
||||
# These often have version mismatches that are expected.
|
||||
_pre_tags = (".dev", "a0", "b0", "rc", "alpha", "beta", "nightly")
|
||||
is_prerelease = any(t in torch_version_raw for t in _pre_tags) or any(
|
||||
t in torchvision_version_raw for t in _pre_tags
|
||||
)
|
||||
|
||||
# Downgrade to warning for custom/source/pre-release builds or formula-predicted
|
||||
if is_custom or is_prerelease or not is_in_known_table:
|
||||
reason = (
|
||||
"custom/source build"
|
||||
if is_custom
|
||||
else "pre-release build"
|
||||
if is_prerelease
|
||||
else "newer torch version"
|
||||
)
|
||||
logger.warning(
|
||||
f"{message}\n"
|
||||
f"Detected a {reason}. "
|
||||
f"Continuing with a warning. "
|
||||
f"Set UNSLOTH_SKIP_TORCHVISION_CHECK=1 to silence this."
|
||||
)
|
||||
return
|
||||
|
||||
raise ImportError(message)
|
||||
|
||||
|
||||
# Fix TRL OpenEnv 0.26 NameError: name 'SamplingParams' is not defined
|
||||
def fix_openenv_no_vllm():
|
||||
|
|
@ -666,6 +799,105 @@ def fix_huggingface_hub():
|
|||
)
|
||||
|
||||
|
||||
def fix_rocm_triton_key_error():
|
||||
"""
|
||||
ROCm + torch.compile can fail if Triton lacks `triton_key`.
|
||||
Disable Inductor/compile only on ROCm when that symbol is missing.
|
||||
"""
|
||||
try:
|
||||
import torch
|
||||
except (ImportError, ModuleNotFoundError):
|
||||
return
|
||||
|
||||
if not getattr(torch.version, "hip", None):
|
||||
return
|
||||
|
||||
try:
|
||||
import triton
|
||||
except (ImportError, ModuleNotFoundError):
|
||||
return
|
||||
|
||||
try:
|
||||
from triton.runtime import triton_key # noqa: F401
|
||||
|
||||
return
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
os.environ.setdefault("TORCHINDUCTOR_DISABLE", "1")
|
||||
os.environ.setdefault("TORCH_COMPILE_DISABLE", "1")
|
||||
logger.info(
|
||||
"Unsloth: ROCm detected and Triton lacks triton_key; "
|
||||
"disabling torch.compile/Inductor to avoid backend crash."
|
||||
)
|
||||
|
||||
|
||||
def check_vllm_torch_sm100_compatibility():
|
||||
"""
|
||||
Check for incompatible vLLM + torch < 2.9.0 + SM100 (Blackwell) combination.
|
||||
|
||||
vLLM's distributed module (device_communicators) crashes with std::bad_alloc
|
||||
when imported on SM100 GPUs (B200/B100) with torch < 2.9.0. This is due to
|
||||
C++ code in vLLM's NCCL/distributed layer being incompatible with older
|
||||
torch versions on the newer Blackwell architecture.
|
||||
|
||||
This check runs early (before vLLM import) to provide a helpful error message
|
||||
instead of a cryptic std::bad_alloc crash.
|
||||
"""
|
||||
# Check if vLLM is installed (without importing it)
|
||||
if importlib.util.find_spec("vllm") is None:
|
||||
return
|
||||
|
||||
# Check torch version
|
||||
try:
|
||||
torch_version = Version(importlib_version("torch"))
|
||||
if torch_version >= Version("2.9.0"):
|
||||
return # torch >= 2.9.0 is compatible
|
||||
except Exception:
|
||||
return # Can't determine torch version, skip check
|
||||
|
||||
# Check if any CUDA GPU is SM100 (Blackwell)
|
||||
try:
|
||||
import torch
|
||||
|
||||
if not torch.cuda.is_available():
|
||||
return
|
||||
|
||||
has_sm100 = False
|
||||
sm100_gpu_name = None
|
||||
for i in range(torch.cuda.device_count()):
|
||||
major, minor = torch.cuda.get_device_capability(i)
|
||||
if major == 10:
|
||||
has_sm100 = True
|
||||
sm100_gpu_name = torch.cuda.get_device_name(i)
|
||||
break
|
||||
|
||||
if not has_sm100:
|
||||
return
|
||||
except Exception:
|
||||
return
|
||||
|
||||
# Get vLLM version for the error message
|
||||
try:
|
||||
vllm_version = importlib_version("vllm")
|
||||
except Exception:
|
||||
vllm_version = "unknown"
|
||||
|
||||
# Incompatible combination detected - raise helpful error
|
||||
raise RuntimeError(
|
||||
f"Unsloth: Incompatible configuration detected.\n\n"
|
||||
f" GPU: {sm100_gpu_name} (SM100 / Blackwell architecture)\n"
|
||||
f" torch version: {torch_version}\n"
|
||||
f" vLLM version: {vllm_version}\n\n"
|
||||
f"vLLM's distributed module crashes with std::bad_alloc on SM100 GPUs "
|
||||
f"(B200/B100/Blackwell) when using torch < 2.9.0.\n\n"
|
||||
f"To fix this, please upgrade torch:\n"
|
||||
f" pip install --upgrade torch>=2.9.0\n\n"
|
||||
f"Alternatively, if you don't need vLLM:\n"
|
||||
f" pip uninstall vllm"
|
||||
)
|
||||
|
||||
|
||||
def fix_vllm_pdl_blackwell():
|
||||
"""
|
||||
Fix vLLM PDL (Programmatic Dependent Launch) bug on Blackwell GPUs (SM100).
|
||||
|
|
@ -779,3 +1011,45 @@ def fix_vllm_pdl_blackwell():
|
|||
else:
|
||||
# Just set the env var - vLLM might be an older version without supports_pdl
|
||||
logger.info(f"Unsloth: Set TRITON_DISABLE_PDL=1 for SM100 ({sm100_gpu_name})")
|
||||
|
||||
|
||||
def patch_openspiel_env_async():
|
||||
"""Apply nest_asyncio for OpenEnv EnvClient async compatibility.
|
||||
|
||||
OpenEnv's EnvClient uses async methods (reset/step). In Jupyter notebooks
|
||||
these work via top-level await, but converted scripts need
|
||||
asyncio.get_event_loop().run_until_complete() wrappers. Applying nest_asyncio
|
||||
ensures nested event loop calls work in all contexts without replacing the
|
||||
original async methods (which would break scripts that already have their own
|
||||
sync wrappers).
|
||||
"""
|
||||
try:
|
||||
import inspect
|
||||
from openenv.core.env_client import EnvClient
|
||||
|
||||
if not inspect.iscoroutinefunction(EnvClient.reset):
|
||||
return # Already sync, nothing to do
|
||||
|
||||
try:
|
||||
import nest_asyncio
|
||||
|
||||
nest_asyncio.apply()
|
||||
logger.info(
|
||||
"Unsloth: Applied nest_asyncio for OpenEnv EnvClient async compatibility"
|
||||
)
|
||||
except ImportError:
|
||||
logger.info(
|
||||
"Unsloth: nest_asyncio not installed, OpenEnv async methods may need manual wrapping"
|
||||
)
|
||||
except (ImportError, AttributeError):
|
||||
pass # openenv not installed
|
||||
|
||||
|
||||
def patch_torchcodec_audio_decoder():
|
||||
"""Call unsloth_zoo's AudioDecoder patch."""
|
||||
try:
|
||||
from unsloth_zoo.dataset_utils import patch_torchcodec_audio_decoder as _patch
|
||||
|
||||
_patch()
|
||||
except (ImportError, AttributeError):
|
||||
pass
|
||||
|
|
|
|||
|
|
@ -59,9 +59,11 @@ __all__ = [
|
|||
"unsloth_fused_ce_loss",
|
||||
"patch_unsloth_smart_gradient_checkpointing",
|
||||
"unpatch_unsloth_smart_gradient_checkpointing",
|
||||
"apply_unsloth_gradient_checkpointing",
|
||||
"patch_compiled_autograd",
|
||||
"process_vision_info",
|
||||
"unsloth_compile_transformers",
|
||||
"prefer_flex_attn_if_supported",
|
||||
"patch_fast_lora",
|
||||
"validate_loftq_config",
|
||||
"RaiseUninitialized",
|
||||
|
|
@ -148,6 +150,68 @@ from unsloth_zoo.temporary_patches import (
|
|||
TEMPORARY_PATCHES,
|
||||
)
|
||||
|
||||
|
||||
def apply_unsloth_gradient_checkpointing(
|
||||
use_gradient_checkpointing, max_seq_length, dtype
|
||||
):
|
||||
"""
|
||||
Apply gradient checkpointing with smart heuristics.
|
||||
|
||||
For seq < 512, the overhead of gradient offloading in gc="unsloth" mode
|
||||
is not worth it. Benchmarks show standard gc is faster for small sequences.
|
||||
|
||||
Args:
|
||||
use_gradient_checkpointing: "unsloth", True, False, or None
|
||||
max_seq_length: The maximum sequence length
|
||||
dtype: The model dtype for patching
|
||||
|
||||
Returns:
|
||||
The effective use_gradient_checkpointing value (may change from "unsloth" to True)
|
||||
"""
|
||||
if use_gradient_checkpointing == "unsloth":
|
||||
# Gradient offloading overhead is not worth it for small sequences.
|
||||
# Benchmarks show crossover point is around seq_len 384-512.
|
||||
# For seq < 512, standard gradient checkpointing is faster.
|
||||
if max_seq_length < 512:
|
||||
unpatch_unsloth_smart_gradient_checkpointing()
|
||||
return True
|
||||
else:
|
||||
patch_unsloth_smart_gradient_checkpointing(dtype = dtype)
|
||||
return "unsloth"
|
||||
elif use_gradient_checkpointing in (True, False):
|
||||
# User explicitly set True or False - unpatch any previous "unsloth" patching
|
||||
unpatch_unsloth_smart_gradient_checkpointing()
|
||||
return use_gradient_checkpointing
|
||||
return use_gradient_checkpointing
|
||||
|
||||
|
||||
def prefer_flex_attn_if_supported(model_class, config):
|
||||
if os.environ.get("UNSLOTH_ENABLE_FLEX_ATTENTION", "1") == "0":
|
||||
return None
|
||||
try:
|
||||
from transformers.utils.import_utils import is_torch_flex_attn_available
|
||||
|
||||
if not is_torch_flex_attn_available():
|
||||
return None
|
||||
if model_class is None or not getattr(
|
||||
model_class, "_supports_flex_attn", False
|
||||
):
|
||||
return None
|
||||
# GPT-OSS uses eager attention during inference since flex attention
|
||||
# returns incorrect results (likely due to left padding issues).
|
||||
# Skip setting flex_attention to avoid BlockMask type errors.
|
||||
model_type = getattr(config, "model_type", "") if config else ""
|
||||
if model_type == "gpt_oss":
|
||||
return None
|
||||
if config is not None:
|
||||
setattr(config, "_attn_implementation", "flex_attention")
|
||||
if hasattr(config, "attn_implementation"):
|
||||
setattr(config, "attn_implementation", "flex_attention")
|
||||
return "flex_attention"
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
for temporary_patch in TEMPORARY_PATCHES:
|
||||
temporary_patch()
|
||||
|
||||
|
|
@ -1088,8 +1152,12 @@ def has_internet(host = "8.8.8.8", port = 53, timeout = 3):
|
|||
return False
|
||||
try:
|
||||
socket.setdefaulttimeout(timeout)
|
||||
socket.socket(socket.AF_INET, socket.SOCK_STREAM).connect((host, port))
|
||||
return True
|
||||
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
try:
|
||||
sock.connect((host, port))
|
||||
return True
|
||||
finally:
|
||||
sock.close()
|
||||
except socket.error as ex:
|
||||
return False
|
||||
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ import functools
|
|||
from typing import Optional, Tuple, List, Union
|
||||
|
||||
from ._utils import *
|
||||
from ._utils import patch_unsloth_smart_gradient_checkpointing
|
||||
from ._utils import apply_unsloth_gradient_checkpointing
|
||||
from ._utils import __version__, importlib_version
|
||||
from ._utils import move_to_device
|
||||
from ._utils import (
|
||||
|
|
@ -152,21 +152,22 @@ from peft.utils.other import ModulesToSaveWrapper
|
|||
def _offload_frozen_module_for_training(
|
||||
module: ModulesToSaveWrapper,
|
||||
device_type: str,
|
||||
offload_device: str = "cpu",
|
||||
offload_device: Optional[str] = "cpu",
|
||||
) -> None:
|
||||
"""
|
||||
Offload frozen module to CPU and configure trainable copy for mixed precision training.
|
||||
|
||||
This function optimizes memory usage by:
|
||||
1. Moving the trainable copy to the target device with appropriate precision
|
||||
2. Offloading the original frozen module to CPU/disk to free VRAM
|
||||
2. Optionally offloading the original frozen module to CPU/disk to free VRAM
|
||||
3. Converting float16 to float32 for compatibility with certain GPUs (e.g., Tesla T4)
|
||||
|
||||
Args:
|
||||
module: The module to configure. Must be a ModulesToSaveWrapper with a
|
||||
`modules_to_save` attribute containing trainable and original modules.
|
||||
device_type: Target device string for training (e.g., "cuda:0", "xpu:0")
|
||||
offload_device: Device to offload frozen parameters (default: "cpu")
|
||||
offload_device: Device to offload frozen parameters (default: "cpu").
|
||||
If None, the original frozen module remains on its current device.
|
||||
Note: Currently only "cpu" is supported; disk offloading is planned.
|
||||
|
||||
Returns:
|
||||
|
|
@ -174,7 +175,7 @@ def _offload_frozen_module_for_training(
|
|||
|
||||
Note:
|
||||
- Float16 weights are automatically promoted to float32 for GPU compatibility
|
||||
- Original frozen parameters are moved to CPU to reduce active VRAM usage
|
||||
- When offload_device is specified, frozen parameters are moved to free VRAM
|
||||
- Future versions will support disk-based offloading for even larger models
|
||||
|
||||
See Also:
|
||||
|
|
@ -196,7 +197,8 @@ def _offload_frozen_module_for_training(
|
|||
module.modules_to_save.default.requires_grad_(True)
|
||||
|
||||
# [TODO] Move old module to CPU - should be disk!
|
||||
module.original_module.to(device = offload_device, non_blocking = True)
|
||||
if offload_device is not None:
|
||||
module.original_module.to(device = offload_device, non_blocking = True)
|
||||
module.original_module.requires_grad_(False)
|
||||
|
||||
|
||||
|
|
@ -2286,6 +2288,10 @@ class FastLlamaModel:
|
|||
model_function = MODEL_FOR_CAUSAL_LM_MAPPING[model_config.__class__]
|
||||
IS_FALCON_H1 = model_config.model_type.startswith("falcon_h1")
|
||||
|
||||
preferred_attn_impl = (
|
||||
prefer_flex_attn_if_supported(model_function, model_config) or "eager"
|
||||
)
|
||||
|
||||
has_rope_scaling = False
|
||||
try:
|
||||
with open(inspect.getfile(model_function), "r", encoding = "utf-8") as file:
|
||||
|
|
@ -2364,7 +2370,7 @@ class FastLlamaModel:
|
|||
token = token,
|
||||
max_position_embeddings = max_position_embeddings,
|
||||
trust_remote_code = trust_remote_code,
|
||||
attn_implementation = "eager",
|
||||
attn_implementation = preferred_attn_impl,
|
||||
**kwargs,
|
||||
)
|
||||
elif not fast_inference:
|
||||
|
|
@ -2376,7 +2382,7 @@ class FastLlamaModel:
|
|||
token = token,
|
||||
max_position_embeddings = max_position_embeddings,
|
||||
trust_remote_code = trust_remote_code,
|
||||
attn_implementation = "eager",
|
||||
attn_implementation = preferred_attn_impl,
|
||||
**kwargs,
|
||||
)
|
||||
model.fast_generate = make_fast_generate_wrapper(model.generate)
|
||||
|
|
@ -2693,10 +2699,12 @@ class FastLlamaModel:
|
|||
return model
|
||||
transformers_set_seed(random_state)
|
||||
|
||||
if use_gradient_checkpointing == "unsloth":
|
||||
patch_unsloth_smart_gradient_checkpointing(
|
||||
dtype = model.get_input_embeddings().weight.dtype
|
||||
)
|
||||
# Apply gradient checkpointing with smart heuristics
|
||||
max_seq = getattr(model, "max_seq_length", 512)
|
||||
dtype = model.get_input_embeddings().weight.dtype
|
||||
use_gradient_checkpointing = apply_unsloth_gradient_checkpointing(
|
||||
use_gradient_checkpointing, max_seq, dtype
|
||||
)
|
||||
|
||||
if type(r) is not int:
|
||||
raise TypeError(f"Unsloth: Rank of {str(r)} must be an integer.")
|
||||
|
|
@ -3081,35 +3089,17 @@ class FastLlamaModel:
|
|||
print("Unsloth: Training embed_tokens in mixed precision to save VRAM")
|
||||
assert hasattr(model.get_input_embeddings(), "modules_to_save")
|
||||
|
||||
new_dtype = (
|
||||
model.get_input_embeddings().modules_to_save.default.weight.dtype
|
||||
_offload_frozen_module_for_training(
|
||||
model.get_input_embeddings(), DEVICE_TYPE_TORCH, offload_device = None
|
||||
)
|
||||
if new_dtype == torch.float16:
|
||||
# See https://github.com/unslothai/unsloth/pull/1200
|
||||
# Tesla T4 must use float32 and not float16
|
||||
new_dtype = torch.float32
|
||||
|
||||
model.get_input_embeddings().modules_to_save.default.to(
|
||||
device = DEVICE_TYPE_TORCH, dtype = new_dtype, non_blocking = True
|
||||
)
|
||||
model.get_input_embeddings().modules_to_save.default.requires_grad_(True)
|
||||
|
||||
if train_lm_head:
|
||||
print("Unsloth: Training lm_head in mixed precision to save VRAM")
|
||||
assert hasattr(model.get_output_embeddings(), "modules_to_save")
|
||||
|
||||
new_dtype = (
|
||||
model.get_output_embeddings().modules_to_save.default.weight.dtype
|
||||
_offload_frozen_module_for_training(
|
||||
model.get_output_embeddings(), DEVICE_TYPE_TORCH, offload_device = None
|
||||
)
|
||||
if new_dtype == torch.float16:
|
||||
# See https://github.com/unslothai/unsloth/pull/1200
|
||||
# Tesla T4 must use float32 and not float16
|
||||
new_dtype = torch.float32
|
||||
|
||||
model.get_output_embeddings().modules_to_save.default.to(
|
||||
device = DEVICE_TYPE_TORCH, dtype = new_dtype, non_blocking = True
|
||||
)
|
||||
model.get_output_embeddings().modules_to_save.default.requires_grad_(True)
|
||||
|
||||
# Patch tokenizer to pad to the right
|
||||
internal_model = model
|
||||
|
|
|
|||
|
|
@ -88,7 +88,7 @@ from ._utils import (
|
|||
patch_compiling_bitsandbytes,
|
||||
patch_model_and_tokenizer,
|
||||
prepare_model_for_kbit_training,
|
||||
patch_unsloth_smart_gradient_checkpointing,
|
||||
apply_unsloth_gradient_checkpointing,
|
||||
patch_compiled_autograd,
|
||||
process_vision_info,
|
||||
unsloth_compile_transformers,
|
||||
|
|
@ -559,8 +559,10 @@ class FastLanguageModel(FastLlamaModel):
|
|||
**kwargs,
|
||||
)
|
||||
|
||||
if use_gradient_checkpointing == "unsloth":
|
||||
patch_unsloth_smart_gradient_checkpointing(dtype = dtype)
|
||||
# Apply gradient checkpointing with smart heuristics
|
||||
use_gradient_checkpointing = apply_unsloth_gradient_checkpointing(
|
||||
use_gradient_checkpointing, max_seq_length, dtype
|
||||
)
|
||||
|
||||
# Check if this is local model since the tokenizer gets overwritten
|
||||
if (
|
||||
|
|
@ -1188,9 +1190,10 @@ class FastModel(FastBaseModel):
|
|||
os.environ["UNSLOTH_FORCE_FLOAT32"] = "1"
|
||||
dtype = torch.bfloat16 # Change to bfloat16 loading
|
||||
break
|
||||
# Patch gradient checkpointing
|
||||
if use_gradient_checkpointing == "unsloth":
|
||||
patch_unsloth_smart_gradient_checkpointing(dtype = dtype)
|
||||
# Apply gradient checkpointing with smart heuristics
|
||||
use_gradient_checkpointing = apply_unsloth_gradient_checkpointing(
|
||||
use_gradient_checkpointing, max_seq_length, dtype
|
||||
)
|
||||
with redirector:
|
||||
patch_loss_functions(torch_compile = False)
|
||||
model_types, supports_sdpa = unsloth_compile_transformers(
|
||||
|
|
|
|||
91
unsloth/models/rl.py
Normal file → Executable file
91
unsloth/models/rl.py
Normal file → Executable file
|
|
@ -26,6 +26,7 @@ from unsloth_zoo.compiler import create_new_function
|
|||
from unsloth_zoo.log import logger
|
||||
from unsloth_zoo.logging_utils import PatchRLStatistics
|
||||
from unsloth_zoo.rl_replacements import RL_REPLACEMENTS
|
||||
from ..device_type import DEVICE_TYPE
|
||||
from .rl_replacements import (
|
||||
RL_EXTRA_ARGS,
|
||||
RL_FUNCTIONS,
|
||||
|
|
@ -251,6 +252,7 @@ from torch.nn import functional as F
|
|||
import inspect
|
||||
from transformers import DataCollatorForSeq2Seq, DataCollatorForLanguageModeling as TransformersDataCollatorForLanguageModeling
|
||||
from transformers.training_args import ParallelMode
|
||||
from unsloth_zoo.device_type import DEVICE_TYPE, device_synchronize
|
||||
|
||||
# Wrap trainer with padding to right and enable training mode
|
||||
# Also patches W&B since multiple runs must use wandb.finish()
|
||||
|
|
@ -355,6 +357,7 @@ class Unsloth{RLConfig_name}({RLConfig_name}):
|
|||
)
|
||||
self.unsloth_logit_chunk_multiplier = unsloth_logit_chunk_multiplier
|
||||
{max_seq_length_post}
|
||||
{RLConfig_post}
|
||||
pass
|
||||
|
||||
{RLTrainer_extras}
|
||||
|
|
@ -417,7 +420,7 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"):
|
|||
try:
|
||||
trainer = eval(f"trl.trainer.{trainer_file}")
|
||||
except Exception as error:
|
||||
print(f"Unsloth: Could not import trl.trainer.{trainer_file}: {error}")
|
||||
logger.info(f"Unsloth: Could not import trl.trainer.{trainer_file}: {error}")
|
||||
return
|
||||
|
||||
# Get SFTTrainer and SFTConfig names
|
||||
|
|
@ -888,6 +891,15 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"):
|
|||
)
|
||||
extra_args += learning_rate_check
|
||||
|
||||
# Fix num_train_epochs = None causing TypeError in Trainer.__init__
|
||||
# Trainer does `args.num_train_epochs > 0` which fails when None
|
||||
if "num_train_epochs" in call_args:
|
||||
num_train_epochs_check = (
|
||||
"if num_train_epochs is None:\n"
|
||||
" num_train_epochs = 3.0 # Default to 3 epochs if None, max_steps will override\n"
|
||||
)
|
||||
extra_args += num_train_epochs_check
|
||||
|
||||
# Check if max_seq_length is NOT defined (max_length is now default)
|
||||
if "max_seq_length" not in call_args and "max_length" in call_args:
|
||||
max_seq_length_pre = """max_seq_length : Optional[int] = field(
|
||||
|
|
@ -1023,6 +1035,18 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"):
|
|||
RLConfig_extra_args = extra_args
|
||||
RLConfig_call_args = call_args
|
||||
|
||||
# TRL 0.27.0+ forces use_reentrant=False in gradient_checkpointing_kwargs.
|
||||
# Unsloth gradient checkpointing requires use_reentrant=True, so we remove
|
||||
# the setting after super().__init__() when it gets auto-applied.
|
||||
RLConfig_post = ""
|
||||
if trl_version >= Version("0.27.0") and RLConfig_name == "GRPOConfig":
|
||||
RLConfig_post = (
|
||||
" # Unsloth: Remove use_reentrant=False forced by TRL 0.27.0+\n"
|
||||
" if getattr(self, 'gradient_checkpointing_kwargs', None) is not None:\n"
|
||||
" if 'use_reentrant' in self.gradient_checkpointing_kwargs:\n"
|
||||
" del self.gradient_checkpointing_kwargs['use_reentrant']\n"
|
||||
)
|
||||
|
||||
# Patch vLLM and other functions
|
||||
RLTrainer_extras = patch_functions(
|
||||
RLTrainer, trainer_file, RLTrainer_name, all_imports, imports
|
||||
|
|
@ -1075,6 +1099,7 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"):
|
|||
RLConfig_extra_args = RLConfig_extra_args,
|
||||
RLConfig_call_args = RLConfig_call_args,
|
||||
RLConfig_kwargs = ",**kwargs"[1 if RLConfig_call_args.endswith(",") else 0 :],
|
||||
RLConfig_post = RLConfig_post,
|
||||
RLTrainer_extras = RLTrainer_extras,
|
||||
RLTrainer_post = RLTrainer_post,
|
||||
RL_pre = RL_pre,
|
||||
|
|
@ -1090,6 +1115,68 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"):
|
|||
align_logprobs_with_mask_code = align_logprobs_with_mask_code,
|
||||
)
|
||||
|
||||
if RLTrainer_name == "GRPOTrainer":
|
||||
# Base torch_compile_options shared by all device types
|
||||
base_options = """torch_compile_options = {
|
||||
"epilogue_fusion" : True,
|
||||
"max_autotune" : False,
|
||||
"shape_padding" : True,
|
||||
"trace.enabled" : False,"""
|
||||
|
||||
# Generate torch_compile_options based on device type
|
||||
if DEVICE_TYPE == "cuda":
|
||||
# CUDA-specific options (added to base options)
|
||||
new_options = (
|
||||
base_options
|
||||
+ """
|
||||
"triton.enable_persistent_tma_matmul": torch.cuda.get_device_capability()[0] >= 9,
|
||||
"cuda.cutlass_epilogue_fusion_enabled": torch.cuda.get_device_capability()[0] >= 9,
|
||||
"cuda.cutlass_tma_only": torch.cuda.get_device_capability()[0] >= 9,
|
||||
"cuda.compile_opt_level" : "-O2",
|
||||
"cuda.enable_cuda_lto" : True,
|
||||
}"""
|
||||
)
|
||||
else:
|
||||
# XPU, HIP, and other device types use base options only
|
||||
new_options = (
|
||||
base_options
|
||||
+ """
|
||||
}"""
|
||||
)
|
||||
|
||||
pattern = r"torch_compile_options\s*=\s*\{[^}]*\}"
|
||||
|
||||
RLTrainer_source = re.sub(
|
||||
pattern, new_options, RLTrainer_source, flags = re.DOTALL
|
||||
)
|
||||
|
||||
if trl_version >= Version("0.27.0"):
|
||||
peft_pattern = (
|
||||
r"\s*if is_peft_available\(\) and is_peft_model\(model\) and args\.beta != 0\.0:"
|
||||
r".*?"
|
||||
r"param\.data = param\.data\.to\(torch\.bfloat16\)"
|
||||
)
|
||||
|
||||
replacement_comment = "\n # PEFT initialization logic removed via script for trl >= 0.27.0\n"
|
||||
|
||||
RLTrainer_source = re.sub(
|
||||
peft_pattern, replacement_comment, RLTrainer_source, flags = re.DOTALL
|
||||
)
|
||||
|
||||
elif trl_version >= Version("0.26.0"):
|
||||
peft_block_pattern = (
|
||||
r"\s*if is_peft_available\(\) and isinstance\(model, PeftModel\) and peft_config is not None:"
|
||||
r".*?"
|
||||
r"param\.data = param\.data\.to\(torch\.bfloat16\)"
|
||||
)
|
||||
|
||||
RLTrainer_source = re.sub(
|
||||
peft_block_pattern,
|
||||
"\n # TRL PEFT 0.26.0 initialization logic removed on unsloth side.\n",
|
||||
RLTrainer_source,
|
||||
flags = re.DOTALL,
|
||||
)
|
||||
|
||||
if RLTrainer_name == "SFTTrainer":
|
||||
original_text = 'self._signature_columns = ["input_ids", "attention_mask", "completion_mask"]'
|
||||
new_text = 'self._signature_columns = ["input_ids", "attention_mask", "completion_mask","labels"]'
|
||||
|
|
@ -1193,6 +1280,8 @@ def patch_functions(RLTrainer, trainer_file, RLTrainer_name, all_imports, import
|
|||
init = init.replace(
|
||||
"model = self._prepare_peft_model(model, peft_config, args)\n", "pass\n"
|
||||
)
|
||||
# TRL 0.22.0+ uses prepare_peft_model as a standalone function
|
||||
init = init.replace("model = prepare_peft_model(model, peft_config, args)", "pass")
|
||||
|
||||
# Skip add_adapter("ref") for reference model computation
|
||||
# Unsloth: We comment out the "ref" adapter creation because:
|
||||
|
|
|
|||
149
unsloth/models/rl_replacements.py
Normal file → Executable file
149
unsloth/models/rl_replacements.py
Normal file → Executable file
|
|
@ -27,8 +27,10 @@ import inspect
|
|||
from collections import defaultdict
|
||||
from unsloth_zoo.rl_replacements import RL_REPLACEMENTS, left_pack_padding
|
||||
from unsloth_zoo.utils import Version
|
||||
from trl import __version__ as trl_version_raw
|
||||
from importlib.metadata import version as importlib_version
|
||||
from unsloth_zoo.log import logger
|
||||
from unsloth_zoo.device_type import device_synchronize
|
||||
import importlib.util
|
||||
from ..device_type import (
|
||||
is_hip,
|
||||
|
|
@ -56,6 +58,14 @@ torch_compile_options = {
|
|||
"triton.cudagraphs": False,
|
||||
}
|
||||
|
||||
try:
|
||||
trl_version = Version(trl_version_raw)
|
||||
except Exception:
|
||||
try:
|
||||
trl_version = Version(importlib_version("trl"))
|
||||
except Exception:
|
||||
trl_version = Version("0.0.0")
|
||||
|
||||
|
||||
# Check untrained tokens
|
||||
def sft_trainer_fix_untrained_tokens(call_args, extra_args):
|
||||
|
|
@ -75,6 +85,16 @@ def sft_trainer_fix_untrained_tokens(call_args, extra_args):
|
|||
RL_EXTRA_ARGS["sft_trainer"].append(sft_trainer_fix_untrained_tokens)
|
||||
|
||||
|
||||
# Fix top_k for GRPO vLLM.
|
||||
# https://github.com/huggingface/trl/pull/4695 with this change trl added top_k in GRPOConfig and defaults to 0
|
||||
# We don't want that since vllm's all include top_k is -1 and 0 returns an error on SamplingParams creation.
|
||||
def grpo_config_fix_vllm_top_k(old_RLTrainer_source, old_RLConfig_source):
|
||||
return "if use_vllm and (top_k is None or top_k == 0): top_k = -1\n"
|
||||
|
||||
|
||||
RL_CONFIG_CHANGES["grpo_trainer"].append(grpo_config_fix_vllm_top_k)
|
||||
|
||||
|
||||
# Remove DPO columns which might randomnly be tokenized
|
||||
def dpo_trainer_fix_columns(call_args, extra_args):
|
||||
if "model" in call_args and "train_dataset" in call_args:
|
||||
|
|
@ -236,6 +256,30 @@ def grpo_trainer__generate_single_turn(function_name, function):
|
|||
"",
|
||||
function,
|
||||
)
|
||||
|
||||
# TRL 0.24.0-0.25.1 truncation regression fix
|
||||
#
|
||||
# TRL 0.22.2-0.23.1 used smart truncation via truncate_with_protected_tokens():
|
||||
# - Tokenizes first without truncation
|
||||
# - Then truncates keeping the RIGHTMOST tokens (preserves assistant turn)
|
||||
# - Protects special tokens (image_token, vision_start/end) from removal
|
||||
#
|
||||
# TRL 0.24.0-0.25.1 removed this and passed kwargs directly to the tokenizer:
|
||||
# max_length=self.max_prompt_length, truncation=True, add_special_tokens=False
|
||||
# This causes issues because tokenizer truncation doesn't protect special tokens
|
||||
# and may not preserve the end of the prompt properly.
|
||||
#
|
||||
# TRL 0.26.2+ removed these kwargs entirely (no tokenizer-level truncation).
|
||||
#
|
||||
# Fix: Remove these kwargs so TRL 0.24.0-0.25.1 behaves like 0.26.2+ (no truncation).
|
||||
# This is a no-op for versions that don't have these kwargs (0.22.2-0.23.1, 0.26.2+).
|
||||
for pattern in [
|
||||
r'["\']?max_length["\']?\s*[:=]\s*self\.max_prompt_length\s*,\s*\n?',
|
||||
r'["\']?truncation["\']?\s*[:=]\s*True\s*,\s*\n?',
|
||||
r'["\']?add_special_tokens["\']?\s*[:=]\s*False\s*,\s*\n?',
|
||||
]:
|
||||
function = re.sub(pattern, "", function)
|
||||
|
||||
return function
|
||||
|
||||
|
||||
|
|
@ -283,7 +327,7 @@ def grpo_trainer__generate_and_score_completions(function_name, function):
|
|||
re.MULTILINE,
|
||||
)
|
||||
|
||||
replacement_text = """
|
||||
replacement_text = """
|
||||
if self.args.gradient_accumulation_steps % generate_every != 0 or (
|
||||
self.use_vllm
|
||||
):"""
|
||||
|
|
@ -365,7 +409,7 @@ def grpo_trainer__generate_and_score_completions(function_name, function):
|
|||
replacement_string = """ if "image_sizes" in prompt_inputs:
|
||||
output["image_sizes"] = prompt_inputs["image_sizes"]
|
||||
if max_left_pad is not None:
|
||||
output["max_left_pad"] = torch.tensor(prompt_ids.shape[0] * [max_left_pad]).unsqueeze(-1)
|
||||
output["max_left_pad"] = torch.tensor(prompt_ids.shape[0] * [max_left_pad]).unsqueeze(-1)
|
||||
try:
|
||||
if self.use_vllm and getattr(self, "vllm_importance_sampling_correction", False):
|
||||
output["sampling_per_token_logps"] = sampling_per_token_logps
|
||||
|
|
@ -374,6 +418,31 @@ def grpo_trainer__generate_and_score_completions(function_name, function):
|
|||
|
||||
function = function.replace(string_to_find, replacement_string)
|
||||
|
||||
# TRL 0.24.0+ extracts prompts = [x["prompt"] for x in inputs], losing metadata
|
||||
# like reasoning_effort. Inject code to store per-sample chat_template_kwargs on self.
|
||||
_metadata_extraction = (
|
||||
"\n"
|
||||
" # Unsloth: Extract per-sample chat_template_kwargs before metadata is lost\n"
|
||||
" _ct_ = getattr(self.processing_class, 'chat_template', None) or ''\n"
|
||||
" _sk_ = {'prompt', 'chosen', 'rejected', 'completion', 'messages', 'label',\n"
|
||||
" 'images', 'image', 'videos', 'video', 'audios', 'audio'}\n"
|
||||
" self._unsloth_batch_chat_kwargs = []\n"
|
||||
" for _inp_ in inputs:\n"
|
||||
" _kw_ = {}\n"
|
||||
" if isinstance(_inp_, dict):\n"
|
||||
" for _k_ in _inp_.keys() - _sk_:\n"
|
||||
" if _k_ in _ct_ and isinstance(_inp_[_k_], str):\n"
|
||||
" _kw_[_k_] = _inp_[_k_]\n"
|
||||
" self._unsloth_batch_chat_kwargs.append(_kw_)\n"
|
||||
)
|
||||
# Insert after: prompts = [x["prompt"] for x in inputs]
|
||||
_target_line = 'prompts = [x["prompt"] for x in inputs]'
|
||||
if _target_line in function:
|
||||
function = function.replace(
|
||||
_target_line,
|
||||
_target_line + _metadata_extraction,
|
||||
)
|
||||
|
||||
# This path is for TRL 0.24.0 images is a variable exclusive to this version
|
||||
string_to_find = """ if images is not None:
|
||||
output["num_images"] = num_images"""
|
||||
|
|
@ -381,7 +450,7 @@ def grpo_trainer__generate_and_score_completions(function_name, function):
|
|||
replacement_string = """ if images is not None:
|
||||
output["num_images"] = num_images
|
||||
if max_left_pad is not None:
|
||||
output["max_left_pad"] = torch.tensor(prompt_ids.shape[0] * [max_left_pad]).unsqueeze(-1)
|
||||
output["max_left_pad"] = torch.tensor(prompt_ids.shape[0] * [max_left_pad]).unsqueeze(-1)
|
||||
try:
|
||||
if self.use_vllm and getattr(self, "vllm_importance_sampling_correction", False):
|
||||
output["sampling_per_token_logps"] = sampling_per_token_logps
|
||||
|
|
@ -390,6 +459,17 @@ def grpo_trainer__generate_and_score_completions(function_name, function):
|
|||
|
||||
function = function.replace(string_to_find, replacement_string)
|
||||
|
||||
if trl_version >= Version("0.25.0"):
|
||||
# We replace the call using 'completions' with one using 'completions_text'
|
||||
string_to_find = " rewards_per_func = self._calculate_rewards(inputs, prompts, completions, completion_ids_list)"
|
||||
replacement_string = (
|
||||
" if images is not None:\n"
|
||||
" rewards_per_func = self._calculate_rewards(inputs, prompts_text, completions_text, completion_ids_list)\n"
|
||||
" else:\n"
|
||||
" rewards_per_func = self._calculate_rewards(inputs, prompts, completions, completion_ids_list)"
|
||||
)
|
||||
function = function.replace(string_to_find, replacement_string)
|
||||
|
||||
if "wake_up()" not in function:
|
||||
# Sleep functionality has been added to trl in v0.23.0. We do not want to redo this.
|
||||
# https://github.com/huggingface/trl/commit/edbe8234bc7e528f72ac76607de9d3e4753e2709
|
||||
|
|
@ -434,9 +514,10 @@ def grpo_trainer_fix_maybe_apply_chat_template(function_name, function):
|
|||
_chat_template_ = getattr(self.processing_class, "chat_template", None)
|
||||
if _chat_template_ is None: _chat_template_ = ""
|
||||
_supported_keys_ = set(("prompt", "chosen", "rejected", "completion", "messages", "label"))
|
||||
_batch_chat_kwargs_ = getattr(self, "_unsloth_batch_chat_kwargs", None)
|
||||
|
||||
prompts_text = []
|
||||
for _example_ in __INPUTS__REPLACEMENT__:
|
||||
for _idx_, _example_ in enumerate(__INPUTS__REPLACEMENT__):
|
||||
_tokenizer_kwargs_ = {}
|
||||
if type(_example_) is not dict:
|
||||
_example_ = {"prompt": _example_}
|
||||
|
|
@ -446,6 +527,10 @@ def grpo_trainer_fix_maybe_apply_chat_template(function_name, function):
|
|||
v = _example_[k]
|
||||
if type(v) is str:
|
||||
_tokenizer_kwargs_[k] = v
|
||||
if _batch_chat_kwargs_ is not None and _idx_ < len(_batch_chat_kwargs_):
|
||||
for _bk_, _bv_ in _batch_chat_kwargs_[_idx_].items():
|
||||
if _bk_ not in _tokenizer_kwargs_:
|
||||
_tokenizer_kwargs_[_bk_] = _bv_
|
||||
_x_ = maybe_apply_chat_template(_example_, self.processing_class, **_tokenizer_kwargs_)["prompt"]
|
||||
prompts_text.append(_x_)
|
||||
"""
|
||||
|
|
@ -771,7 +856,7 @@ def grpo_trainer__get_per_token_logps_and_entropies(function_name, function):
|
|||
)
|
||||
# This is needed to avoid race conditions with GPT OSS offload_embbed=True
|
||||
# However, it seems that this line does not slow down or disrupt models.
|
||||
torch.cuda.synchronize()
|
||||
device_synchronize()
|
||||
all_logprobs_list.append(logprobs_chunk)
|
||||
logprobs = torch.cat(all_logprobs_list, dim = 0)
|
||||
entropies = None
|
||||
|
|
@ -914,7 +999,7 @@ def grpo_trainer_compute_loss(function_name, function):
|
|||
|
||||
max_left_pad = inputs.get("max_left_pad", 0)
|
||||
if per_token_logps is not None:
|
||||
loss, completion_length, mean_kl, delta, flat_is_ratio = (
|
||||
loss, completion_length, mean_kl, delta, flat_is_ratio, coef_1 = (
|
||||
grpo_compute_loss_slow(
|
||||
ref_logps,
|
||||
per_token_logps,
|
||||
|
|
@ -944,7 +1029,7 @@ def grpo_trainer_compute_loss(function_name, function):
|
|||
)
|
||||
else:
|
||||
if hasattr(self.args, "loss_type"):
|
||||
loss, completion_length, mean_kl, delta, flat_is_ratio = (
|
||||
loss, completion_length, mean_kl, delta, flat_is_ratio, coef_1 = (
|
||||
grpo_accumulated_loss(
|
||||
trainer = self,
|
||||
input_ids = _input_ids,
|
||||
|
|
@ -976,7 +1061,7 @@ def grpo_trainer_compute_loss(function_name, function):
|
|||
)
|
||||
else:
|
||||
# to ensure backwards compatibility with trl 0.15.2 and maybe even 0.17
|
||||
loss, completion_length, mean_kl = grpo_accumulated_loss(
|
||||
loss, completion_length, mean_kl, coef_1 = grpo_accumulated_loss(
|
||||
trainer = self,
|
||||
input_ids = _input_ids,
|
||||
logits_to_keep = logits_to_keep,
|
||||
|
|
@ -991,7 +1076,6 @@ def grpo_trainer_compute_loss(function_name, function):
|
|||
logit_scale_divide = logit_scale_divide,
|
||||
attention_mask = attention_mask,
|
||||
)
|
||||
|
||||
if "train" in self._metrics:
|
||||
mode = "eval" if self.control.should_evaluate else "train"
|
||||
self._metrics[mode]["completion_length"].append(completion_length.item())
|
||||
|
|
@ -1053,6 +1137,53 @@ def grpo_trainer_compute_loss(function_name, function):
|
|||
.item()
|
||||
)
|
||||
|
||||
completion_token_count = completion_mask.sum().clamp(min = 1.0)
|
||||
|
||||
def masked_batch_mean(x):
|
||||
if x.shape[1] == 1: # when importance_sampling_level == "sequence"
|
||||
return x.mean()
|
||||
else:
|
||||
return (x * completion_mask).sum() / completion_token_count
|
||||
|
||||
if advantages.dim() == 1:
|
||||
advantages = advantages.unsqueeze(1)
|
||||
|
||||
if self.loss_type in ["grpo", "bnpo", "dr_grpo", "dapo"]:
|
||||
# Compute the clipped probability ratios
|
||||
is_low_clipped = (coef_1 < 1 - self.epsilon_low) & (advantages < 0)
|
||||
is_high_clipped = (coef_1 > 1 + self.epsilon_high) & (advantages > 0)
|
||||
is_region_clipped = is_low_clipped | is_high_clipped
|
||||
|
||||
low_clip = masked_batch_mean(is_low_clipped.float())
|
||||
high_clip = masked_batch_mean(is_high_clipped.float())
|
||||
clip_ratio = masked_batch_mean(is_region_clipped.float())
|
||||
|
||||
gathered_low_clip = self.accelerator.gather(low_clip)
|
||||
self._metrics[mode]["clip_ratio/low_mean"].append(
|
||||
gathered_low_clip.nanmean().item()
|
||||
)
|
||||
self._metrics[mode]["clip_ratio/low_min"].append(
|
||||
nanmin(gathered_low_clip).item()
|
||||
)
|
||||
gathered_high_clip = self.accelerator.gather(high_clip)
|
||||
self._metrics[mode]["clip_ratio/high_mean"].append(
|
||||
gathered_high_clip.nanmean().item()
|
||||
)
|
||||
self._metrics[mode]["clip_ratio/high_max"].append(
|
||||
nanmax(gathered_high_clip).item()
|
||||
)
|
||||
gathered_clip_ratio = self.accelerator.gather(clip_ratio)
|
||||
self._metrics[mode]["clip_ratio/region_mean"].append(
|
||||
gathered_clip_ratio.nanmean().item()
|
||||
)
|
||||
elif self.loss_type == "cispo":
|
||||
is_cispo_clipped = (coef_1 > self.epsilon_high) & (advantages > 0)
|
||||
cispo_clip_ratio = masked_batch_mean(is_cispo_clipped.float())
|
||||
gathered_cispo_clip_ratio = self.accelerator.gather(cispo_clip_ratio)
|
||||
self._metrics[mode]["cispo_clip_ratio"].append(
|
||||
gathered_cispo_clip_ratio.nanmean().item()
|
||||
)
|
||||
|
||||
return loss
|
||||
|
||||
function = inspect.getsource(compute_loss)
|
||||
|
|
|
|||
|
|
@ -517,9 +517,23 @@ class FastBaseModel:
|
|||
correct_dtype = None
|
||||
|
||||
# Stop SDPA for some archs like Pixtral / Mistral3
|
||||
flex_attn_impl = None
|
||||
if auto_config is None:
|
||||
auto_config = AutoConfig.from_pretrained(
|
||||
model_name,
|
||||
token = token,
|
||||
trust_remote_code = trust_remote_code,
|
||||
)
|
||||
try:
|
||||
model_class = auto_model._model_mapping[auto_config.__class__]
|
||||
except Exception:
|
||||
model_class = None
|
||||
flex_attn_impl = prefer_flex_attn_if_supported(model_class, auto_config)
|
||||
|
||||
default_attn_impl = "flex_attention" if flex_attn_impl else "sdpa"
|
||||
if not ("attn_implementation" in kwargs):
|
||||
kwargs["attn_implementation"] = "sdpa"
|
||||
if not supports_sdpa:
|
||||
kwargs["attn_implementation"] = default_attn_impl
|
||||
if not supports_sdpa and kwargs.get("attn_implementation") == "sdpa":
|
||||
if os.environ.get("UNSLOTH_ENABLE_FLEX_ATTENTION", "0") == "0":
|
||||
print(
|
||||
f"Unsloth: {model_type_arch.title()} does not support SDPA - switching to fast eager."
|
||||
|
|
@ -651,12 +665,19 @@ class FastBaseModel:
|
|||
|
||||
kwargs = add_dtype_kwargs(torch_dtype, kwargs)
|
||||
|
||||
model_config = AutoConfig.from_pretrained(
|
||||
model_name,
|
||||
token = token,
|
||||
attn_implementation = "sdpa" if supports_sdpa else "eager",
|
||||
trust_remote_code = trust_remote_code,
|
||||
)
|
||||
config_attn_impl = kwargs.get("attn_implementation", None)
|
||||
if config_attn_impl is None:
|
||||
config_attn_impl = "sdpa" if supports_sdpa else "eager"
|
||||
if auto_config is None:
|
||||
auto_config = AutoConfig.from_pretrained(
|
||||
model_name,
|
||||
token = token,
|
||||
trust_remote_code = trust_remote_code,
|
||||
)
|
||||
setattr(auto_config, "_attn_implementation", config_attn_impl)
|
||||
if hasattr(auto_config, "attn_implementation"):
|
||||
setattr(auto_config, "attn_implementation", config_attn_impl)
|
||||
model_config = auto_config
|
||||
verify_fp8_support_if_applicable(model_config)
|
||||
|
||||
raise_handler = RaiseUninitialized()
|
||||
|
|
|
|||
|
|
@ -602,7 +602,10 @@ def load_correct_tokenizer(
|
|||
old_chat_template = getattr(tokenizer, "chat_template", None)
|
||||
|
||||
# Ignore mistral type models since they don't have an add_generation_prompt
|
||||
if "mistral" in str(getattr(tokenizer, "name_or_path", "")).lower():
|
||||
if any(
|
||||
s in str(getattr(tokenizer, "name_or_path", "")).lower()
|
||||
for s in ["mistral", "qwen3guard"]
|
||||
):
|
||||
chat_template = old_chat_template
|
||||
|
||||
# Also check Llama-2 old style models
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue