Merge branch 'main' into pr/3673
This commit is contained in:
Daniel Han
commit
0deca31071
9 changed files with 74 additions and 35 deletions
|
|
@ -1,6 +1,6 @@
|
|||
repos:
|
||||
- repo: https://github.com/astral-sh/ruff-pre-commit
|
||||
rev: v0.14.6
|
||||
rev: v0.14.8
|
||||
hooks:
|
||||
- id: ruff
|
||||
args:
|
||||
|
|
|
|||
11
README.md
11
README.md
|
|
@ -23,14 +23,15 @@ Notebooks are beginner friendly. Read our [guide](https://docs.unsloth.ai/get-st
|
|||
| Model | Free Notebooks | Performance | Memory use |
|
||||
|-----------|---------|--------|----------|
|
||||
| **gpt-oss (20B)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-Fine-tuning.ipynb) | 1.5x faster | 70% less |
|
||||
| **Qwen3 (14B)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(14B)-Reasoning-Conversational.ipynb) | 2x faster | 70% less |
|
||||
| **Mistral Ministral 3 (3B)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Ministral_3_VL_(3B)_Vision.ipynb) | 1.5x faster | 60% less |
|
||||
| **gpt-oss (20B): GRPO** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/gpt-oss-(20B)-GRPO.ipynb) | 2x faster | 80% less |
|
||||
| **Qwen3: Advanced GRPO** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_(4B)-GRPO.ipynb) | 2x faster | 50% less |
|
||||
| **Qwen3-VL (8B): GSPO** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision-GRPO.ipynb) | 1.5x faster | 80% less |
|
||||
| **Qwen3-VL (8B)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_VL_(8B)-Vision.ipynb) | 2x faster | 50% less |
|
||||
| **Gemma 3 (270M)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(270M).ipynb) | 1.7x faster | 60% less |
|
||||
| **Gemma 3n (4B)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3N_(4B)-Conversational.ipynb) | 1.5x faster | 50% less |
|
||||
| **DeepSeek-OCR (3B)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Deepseek_OCR_(3B).ipynb) | 1.5x faster | 30% less |
|
||||
| **Llama 3.1 (8B)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-Alpaca.ipynb) | 2x faster | 70% less |
|
||||
| **Llama 3.1 (8B) Alpaca** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-Alpaca.ipynb) | 2x faster | 70% less |
|
||||
| **Llama 3.2 Conversational** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(1B_and_3B)-Conversational.ipynb) | 2x faster | 70% less |
|
||||
| **Orpheus-TTS (3B)** | [▶️ Start for free](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Orpheus_(3B)-TTS.ipynb) | 1.5x faster | 50% less |
|
||||
|
||||
- See all our notebooks for: [Kaggle](https://github.com/unslothai/notebooks?tab=readme-ov-file#-kaggle-notebooks), [GRPO](https://docs.unsloth.ai/get-started/unsloth-notebooks#grpo-reasoning-rl-notebooks), **[TTS](https://docs.unsloth.ai/get-started/unsloth-notebooks#text-to-speech-tts-notebooks)** & [Vision](https://docs.unsloth.ai/get-started/unsloth-notebooks#vision-multimodal-notebooks)
|
||||
|
|
@ -50,17 +51,19 @@ Use our official [Unsloth Docker image](https://hub.docker.com/r/unsloth/unsloth
|
|||
For RTX 50x, B200, 6000 GPUs: `pip install unsloth`. Read our [Blackwell Guide](https://docs.unsloth.ai/basics/training-llms-with-blackwell-rtx-50-series-and-unsloth) and [DGX Spark Guide](https://docs.unsloth.ai/new/fine-tuning-llms-with-nvidia-dgx-spark-and-unsloth) for more details.
|
||||
|
||||
## 🦥 Unsloth News
|
||||
- **Ministral 3** by Mistral: Run Ministral 3 or fine-tune with our vision or RL sodoku notebook. [Guide](https://docs.unsloth.ai/new/ministral-3) • [Notebooks](https://docs.unsloth.ai/new/ministral-3#fine-tuningb)
|
||||
- **500K Context Fine-tuning**: Training a 20B model with >500K token context windows is now possible on a single 80GB GPU. [Blog](https://docs.unsloth.ai/new/500k-context-length-fine-tuning)
|
||||
- **FP8 Reinforcement Learning**: You can now do FP8 GRPO on consumer GPUs. [Blog](https://docs.unsloth.ai/new/fp8-reinforcement-learning) • [Notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen3_8B_FP8_GRPO.ipynb)
|
||||
- **DeepSeek-OCR**: Fine-tune to improve language understanding by 89%. [Guide](https://docs.unsloth.ai/new/deepseek-ocr-run-and-fine-tune) • [Notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Deepseek_OCR_(3B).ipynb)
|
||||
- **Docker**: Use Unsloth with no setup & environment issues with our new image. [Guide](https://docs.unsloth.ai/new/how-to-train-llms-with-unsloth-and-docker) • [Docker image](https://hub.docker.com/r/unsloth/unsloth)
|
||||
- **gpt-oss RL**: Introducing the fastest possible inference for gpt-oss RL! [Read blog](https://docs.unsloth.ai/new/gpt-oss-reinforcement-learning)
|
||||
- **Vision RL**: You can now train VLMs with GRPO or GSPO in Unsloth! [Read guide](https://docs.unsloth.ai/new/vision-reinforcement-learning-vlm-rl)
|
||||
- **Quantization-Aware Training**: We collabed with Pytorch, recovering ~70% accuracy. [Read blog](https://docs.unsloth.ai/new/quantization-aware-training-qat)
|
||||
- **gpt-oss** by OpenAI: Read our [Unsloth Flex Attention](https://docs.unsloth.ai/new/long-context-gpt-oss-training) blog and [gpt-oss Guide](https://docs.unsloth.ai/basics/gpt-oss). 20B works on 14GB VRAM. 120B on 65GB.
|
||||
|
||||
<details>
|
||||
<summary>Click for more news</summary>
|
||||
|
||||
- **Quantization-Aware Training**: We collabed with Pytorch, recovering ~70% accuracy. [Read blog](https://docs.unsloth.ai/new/quantization-aware-training-qat)
|
||||
- **Memory-efficient RL**: We're introducing even better RL. Our new kernels & algos allows faster RL with 50% less VRAM & 10× more context. [Read blog](https://docs.unsloth.ai/new/memory-efficient-rl)
|
||||
- **Gemma 3n** by Google: [Read Blog](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune). We [uploaded GGUFs, 4-bit models](https://huggingface.co/collections/unsloth/gemma-3n-685d3874830e49e1c93f9339).
|
||||
- **[Text-to-Speech (TTS)](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning)** is now supported, including `sesame/csm-1b` and STT `openai/whisper-large-v3`.
|
||||
|
|
|
|||
|
|
@ -55,12 +55,12 @@ huggingfacenotorch = [
|
|||
"huggingface_hub>=0.34.0",
|
||||
"hf_transfer",
|
||||
"diffusers",
|
||||
"transformers>=4.51.3,!=4.52.0,!=4.52.1,!=4.52.2,!=4.52.3,!=4.53.0,!=4.54.0,!=4.55.0,!=4.55.1,!=4.57.0,<=4.57.2",
|
||||
"transformers>=4.51.3,!=4.52.0,!=4.52.1,!=4.52.2,!=4.52.3,!=4.53.0,!=4.54.0,!=4.55.0,!=4.55.1,!=4.57.0,<=4.57.3",
|
||||
"trl>=0.18.2,!=0.19.0,<=0.24.0",
|
||||
]
|
||||
huggingface = [
|
||||
"unsloth[huggingfacenotorch]",
|
||||
"unsloth_zoo>=2025.11.6",
|
||||
"unsloth_zoo>=2025.12.3",
|
||||
"torchvision",
|
||||
"unsloth[triton]",
|
||||
]
|
||||
|
|
@ -285,10 +285,16 @@ cu130onlytorch290 = [
|
|||
"xformers @ https://download.pytorch.org/whl/cu130/xformers-0.0.33.post1-cp39-abi3-win_amd64.whl ; (sys_platform == 'win32')",
|
||||
]
|
||||
cu126onlytorch291 = [
|
||||
"xformers @ https://download.pytorch.org/whl/cu126/xformers-0.0.33.post2-cp39-abi3-manylinux_2_28_x86_64.whl ; ('linux' in sys_platform)",
|
||||
"xformers @ https://download.pytorch.org/whl/cu126/xformers-0.0.33.post2-cp39-abi3-win_amd64.whl ; (sys_platform == 'win32')",
|
||||
]
|
||||
cu128onlytorch291 = [
|
||||
"xformers @ https://download.pytorch.org/whl/cu128/xformers-0.0.33.post2-cp39-abi3-manylinux_2_28_x86_64.whl ; ('linux' in sys_platform)",
|
||||
"xformers @ https://download.pytorch.org/whl/cu128/xformers-0.0.33.post2-cp39-abi3-win_amd64.whl ; (sys_platform == 'win32')",
|
||||
]
|
||||
cu130onlytorch291 = [
|
||||
"xformers @ https://download.pytorch.org/whl/cu130/xformers-0.0.33.post2-cp39-abi3-manylinux_2_28_x86_64.whl ; ('linux' in sys_platform)",
|
||||
"xformers @ https://download.pytorch.org/whl/cu130/xformers-0.0.33.post2-cp39-abi3-win_amd64.whl ; (sys_platform == 'win32')",
|
||||
]
|
||||
cu118 = [
|
||||
"unsloth[huggingface]",
|
||||
|
|
@ -517,10 +523,10 @@ colab-ampere-torch220 = [
|
|||
"flash-attn>=2.6.3 ; ('linux' in sys_platform)",
|
||||
]
|
||||
colab-new = [
|
||||
"unsloth_zoo>=2025.11.6",
|
||||
"unsloth_zoo>=2025.12.3",
|
||||
"packaging",
|
||||
"tyro",
|
||||
"transformers>=4.51.3,!=4.52.0,!=4.52.1,!=4.52.2,!=4.52.3,!=4.53.0,!=4.54.0,!=4.55.0,!=4.55.1,!=4.57.0,<=4.57.2",
|
||||
"transformers>=4.51.3,!=4.52.0,!=4.52.1,!=4.52.2,!=4.52.3,!=4.53.0,!=4.54.0,!=4.55.0,!=4.55.1,!=4.57.0,<=4.57.3",
|
||||
"datasets>=3.4.1,!=4.0.*,!=4.1.0,<4.4.0",
|
||||
"sentencepiece>=0.2.0",
|
||||
"tqdm",
|
||||
|
|
|
|||
|
|
@ -64,7 +64,7 @@ from importlib.metadata import PackageNotFoundError
|
|||
# Check for unsloth_zoo
|
||||
try:
|
||||
unsloth_zoo_version = importlib_version("unsloth_zoo")
|
||||
if Version(unsloth_zoo_version) < Version("2025.11.2"):
|
||||
if Version(unsloth_zoo_version) < Version("2025.12.3"):
|
||||
print(
|
||||
"Unsloth: Please update Unsloth and Unsloth-Zoo to the latest version!\n"
|
||||
"Do this via `pip install --upgrade --force-reinstall --no-cache-dir --no-deps unsloth unsloth_zoo`"
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@
|
|||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
__version__ = "2025.11.6"
|
||||
__version__ = "2025.12.1"
|
||||
|
||||
__all__ = [
|
||||
"SUPPORTS_BFLOAT16",
|
||||
|
|
@ -761,6 +761,13 @@ elif DEVICE_TYPE == "xpu":
|
|||
|
||||
# =============================================
|
||||
# Get Xformers
|
||||
# Silence xformers CUDA mismatch warnings before import
|
||||
try:
|
||||
_xformers_logger = logging.getLogger("xformers")
|
||||
_xformers_logger.setLevel(logging.ERROR)
|
||||
del _xformers_logger
|
||||
except:
|
||||
pass
|
||||
try:
|
||||
from xformers import __version__ as xformers_version
|
||||
|
||||
|
|
@ -835,10 +842,11 @@ except ModuleNotFoundError:
|
|||
xformers_attention = None
|
||||
xformers_version = None
|
||||
except Exception as e:
|
||||
print(
|
||||
"========\nSwitching to PyTorch attention since your Xformers is broken.\n========\n"
|
||||
)
|
||||
print(str(e))
|
||||
if os.environ.get("UNSLOTH_ENABLE_LOGGING", "0") != "0":
|
||||
print(
|
||||
"========\nSwitching to PyTorch attention since your Xformers is broken.\n========\n"
|
||||
)
|
||||
print(str(e))
|
||||
xformers = None
|
||||
xformers_attention = None
|
||||
xformers_version = None
|
||||
|
|
|
|||
|
|
@ -164,6 +164,7 @@ def GemmaModel_fast_forward_inference(
|
|||
past_key_values,
|
||||
position_ids,
|
||||
attention_mask = None,
|
||||
**kwargs,
|
||||
):
|
||||
out_weights = tuple(
|
||||
torch.empty_like(
|
||||
|
|
|
|||
|
|
@ -426,6 +426,7 @@ def Gemma2Model_fast_forward_inference(
|
|||
past_key_values,
|
||||
position_ids,
|
||||
attention_mask = None,
|
||||
**kwargs,
|
||||
):
|
||||
out_weights = tuple(
|
||||
torch.empty_like(
|
||||
|
|
|
|||
|
|
@ -468,13 +468,24 @@ def _patch_trl_rl_trainers(trainer_file = "grpo_trainer"):
|
|||
" args.fp16 = False\n"
|
||||
" args.bf16 = False\n"
|
||||
" os.environ['ACCELERATE_MIXED_PRECISION'] = 'no'\n"
|
||||
" if hasattr(args, 'mixed_precision'): args.mixed_precision = 'no'\n"
|
||||
" # args.mixed_precision is a new argument which needs to be set now\n"
|
||||
"elif (not use_bf16 and not use_fp16) and mixed_precision_dtype == 'float32':\n"
|
||||
" # Mixed precision training\n"
|
||||
" args.fp16 = float16\n"
|
||||
" args.bf16 = not float16\n"
|
||||
" os.environ['ACCELERATE_MIXED_PRECISION'] = 'fp16' if float16 else 'bf16'\n"
|
||||
" if hasattr(args, 'mixed_precision'): args.mixed_precision = 'fp16' if float16 else 'bf16'\n"
|
||||
" # args.mixed_precision is a new argument which needs to be set now\n"
|
||||
"elif mixed_precision_dtype == 'bfloat16':\n"
|
||||
" # Both False since bfloat16 full finetuning doesn't do any autocasting.\n"
|
||||
" args.fp16 = False\n"
|
||||
" args.bf16 = False\n"
|
||||
" os.environ['ACCELERATE_MIXED_PRECISION'] = 'no'\n"
|
||||
" if hasattr(args, 'mixed_precision'): args.mixed_precision = 'no'\n"
|
||||
" # args.mixed_precision is a new argument which needs to be set now\n"
|
||||
"\n"
|
||||
)
|
||||
"elif mixed_precision_dtype == 'bfloat16':\n" " # Both False since bfloat16 full finetuning doesn't do any autocasting.\n" " args.fp16 = False\n" " args.bf16 = False\n" " os.environ['ACCELERATE_MIXED_PRECISION'] = 'no'\n"
|
||||
extra_args += mixed_precision
|
||||
|
||||
# Check if per_device_eval_batch_size (default 8) bigger than bsz
|
||||
|
|
|
|||
|
|
@ -672,25 +672,34 @@ class FastBaseModel:
|
|||
model.fast_generate = model.generate
|
||||
model.fast_generate_batches = error_out_no_vllm
|
||||
if offload_embedding:
|
||||
embed_tokens = model.get_input_embeddings()
|
||||
nbytes = embed_tokens.weight.numel() * embed_tokens.weight.itemsize
|
||||
ngb = round(nbytes / 1024 / 1024 / 1024, 2)
|
||||
print(f"Unsloth: Offloading embeddings to RAM to save {ngb} GB.")
|
||||
embed_tokens.to("cpu")
|
||||
if bool(
|
||||
os.environ.get("WSL_DISTRO_NAME") or os.environ.get("WSL_INTEROP")
|
||||
):
|
||||
# WSL doesn't work with offloaded embeddings
|
||||
pass
|
||||
elif os.name == "nt":
|
||||
# Windows doesn't work with offloaded embeddings
|
||||
pass
|
||||
else:
|
||||
embed_tokens = model.get_input_embeddings()
|
||||
nbytes = embed_tokens.weight.numel() * embed_tokens.weight.itemsize
|
||||
ngb = round(nbytes / 1024 / 1024 / 1024, 2)
|
||||
print(f"Unsloth: Offloading embeddings to RAM to save {ngb} GB.")
|
||||
embed_tokens.to("cpu")
|
||||
|
||||
# Add hooks to move inputs to CPU and back to CUDA
|
||||
# [TODO] Doesn't seem to work!
|
||||
# def pre_hook(module, args):
|
||||
# args[0]._old_device = args[0].device
|
||||
# return (args[0].to("cpu", non_blocking = True))
|
||||
# def post_hook(module, args, output):
|
||||
# old_device = getattr(args[0], "_old_device", "cuda")
|
||||
# return output.to(old_device, non_blocking = True)
|
||||
# embed_tokens.register_forward_pre_hook(pre_hook, prepend = True)
|
||||
# embed_tokens.register_forward_hook (post_hook, prepend = True)
|
||||
# Must free GPU memory otherwise will not free!
|
||||
torch.cuda.empty_cache()
|
||||
gc.collect()
|
||||
# Add hooks to move inputs to CPU and back to CUDA
|
||||
# [TODO] Doesn't seem to work!
|
||||
# def pre_hook(module, args):
|
||||
# args[0]._old_device = args[0].device
|
||||
# return (args[0].to("cpu", non_blocking = True))
|
||||
# def post_hook(module, args, output):
|
||||
# old_device = getattr(args[0], "_old_device", "cuda")
|
||||
# return output.to(old_device, non_blocking = True)
|
||||
# embed_tokens.register_forward_pre_hook(pre_hook, prepend = True)
|
||||
# embed_tokens.register_forward_hook (post_hook, prepend = True)
|
||||
# Must free GPU memory otherwise will not free!
|
||||
torch.cuda.empty_cache()
|
||||
gc.collect()
|
||||
else:
|
||||
from unsloth_zoo.vllm_utils import (
|
||||
load_vllm,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue