* Fix torchvision compatibility check for source builds and future torch versions
The torchvision version check raised a hard ImportError for custom/source-built
PyTorch installations (e.g. AMD ROCm from source with +git* suffixes), even when
the actual build was functional. This also silently skipped any torch version
not already in the hardcoded table, giving no warning at all for future releases.
Changes:
- Detect custom/source builds by checking the raw version string's local
identifier against known standard prefixes (cu, rocm, cpu, xpu). Our custom
Version() strips local identifiers via regex, so detection must happen on the
raw string before parsing.
- Downgrade to a warning (instead of ImportError) for custom/source builds,
since their version numbers may not follow standard PyPI release pairings.
- Add formula-based inference for future torch versions not yet in the table.
The torch->torchvision minor version formula (torch 2.x -> tv 0.(x+15)) has
held for every release from torch 2.0 through 2.9. For formula-predicted
versions, mismatches produce a warning rather than a hard error.
- Add UNSLOTH_SKIP_TORCHVISION_CHECK=1 env var to skip the check entirely.
- Wrap importlib_version and Version calls in try/except so broken metadata
never crashes the import.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: stricter regex, case insensitivity, pre-release detection
Fixes three edge cases found during review:
1. Regex precision: cu/xpu now require a trailing digit (cu\d, xpu\d) to
avoid false negatives on suffixes like "+custom_build" that happen to
start with "cu". cpu/xpu match as exact strings only.
2. Case insensitivity: added re.IGNORECASE so "+ROCM6.3" and "+CPU" are
correctly recognized as standard builds rather than custom ones.
3. Pre-release detection: nightly/dev/alpha/beta/rc builds with standard
CUDA/ROCm suffixes (e.g. "2.7.0.dev20250301+cu124") now produce a
warning instead of a hard ImportError. These builds commonly have
version mismatches that are expected during development.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address PR review comments: fullmatch, env var casing, torchvision pre-release
1. Switch re.match to re.fullmatch for the custom build regex so the
entire local identifier must match. Fixes false negatives where
suffixes like +cu124_custom were misclassified as standard because
re.match only checked the start of the string.
2. Use .lower() for the UNSLOTH_SKIP_TORCHVISION_CHECK env var so
any casing of "true" / "TRUE" / etc. is accepted.
3. Check torchvision_version_raw for pre-release tags in addition to
torch_version_raw, so a stable torch paired with a nightly
torchvision (e.g. 0.23.0.dev...) also gets a warning instead of
a hard ImportError.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
vLLM's distributed module (device_communicators) crashes with std::bad_alloc
when imported on SM100 GPUs (B200/B100/Blackwell) with torch < 2.9.0.
This adds an early check that runs before vLLM is imported, providing a
helpful error message instead of a cryptic C++ exception.
The check:
1. Detects if vLLM is installed
2. Checks if torch version is < 2.9.0
3. Checks if any GPU is SM100 (Blackwell)
4. If all conditions met, raises RuntimeError with clear upgrade instructions
* Add TRL truncation regression and metadata loss fixes
Fix 1: TRL 0.24.0-0.25.1 right-truncation regression
- These versions pass max_length=self.max_prompt_length and truncation=True
to the tokenizer, which right-truncates prompts and strips the assistant
turn suffix
- Use regex to remove these kwargs from the generated code
Fix 3: Metadata loss for chat_template_kwargs
- TRL 0.24.0+ extracts prompts = [x["prompt"] for x in inputs], losing metadata
like reasoning_effort
- Inject code to store per-sample chat_template_kwargs on self before extraction
- Preserve these kwargs in prompts_text generation for all TRL versions
Tested with TRL versions 0.22.2, 0.23.1, 0.24.0, 0.25.1, 0.26.2, and 0.27.1.
* Update Fix 1 comment with detailed TRL version behavior explanation
Expand the comment for the TRL 0.24.0-0.25.1 truncation regression fix
to clarify what each TRL version does:
- TRL 0.22.2-0.23.1: Uses truncate_with_protected_tokens() for smart
truncation that preserves rightmost tokens and protects special tokens
- TRL 0.24.0-0.25.1: Removed smart truncation, passes kwargs directly
to tokenizer (max_length, truncation=True, add_special_tokens=False)
- TRL 0.26.2+: Removed these kwargs entirely
The fix removes these problematic kwargs so 0.24.0-0.25.1 behaves like
0.26.2+ (no tokenizer-level truncation).
---------
Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>
When users pass `num_train_epochs=None` to GRPOConfig (relying on
max_steps to control training duration), Trainer.__init__ fails with:
TypeError: '>' not supported between instances of 'NoneType' and 'int'
This happens because transformers.Trainer does `args.num_train_epochs > 0`
in its __init__ which fails when the value is None.
This fix converts None to 3.0 (the default) before Trainer initialization.
The actual training duration is still controlled by max_steps since it
takes precedence when both are set.
Example that now works:
```python
config = GRPOConfig(
num_train_epochs=None, # Previously caused TypeError
max_steps=500, # This controls actual duration
...
)
```
* [fix] Vision GRPO string prompts and OpenEnv async compatibility
- Guard prepare_multimodal_messages in GRPO trainer to skip processing
when prompts are pre-templated strings. Notebooks that pre-apply
apply_chat_template() produce strings with image tokens already
embedded; calling prepare_multimodal_messages on those crashes with
TypeError.
- Apply nest_asyncio when OpenEnv EnvClient exposes async reset/step,
so scripts using run_until_complete() wrappers work in all contexts.
- Add wrapper to call patch_torchcodec_audio_decoder() from unsloth_zoo
for AudioDecoder dict-compatibility.
* Add apply_chat_template guard for pre-templated string prompts in Vision GRPO
When notebooks pre-apply apply_chat_template, prompts become strings.
The existing guard skips prepare_multimodal_messages for strings. This
adds a second guard to skip apply_chat_template in the forward_kwargs
block, using prompts directly as prompts_text instead. Covers both
TRL 0.25.x (no tools param) and TRL 0.26.2+ (with tools=self.tools).
Non-matching replacements silently pass for older TRL versions.
* Add TRL 0.25.1 single-line variant for apply_chat_template guard
TRL 0.25.1 uses single-line formatting for apply_chat_template:
apply_chat_template({"prompt": prompt}, ...)["prompt"]
While TRL 0.26.2+ uses multi-line formatting:
apply_chat_template(
{"prompt": prompt}, ...
)["prompt"]
Add both variants to ensure full backwards compatibility.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix TRL 0.27.0 GRPO compatibility and PEFT model handling
- Remove use_reentrant=False from gradient_checkpointing_kwargs for TRL 0.27.0+
TRL 0.27.0 auto-sets use_reentrant=False in GRPOConfig.__post_init__, but
Unsloth gradient checkpointing requires use_reentrant=True. This adds a
post-init cleanup that removes the setting when present.
- Handle prepare_peft_model standalone function pattern for TRL 0.22.0+
TRL changed from self._prepare_peft_model() method to prepare_peft_model()
standalone function. Both patterns are now bypassed to let Unsloth handle
PEFT model preparation.
Tested with TRL versions 0.22.2, 0.23.1, 0.24.0, 0.25.1, 0.26.2, and 0.27.1.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* reduce code duplication
* address reviewer feedback: keep original function name
- Keep original function name `_offload_frozen_module_for_training`
- Make `offload_device` parameter Optional (can be None)
- Keep original error handling (return None for missing modules_to_save)
- Maintain code deduplication by reusing the helper function
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Use standard gradient checkpointing for small sequence lengths
When max_seq_length < 512, the overhead of gradient offloading in
gc="unsloth" mode is not worth it. Benchmarks on B200 show:
| seq_len | gc=unsloth | gc=True | Difference |
|---------|------------|----------|------------|
| 256 | 6,803 t/s | 6,993 t/s| +2.8% |
| 384 | 9,889 t/s | 9,963 t/s| +0.7% |
| 512 | 13,151 t/s | 13,092 t/s| -0.4% |
| 1024 | 26,662 t/s | 25,094 t/s| -5.9% |
The crossover point is around seq_len 384-512. For sequences shorter
than 512, we now automatically use standard gradient checkpointing
instead of the custom offloading implementation.
Additionally, when user explicitly sets use_gradient_checkpointing to
True or False in get_peft_model, it now correctly overrides any
previous "unsloth" patching from from_pretrained. This ensures
consistent behavior regardless of the order of function calls.
Updated in three locations:
- FastLlamaModel.get_peft_model (llama.py)
- FastLanguageModel.from_pretrained (loader.py)
- FastModel.from_pretrained (loader.py)
* Refactor: extract gradient checkpointing heuristic into utility function
Addresses code review feedback to reduce duplication. The gradient
checkpointing heuristic logic was duplicated in 3 places:
- FastLlamaModel.get_peft_model (llama.py)
- FastLanguageModel.from_pretrained (loader.py)
- FastModel.from_pretrained (loader.py)
Created apply_unsloth_gradient_checkpointing() utility function in
_utils.py that handles:
- Heuristic: seq < 512 falls back to standard gc
- Explicit True/False overrides unpatch previous patching
- Returns the effective use_gradient_checkpointing value
Net reduction of ~6 lines while improving maintainability.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix for intel devices
* Refactor torch_compile_options to use base options with device-specific extensions
- Extract common options into base_options shared by all device types
- CUDA devices get additional CUDA-specific options
- XPU, HIP, and other devices use base options only
- Reduces code duplication and improves maintainability
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: danielhanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
