* [transformers] [v5] remove unused hybridcache (#3910)
* remote unused hybridcache
* cleanup
* Fix top_k on trl GRPO
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add torch compile options for GRPOTrainer
* Update CUDA settings based on device capability
* Add triton persistent TMA matmul condition
* Fix syntax for triton.enable_persistent_tma_matmul
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update rl.py
* Update rl.py
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Guard torch.compile on ROCm when triton_key missing
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update unsloth/import_fixes.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten ROCm Triton import handling
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Rachel Li <rachelliqx07@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* add FastSentenceTransformer
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gemini code review suggestions
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* unsloth-zoo patch only fixed usage for XLMRobertaForMaskedLM, this is a fix for XLMRobertaModel
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* refactor do_lower_case
* add some comments
* force disable FP8 loading
* refactor pooling detection, add missing pooling types
* add save_pretrained_merged method which gets modules and config
* fix _save_pretrained_merged
* rename read_pooling_mode, load modules instead of hard-coding em
* comment
* revert save_pretrained_merged change
* propagate trust_remote_code properly
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add super hacky mpnet patch from hell
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* refactor _load_modules, add for_inference to from_pretrained, add transformers 5 code for mpnet, add distilbert patches
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add ModernBert
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* deberta-v2 support (provisional), fix remote_code
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add generic add_pooling_layer logic
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix for missing config
* add push_to_hub_merged
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* edit messages, throw exception if no HF token
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix device_map mismatch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add comments, move import, other suggestions by Datta0
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* re-add adapter removal to save_pretrained_merged, but if saving to folder which had adapters before, leave them
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add unsloth branding to save_pretrained_merged
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* propagate dtype to internal module when loading for inference
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix mpnet gradient checkpointing for torch >= 2.9
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* same thing for transformers 5, oops =)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix FastSentenceTransformer performance: 6x speedup via torch.compile + SDPA
The original implementation was 31% slower than naive SentenceTransformer due to
conflicting decorators from Unsloth's auto-compiler (@torch.compile on attention
modules but @torch.compiler.disable on sub-modules).
Changes:
- Add fast encoder path that bypasses Unsloth patching for encoder models
- Use native torch.compile with mode="reduce-overhead" for 6x speedup
- Auto-detect and enable SDPA for models that support it (BERT, RoBERTa, etc.)
- Change defaults: load_in_16bit=True, load_in_4bit=False (16-bit is optimal)
- Change default: use_gradient_checkpointing=False (conflicts with torch.compile)
- Add UNSLOTH_COMPILE_DISABLE=1 env var to fall back to old path if needed
Supported encoder types: mpnet, bert, distilbert, roberta, xlm-roberta, albert, electra
Benchmark results (BS=32, seq_len=128):
- Naive 16-bit LoRA: 13-50ms per iter
- Unsloth 16-bit LoRA: 2-9ms per iter (5.4x-6.7x faster)
- Memory usage: 61MB-1.3GB (even largest model fits easily)
Note: 4-bit + torch.compile has a PyTorch bug (pytorch/pytorch#90665).
4-bit is also 1.7-1.9x slower than 16-bit due to dequantization overhead,
so 16-bit is recommended for these small encoder models anyway.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use Unsloth's prepare_model_for_kbit_training for consistency
Changed from peft.prepare_model_for_kbit_training to
unsloth.models._utils.prepare_model_for_kbit_training.
Unsloth's version provides:
- Float32 mixed precision upcasting for LoRA layers
- Better numerical stability
- Consistency with rest of Unsloth codebase
* Use relative imports and add float16 machine support
- Changed absolute import to relative: from ._utils import prepare_model_for_kbit_training
- Added SUPPORTS_BFLOAT16 import for proper dtype detection
- Handle devices that don't support bfloat16 by falling back to float16
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add save_pretrained_torchao
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add auto-compile for torch.compile based on training step breakeven analysis
Changes:
- Change default compile_mode from "reduce-overhead" to "default" since CUDA
Graphs (used by reduce-overhead) is incompatible with PEFT/LoRA
- Add _estimate_compile_threshold() to calculate minimum steps needed for
torch.compile to be beneficial based on model parameter count
- Add _apply_torch_compile() helper with accelerate unwrap_model bug workaround
- Defer torch.compile application to trainer initialization time so we can
check max_steps against the breakeven threshold
- Patch SentenceTransformerTrainer to auto-apply compile when max_steps
exceeds the calculated threshold
Breakeven thresholds (with 1.2x safety margin):
- 22M params (MiniLM): ~1388 steps
- 110M params (mpnet): ~242 steps
- 335M params (snowflake): ~203 steps
This ensures torch.compile warmup cost is only paid when training is long
enough to benefit from the speedup.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* do QAT preparation for fast path
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix double loading model, thanks Etherl
* do mpnet gradient checkpoint patch if gc is enabled
* remove distilbert patches from mpnet fix
* sanity check on model params, thanks Etherl
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add save_pretrained_gguf, thanks Etherl
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Refine compile threshold estimation for sentence transformers
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com>
* add int8 weight-only QAT scheme, add test, fix tests for current torchao version
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* change quantization to PerAxis
* lambda =/
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* add torchao messages, remove group_size from int8
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* raise exception on missing torchao
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* touch up the torchao imports
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
ModulesToSaveWrapper was removed from peft.tuners.tuners_utils in PEFT
0.16.0. The class has been available in peft.utils.other since at least
PEFT 0.7.1, which is the minimum version Unsloth requires.
This fixes the ImportError when using PEFT >= 0.16.0.
- Fix Kaggle misclassification by prioritizing filesystem markers over env vars
- Preserve telemetry pings when statistics is explicitly provided
- Replace bare except with except Exception
- Minor cleanup based on automated review feedback
Fixed Codex regression: keep snapshot_download pings for explicit statistics values; detection only runs when statistics is None. Also replaced bare except.
Problem: Kaggle notebook environments can expose both KAGGLE_* and COLAB_* environment keys. _get_statistics currently checks COLAB_ before KAGGLE_, causing Kaggle sessions to be labeled colab/colabpro.
Prefer filesystem markers (e.g. /kaggle/working, /content + /opt/colab) before env-key heuristics, then fall back to the existing env-key checks. This avoids misclassification when providers leak overlapping env vars.
Kaggle test notebook: https://www.kaggle.com/code/hnxnq07/kaggle-stats-gathering-test
- Fix test file: use return_tokenized instead of return_tensors
- Fix test file: use text_dataset instead of undefined dataset variable
- Move parameter validation to constructor (fail fast on invalid params)
- Add labels field in tokenized output for causal LM training
- Add empty file handling with clear error message
- Add tests for constructor validation and labels field
This PR fixes the "Arch conditional MMA instruction used without targeting
appropriate compute capability. Aborting." errors that occur when using
FBGEMM on Blackwell GPUs (B200/B100, SM100).
Changes:
- Add stderr filters in import_fixes.py for CUTLASS/FBGEMM MMA errors
- Add warning filters for various deprecation messages
- Update check_fbgemm_gpu_version() to disable FBGEMM instead of raising
an error when old versions are detected
- Update test_has_fbgemm() in fp8.py to catch broader CUTLASS/CUDA errors
and gracefully fall back to Triton kernels
- Update loader_utils.py to disable FBGEMM instead of raising ValueError
for old fbgemm_gpu versions
The key behavior change is that FBGEMM errors no longer crash the script.
Instead, FBGEMM is disabled and Triton kernels are used automatically.
This allows Unsloth to work on SM100 GPUs where CUTLASS SM90 kernels fail,
and also gracefully handles old FBGEMM versions.
