The temp_O output buffer was allocated as (1, bsz, hidden_size) but
torch.matmul expects (bsz, 1, hidden_size) when bsz > 1. With the
wrong shape, PyTorch silently resizes the output tensor, producing a
UserWarning about deprecated resize behavior that will become an error
in a future release.
This only affects the Pattern A allocation path (when attention_size !=
hidden_size, e.g. Mistral Nemo 12b). The Pattern B alias path already
gets the correct shape from temp_QA[1]. When bsz == 1, the shapes are
identical so there is no behavioral change for single-sequence
inference.
Files changed:
- unsloth/models/llama.py
- unsloth/models/gemma2.py
- unsloth/models/granite.py
- unsloth/models/cohere.py
- unsloth/models/qwen3.py
- unsloth/models/falcon_h1.py
* Suppress FBGEMM CUTLASS "Arch conditional MMA" stdout spam on Blackwell GPUs
On Blackwell GPUs (B200/B100, SM100), FBGEMM's f8f8bf16_blockwise kernel
is hardcoded to cutlass::arch::Sm90 with no SM100 code path. When
test_has_fbgemm() probes this kernel, it fires 2304 "ERROR : Arch
conditional MMA instruction used without targeting appropriate compute
capability" lines before aborting and returning zeros.
The existing HidePrintMessage filter on sys.stderr (line 109) does not
catch these because CUDA device-side printf writes to stdout fd 1 at the
C level, bypassing Python's sys.stdout/sys.stderr entirely.
Fix: add suppress_cuda_printf() context manager in import_fixes.py that
redirects fd 1 and fd 2 to /dev/null at the OS level, with
torch.cuda.synchronize() and libc fflush before restoring. Wrap the
test_has_fbgemm() call in fp8.py with this context manager.
Tested on B200 with fbgemm-gpu-genai 1.4.0+cu130 and 1.5.0+cu130:
- Before: 2304 warning lines on every import
- After: 0 warning lines
- UNSLOTH_HAS_FBGEMM correctly set to 0 (Triton fallback works)
- Works with both UNSLOTH_ENABLE_LOGGING=0 and =1
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard _libc init and fflush to prevent fd leak on failure
---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-253.us-east-2.compute.internal>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix VLM processor load degradation and vLLM CUDA version detection
vision.py - Fix VLM processor load for issue #4085:
- Before loading the processor, scan local config files and strip the
_Unsloth_Patched_ prefix. AutoProcessor.from_pretrained silently
degrades to a text-only tokenizer instead of raising an exception
when it encounters the unrecognized class name, so the existing
get_auto_processor fallback never triggers. Sanitizing the configs
before loading fixes backwards compat for old corrupted saves.
- After loading, detect when AutoProcessor returned a text-only
tokenizer for a VLM model (has no image_processor attribute) and
trigger the manual fallback constructor.
import_fixes.py - Fix vLLM CUDA version mismatch detection:
- _is_broken_vllm_error now also matches CUDA shared library errors
(libcudart, libcublas, libnvrtc) with "cannot open shared object
file". Previously it only matched errors containing "vllm._c" in
the message text, which missed cases where the error message was
about the missing CUDA library itself (e.g. vllm built for CUDA 12
on a CUDA 13 system).
- New _get_vllm_cuda_mismatch_message function extracts the CUDA
version from the error, compares to the system CUDA version via
torch.version.cuda, and returns a targeted install command using
the correct GitHub releases wheel URL.
- disable_broken_vllm uses the targeted message when a CUDA mismatch
is detected, falling back to the existing generic message otherwise.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-253.us-east-2.compute.internal>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add `datasets` metadata support to model cards
Add an optional `datasets` parameter to all save/push functions so users
can specify which datasets were used for training. The metadata is set
via `ModelCard.data.datasets` for standard paths and via
`metadata_update` for GGUF and generic save paths.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix datasets metadata for existing repos, add token, improve errors
- Add metadata_update fallback in create_huggingface_repo and
upload_to_huggingface so datasets metadata is set even when the
repo already exists (previously only worked on first creation).
- Pass token=token to all metadata_update calls so they work
without a global HF login.
- Replace silent except:pass with logger.warning_once for
metadata failures so users know if something went wrong.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix generic datasets metadata repo resolution for PR #4076
* Fix create_huggingface_repo username resolution for PR #4076
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
* FP8 per tensor quant support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix security-regression fallout in chat templates and PDL patching
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop security regression test files from PR scope
* Apply suggestion from @danielhanchen
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Guard optional vLLM imports when extension is broken
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove vLLM import guard tests from PR scope
* Block broken vLLM imports like causal_conv1d
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Raise ImportError for stable torchvision mismatches
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove torchvision compatibility tests from PR scope
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
When using device_map='balanced' with multiple GPUs, the labels tensor
may reside on a different device than the logits/losses tensors. This
causes a RuntimeError at the masked_fill_ call in the chunked
cross-entropy forward path.
Fix: explicitly move labels to the same device as logits at the start
of Fast_CrossEntropyLoss.forward(). This is a no-op on single-GPU
setups.
Fixes#4041
* Auto-configure AMDGPU_ASIC_ID_TABLE_PATH on ROCm startup
* Remove ROCm fd2 amdgpu.ids noise filter wrappers
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use PyPI bitsandbytes for amd extra to avoid malformed wheel URL
* Add amd-preview extra for bitsandbytes continuous wheel channel
* Keep amd extra on bitsandbytes>=0.49.1 and remove amd-preview
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Wrap unsloth_zoo import with HIP amdgpu.ids filter
* Refactor ROCm ids filter helpers for readability
* Rename ROCm ids filter helper and annotate call sites
* Remove obsolete amdgpu ids filter alias
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
MI355X (gfx950) has the same 1024-thread workgroup limit as MI300X (gfx942),
but was missing from is_cdna(), causing all Triton kernels to use num_warps=32
(2048 threads) instead of 16 (1024 threads), resulting in OutOfResources crash.
Tested on: 8x AMD Instinct MI355X (gfx950), ROCm 7.1
* Suppress HIP libdrm stderr noise in causal_conv1d probe
* Broaden HIP libdrm stderr suppression for early ROCm startup
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
* Improve HIP GPU name reporting in startup banner
* Drop MI300X arch suffix in banner name
* Normalize _utils.py file mode
* Simplify FA2 fallback text and filter AMD ids noise
* Strip trailing GPU arch suffix via regex
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use gfx lookup default and normalize Ryzen AI naming
* Remove name-path Ryzen AI normalization
* Expand ROCm gfx map to full documented GPU name aliases
* Simplify HIP fallback naming to AMD gfx token
* Remove Ryzen Al torch_name normalization
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Improve HIP GPU name reporting in startup banner
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Handle broken causal_conv1d import at runtime
Add a startup import-time probe for causal_conv1d and disable the fast path when the shared library is ABI broken. This keeps Falcon H1/model loading resilient without requiring env flags.
- Add disable_broken_causal_conv1d in import_fixes.
- Invoke it early from unsloth/__init__ during package init.
- Make Falcon H1 optional imports in loader and models/__init__ soft-fail instead of failing hard.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Enforce unavailable semantics for broken causal_conv1d
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove Falcon H1 import swallowing
* Restore optional Falcon H1 import guard
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove causal_conv1d regression tests
* Trim FA2 fallback messaging
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* convert print to logger
* Print but cleaner
* Hide model on multiple devices
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix typo
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix typo transfomers -> transformers, revert MoE message change
* Update MoE detection message to show num_experts and target_modules
* Fix llama-cli path in save info message
* target_parameters warning for moe
* fix should_convert_module for llm_int8_skip_modules
* fix should_convert_module for llm_int8_skip_modules
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Logging filters
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* negation
* remove should_convert_module patch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
* Fix warmup_ratio deprecation warning for transformers >= 5.0
In transformers 5.0, warmup_ratio is deprecated in favor of
warmup_steps which now accepts float values (< 1 = ratio,
>= 1 = absolute steps).
The compiler now conditionally sets warmup_steps=0.1 on
transformers >= 5.0 (same semantics as warmup_ratio=0.1) and
keeps warmup_ratio=0.1 on older versions where warmup_steps
only accepts int.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Inject token_type_ids for Gemma3 multimodal training on transformers 5.x
In transformers 5.x, create_causal_mask_mapping() raises ValueError when
is_training=True and token_type_ids is None. When doing text-only SFT on
Gemma3 4B (a multimodal model), the dataset_utils detection for
_needs_token_type_ids can miss because:
- The model is wrapped in PeftModel, so type(model).__module__ points to
peft.peft_model instead of transformers
- The processing_class is a tokenizer (not Gemma3Processor), so the
fallback MRO check resolves to a module without create_causal_mask_mapping
This adds a fallback in _unsloth_pre_compute_loss that injects
token_type_ids=zeros when:
1. token_type_ids is not already in inputs
2. The inner model config has model_type "gemma3"
3. The model's module has create_causal_mask_mapping (transformers 5.x)
4. The model is in training mode
On transformers 4.x, create_causal_mask_mapping does not exist so this
check is inert.
Depends on: unslothai/unsloth-zoo#488
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* FP8: Load model on-the-fly in vLLM
**Summary:** Existing support for `load_in_fp8=True` performs
an offline quantization when loading the initial model.
This is no longer necessary as of vllm==0.12.0 (after
https://github.com/vllm-project/vllm/pull/23014), where we
can quantize the model on-the-fly when we load it:
```
llm = LLM(
...
hf_overrides={
"quantization_config_dict_str": json.dumps(torchao_config),
},
)
```
**Note:** Needs https://github.com/unslothai/unsloth-zoo/pull/380
**Test Plan:**
https://gist.github.com/andrewor14/5b85119fae46845d07b608d420907423
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix on-the-fly FP8: always check mapper first, fallback to on-the-fly
The original implementation bypasses the FP8 mapper entirely for
vllm >= 0.12.0, meaning models like Llama-3.2-1B-Instruct and Qwen3-8B
that have pre-quantized FP8-Block/FP8 checkpoints would never use them.
This fixes the priority order:
1. Mapper has a pre-quantized model -> use it (always)
2. Mapper has no match + vllm >= 0.12.0 -> on-the-fly FP8 via torchao
3. Mapper has no match + vllm < 0.12.0 -> offline quantization
Changes:
- loader_utils.py: Move vllm >= 0.12.0 check after mapper lookups
- loader.py: Set load_in_fp8=False when mapper resolves to a
pre-quantized model to prevent double quantization
Tested on B200 with Llama-3.2-1B-Instruct and Qwen3-8B. Corrected code
produces results matching baseline (pre-quantized path preserved).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
* convert print to logger
* Print but cleaner
* Hide model on multiple devices
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix typo
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix typo transfomers -> transformers, revert MoE message change
* Update MoE detection message to show num_experts and target_modules
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
* Fix#3397: Prevent trainer tokenization hang with safe num_proc
* Fix#3397: Add missing import sys for Windows-safe tokenization
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Consolidate with existing num_proc guard in dataset_utils.py
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
* Fix EmbeddingGemma float16 NaN by adding gemma3_text to FORCE_FLOAT32 and SDPA lists
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Inject model reference for dynamic token_type_ids detection in SFTTrainer
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Suppress vLLM v1 executor sleep/wake log messages
Add HideLoggingMessage filters for vllm.v1.executor.abstract logger to
suppress repetitive sleep/wake INFO and WARNING messages that spam training
output when UNSLOTH_VLLM_STANDBY is enabled. The existing filter at line 275
handles the legacy vllm.executor.executor_base path; this adds coverage for
the v1 engine path used by vllm 0.11+.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Silence peft target_parameters RuntimeWarning for MoE models
Wrap _get_peft_model calls with warnings.catch_warnings() to suppress
the "target_parameters were set but no parameter was matched" warning.
This fires on MoE models where expert layers use nn.Parameter naming
that peft warns about but handles correctly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Strip the "anihilate"/"annihilate" warning block from compiled trainer
source so it does not fire when Unsloth auto-enables padding-free mode
with batch size 1 (the common single-GPU case).
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
* Fix dtype mismatch in fp16 + 4-bit/8-bit LoRA training
Two fixes for training with dtype=torch.float16 and load_in_4bit=True:
1. fast_lora.py: fast_dequantize() returns tensors in quant_state.dtype
(typically bfloat16 or float32), but activations may be float16. The
subsequent matmul/addmm operations require matching dtypes. Add dtype
casts after each fast_dequantize() call in LoRA_MLP.backward and
LoRA_QKV.backward (5 locations total).
2. rl.py: TRL unconditionally casts trainable parameters to bfloat16 in
the peft init block. When training with fp16=True, this causes
GradScaler to crash since it requires float32 parameters. Make the
cast conditional -- use float32 when fp16 is enabled, bfloat16
otherwise. This is a no-op for GRPOTrainer (whose peft init block is
already removed by the existing regex), but fixes SFTTrainer and
other TRL trainers.
Tested with Llama-3.2-1B-Instruct 4-bit on both fp16 and bf16 training.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix fp16 + 4-bit LoRA: thread correct_dtype through post_patch
Root cause: fast_dequantize returns tensors in quant_state.dtype, which
for pre-quantized models is bfloat16 (from config.json). The post_patch
methods in llama/gemma/gemma2 call patch_model_and_tokenizer without
passing correct_dtype, so quant_state.dtype is never overridden to match
the user's requested dtype. This causes a dtype mismatch crash in the
backward pass when training with dtype=torch.float16.
Fix: pass the user's dtype from from_pretrained through post_patch to
patch_model_and_tokenizer as correct_dtype, matching the pattern already
used by vision.py.
Revert the 5 symptom-level dtype casts in fast_lora.py (upW, gateW, QW,
KW, VW) since they are no longer needed with quant_state.dtype properly
set at the source.
Tested: fp16+4bit and bf16+4bit Llama-3.2-1B-Instruct 15-step SFT runs
both complete successfully with similar losses (~1.558 vs ~1.563).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove TRL's unconditional bfloat16 cast instead of patching the dtype
TRL 0.26.0+ hardcodes `param.data.to(torch.bfloat16)` for all trainable
params in quantized models, citing the QLoRA paper recommendation. This
is wrong: it ignores the user's requested dtype and breaks GradScaler
when fp16=True. The block exists in sft_trainer, grpo_trainer,
rloo_trainer, and reward_trainer (not dpo_trainer).
Previous fix patched the cast to be dtype-conditional. This commit
replaces the entire guard `if getattr(model, "is_loaded_in_4bit", ...)
or getattr(model, "is_loaded_in_8bit", ...):` with `if False:` to
disable the block entirely. Unsloth already handles adapter dtype via
patch_model_and_tokenizer, making TRL's cast both unnecessary and
harmful.
For GRPOTrainer the enclosing peft init block is already removed by
the regex above, making this a no-op for GRPO.
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix trainer compilation failures from trl.experimental thin wrappers
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix OOM from prepare_model_for_kbit_training overwriting peft_config patching
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
TRL 0.22.x checks _is_vlm (model type) instead of _is_vision_dataset
(dataset content, added in 0.25.1+) in _set_signature_columns_if_needed.
When _is_vlm=True (e.g. Gemma3), signature columns are set to vision-only
["messages","prompt","completion","images"], which has zero overlap with
tokenized text columns [input_ids, labels, attention_mask, ...], causing
a ValueError.
Fix: expand the VLM branch signature columns to include both vision and
text column names. Extra columns not present in the dataset are harmlessly
ignored by _remove_unused_columns (it only raises when zero columns match).
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
* Patch before compile?
* Fix notebook compatibility for transformers 4.57.6 and TRL 0.22-0.27
Fixes several notebook failures discovered during testing all 125
notebooks with transformers==4.57.6 + tRL 0.22.2 and TRL 0.27.1.
Warning suppression (import_fixes.py):
- Suppress torch 2.9+ pin_memory/is_pinned device deprecation warnings
- Suppress cuda.cudart/cuda.nvrtc module deprecation FutureWarning
- Filter vllm "Level is deprecated" stderr noise
- Filter PydanticSerializationUnexpectedValue warnings
- Filter Triton "df: No such file" stderr noise
VLM tokenizer loading (vision.py):
- Add _construct_vlm_processor_fallback() for models where
AutoProcessor.from_pretrained fails (e.g., ERNIE 4.5 VL, LFM2.5-VL)
- Wrap processor loading in try/except with fallback to manual
construction from separate image_processor + tokenizer components
- Add fallback to AutoTokenizer/PreTrainedTokenizerFast when tokenizer
loading or patching fails
TRL 0.27.1 trainer compatibility (trainer.py):
- Add _resolve_trainer_params() to handle thin wrapper trainers that
only have def __init__(self, *args, **kwargs) (e.g., ORPOTrainer
in TRL 0.27.1) by walking MRO for real parameter signature
VLM _is_vlm detection (rl.py):
- Replace blanket _is_vlm=False override with model-architecture-based
detection that checks vision_config or ForConditionalGeneration class
name, fixing VLM training when bare tokenizer is passed as
processing_class
ModernBERT SDPA compatibility (loader.py, sentence_transformer.py):
- Add "modernbert" to DISABLE_SDPA_MODEL_NAMES to avoid stride
alignment issues with torch.compile backward pass
- Add DISABLE_SDPA check for sentence transformer models
Other fixes (_utils.py):
- Suppress false uninitialized weight warnings for VLM
multi_modal_projector.layer_norm
Tested: 92/125 notebooks pass with TRL 0.22.2, 94/125 with TRL 0.27.1.
Remaining failures are infra (missing FFmpeg, network timeouts, GPU
arch) not code bugs.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix KTO shape mismatch on TRL 0.27.2+ and truncation alignment
- Patch KTO get_batch_logps to auto-align logits and labels when Unsloth
model forward truncates input_ids beyond max_seq_length. TRL 0.27.2
changed _process_tokens to only truncate completions (not prompts), so
sequences with long prompts exceed max_seq_length and trigger model-side
truncation. The original ValueError is replaced with min-length alignment.
- Also truncate attention_mask in LlamaModel forward when input_ids are
truncated to max_seq_length, preventing shape mismatches in attention.
- Widen except clause in rl_replacements.py openenv import from
`except ImportError` to `except (ImportError, NameError, Exception)` to
handle vllm SamplingParams NameError in TRL 0.27.2.
* Fix TRL 0.26+ thin wrapper resolution, enable ModernBERT SDPA, clean up warning filters
TRL 0.26+ thin wrapper resolution (rl.py):
- Filter _-prefixed private imports when discovering Trainer/Config classes
- Look up Config in separate *_config.py module when not found in trainer module
- Detect thin wrappers (<1000 chars source) and resolve to experimental parent
via MRO walk; use resolved module for imports and create_new_function
- Enables all 15 trainers to patch successfully (was 5/15 before)
ModernBERT SDPA (loader.py):
- Remove "modernbert" from DISABLE_SDPA_MODEL_NAMES
- SDPA works correctly for both classification and sentence transformers
- Verified: 88.9% accuracy on emotion classification, correct domain-specific
embeddings after sentence transformer fine-tuning
Warning filter cleanup (import_fixes.py):
- Remove cuda.cudart/cuda.nvrtc FutureWarning filters (no such warnings
exist in torch 2.9.1+; proactive suppression is unnecessary)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove multi_modal_projector.layer_norm from uninitialized weight guard
The LFM2.5-VL projector LayerNorm is properly initialized by
transformers and does not need to be excluded from the uninitialized
weight check. The original exclusion was added as a workaround but is
no longer needed after the upstream fix.
* Add transformers 5.0 compat: rope_theta helper, config-as-dim detection, BatchEncoding guard, try/except for TRL trainer source, push_to_hub_token compiler fix
- llama.py: Add _get_rope_theta() helper handling both config.rope_theta and rope_parameters dict
- llama.py: Handle BatchEncoding in unsloth_fast_generate (transformers 5.0+ returns BatchEncoding from apply_chat_template)
- gemma.py: Detect config passed as dim arg in GemmaFixedRotaryEmbedding
- tokenizer_utils.py: Add try/except for TRL trainer getsource in patch_sft_trainer_tokenizer
- rl_replacements.py: Add compiler fix replacing bare pop("push_to_hub_token") with pop(..., None)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use trl.experimental string check instead of char-count heuristic for thin wrapper detection
The <1000 / >1000 char threshold was fragile -- XPOConfig's parent is only
994 chars and would be skipped. All thin wrappers in TRL 0.26+ contain
"trl.experimental" in their deprecation warning, while no real trainer or
config class does, making it a reliable detection marker.
* Move DISABLE_SDPA_MODEL_NAMES import to module level in sentence_transformer
The function-level import was redundant since loader.py is already imported
at module level. Move it to the existing loader import line.
---------
Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Add `inputs_embeds` parameter to `_fast_prepare_inputs_for_generation` so
`model.generate(inputs_embeds=...)` works with Unsloth-patched models.
Changes:
- Add `inputs_embeds=None` to function signature (fixes HF inspect check)
- Track `use_inputs_embeds` flag: True when inputs_embeds provided and no cache
- Conditionally return inputs_embeds on first step, input_ids on subsequent steps
- Handle input_ids being None/empty for batch size and device extraction
- Add attention_mask None-guard before slicing
Fixes: https://github.com/unslothai/unsloth/issues/3798
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: siddhudonda <siddhudonda@users.noreply.github.com>
