- Introduce `AvailableVariables` for displaying variables linked to configs.
- Implement `ChipInput` for dynamic value management in category and subcategory dialogs.
- Add `AuxVariableBadges` to aux nodes for displaying variable references.
- Update inline components with comboboxes for better user experience.
- Replace badges and manual inputs with streamlined reusable components.
* Silence peft target_parameters RuntimeWarning for MoE models
Wrap _get_peft_model calls with warnings.catch_warnings() to suppress
the "target_parameters were set but no parameter was matched" warning.
This fires on MoE models where expert layers use nn.Parameter naming
that peft warns about but handles correctly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Strip the "anihilate"/"annihilate" warning block from compiled trainer
source so it does not fire when Unsloth auto-enables padding-free mode
with batch size 1 (the common single-GPU case).
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
* Fix dtype mismatch in fp16 + 4-bit/8-bit LoRA training
Two fixes for training with dtype=torch.float16 and load_in_4bit=True:
1. fast_lora.py: fast_dequantize() returns tensors in quant_state.dtype
(typically bfloat16 or float32), but activations may be float16. The
subsequent matmul/addmm operations require matching dtypes. Add dtype
casts after each fast_dequantize() call in LoRA_MLP.backward and
LoRA_QKV.backward (5 locations total).
2. rl.py: TRL unconditionally casts trainable parameters to bfloat16 in
the peft init block. When training with fp16=True, this causes
GradScaler to crash since it requires float32 parameters. Make the
cast conditional -- use float32 when fp16 is enabled, bfloat16
otherwise. This is a no-op for GRPOTrainer (whose peft init block is
already removed by the existing regex), but fixes SFTTrainer and
other TRL trainers.
Tested with Llama-3.2-1B-Instruct 4-bit on both fp16 and bf16 training.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix fp16 + 4-bit LoRA: thread correct_dtype through post_patch
Root cause: fast_dequantize returns tensors in quant_state.dtype, which
for pre-quantized models is bfloat16 (from config.json). The post_patch
methods in llama/gemma/gemma2 call patch_model_and_tokenizer without
passing correct_dtype, so quant_state.dtype is never overridden to match
the user's requested dtype. This causes a dtype mismatch crash in the
backward pass when training with dtype=torch.float16.
Fix: pass the user's dtype from from_pretrained through post_patch to
patch_model_and_tokenizer as correct_dtype, matching the pattern already
used by vision.py.
Revert the 5 symptom-level dtype casts in fast_lora.py (upW, gateW, QW,
KW, VW) since they are no longer needed with quant_state.dtype properly
set at the source.
Tested: fp16+4bit and bf16+4bit Llama-3.2-1B-Instruct 15-step SFT runs
both complete successfully with similar losses (~1.558 vs ~1.563).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove TRL's unconditional bfloat16 cast instead of patching the dtype
TRL 0.26.0+ hardcodes `param.data.to(torch.bfloat16)` for all trainable
params in quantized models, citing the QLoRA paper recommendation. This
is wrong: it ignores the user's requested dtype and breaks GradScaler
when fp16=True. The block exists in sft_trainer, grpo_trainer,
rloo_trainer, and reward_trainer (not dpo_trainer).
Previous fix patched the cast to be dtype-conditional. This commit
replaces the entire guard `if getattr(model, "is_loaded_in_4bit", ...)
or getattr(model, "is_loaded_in_8bit", ...):` with `if False:` to
disable the block entirely. Unsloth already handles adapter dtype via
patch_model_and_tokenizer, making TRL's cast both unnecessary and
harmful.
For GRPOTrainer the enclosing peft init block is already removed by
the regex above, making this a no-op for GRPO.
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix trainer compilation failures from trl.experimental thin wrappers
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix OOM from prepare_model_for_kbit_training overwriting peft_config patching
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
TRL 0.22.x checks _is_vlm (model type) instead of _is_vision_dataset
(dataset content, added in 0.25.1+) in _set_signature_columns_if_needed.
When _is_vlm=True (e.g. Gemma3), signature columns are set to vision-only
["messages","prompt","completion","images"], which has zero overlap with
tokenized text columns [input_ids, labels, attention_mask, ...], causing
a ValueError.
Fix: expand the VLM branch signature columns to include both vision and
text column names. Extra columns not present in the dataset are harmlessly
ignored by _remove_unused_columns (it only raises when zero columns match).
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
* Patch before compile?
* Fix notebook compatibility for transformers 4.57.6 and TRL 0.22-0.27
Fixes several notebook failures discovered during testing all 125
notebooks with transformers==4.57.6 + tRL 0.22.2 and TRL 0.27.1.
Warning suppression (import_fixes.py):
- Suppress torch 2.9+ pin_memory/is_pinned device deprecation warnings
- Suppress cuda.cudart/cuda.nvrtc module deprecation FutureWarning
- Filter vllm "Level is deprecated" stderr noise
- Filter PydanticSerializationUnexpectedValue warnings
- Filter Triton "df: No such file" stderr noise
VLM tokenizer loading (vision.py):
- Add _construct_vlm_processor_fallback() for models where
AutoProcessor.from_pretrained fails (e.g., ERNIE 4.5 VL, LFM2.5-VL)
- Wrap processor loading in try/except with fallback to manual
construction from separate image_processor + tokenizer components
- Add fallback to AutoTokenizer/PreTrainedTokenizerFast when tokenizer
loading or patching fails
TRL 0.27.1 trainer compatibility (trainer.py):
- Add _resolve_trainer_params() to handle thin wrapper trainers that
only have def __init__(self, *args, **kwargs) (e.g., ORPOTrainer
in TRL 0.27.1) by walking MRO for real parameter signature
VLM _is_vlm detection (rl.py):
- Replace blanket _is_vlm=False override with model-architecture-based
detection that checks vision_config or ForConditionalGeneration class
name, fixing VLM training when bare tokenizer is passed as
processing_class
ModernBERT SDPA compatibility (loader.py, sentence_transformer.py):
- Add "modernbert" to DISABLE_SDPA_MODEL_NAMES to avoid stride
alignment issues with torch.compile backward pass
- Add DISABLE_SDPA check for sentence transformer models
Other fixes (_utils.py):
- Suppress false uninitialized weight warnings for VLM
multi_modal_projector.layer_norm
Tested: 92/125 notebooks pass with TRL 0.22.2, 94/125 with TRL 0.27.1.
Remaining failures are infra (missing FFmpeg, network timeouts, GPU
arch) not code bugs.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix KTO shape mismatch on TRL 0.27.2+ and truncation alignment
- Patch KTO get_batch_logps to auto-align logits and labels when Unsloth
model forward truncates input_ids beyond max_seq_length. TRL 0.27.2
changed _process_tokens to only truncate completions (not prompts), so
sequences with long prompts exceed max_seq_length and trigger model-side
truncation. The original ValueError is replaced with min-length alignment.
- Also truncate attention_mask in LlamaModel forward when input_ids are
truncated to max_seq_length, preventing shape mismatches in attention.
- Widen except clause in rl_replacements.py openenv import from
`except ImportError` to `except (ImportError, NameError, Exception)` to
handle vllm SamplingParams NameError in TRL 0.27.2.
* Fix TRL 0.26+ thin wrapper resolution, enable ModernBERT SDPA, clean up warning filters
TRL 0.26+ thin wrapper resolution (rl.py):
- Filter _-prefixed private imports when discovering Trainer/Config classes
- Look up Config in separate *_config.py module when not found in trainer module
- Detect thin wrappers (<1000 chars source) and resolve to experimental parent
via MRO walk; use resolved module for imports and create_new_function
- Enables all 15 trainers to patch successfully (was 5/15 before)
ModernBERT SDPA (loader.py):
- Remove "modernbert" from DISABLE_SDPA_MODEL_NAMES
- SDPA works correctly for both classification and sentence transformers
- Verified: 88.9% accuracy on emotion classification, correct domain-specific
embeddings after sentence transformer fine-tuning
Warning filter cleanup (import_fixes.py):
- Remove cuda.cudart/cuda.nvrtc FutureWarning filters (no such warnings
exist in torch 2.9.1+; proactive suppression is unnecessary)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove multi_modal_projector.layer_norm from uninitialized weight guard
The LFM2.5-VL projector LayerNorm is properly initialized by
transformers and does not need to be excluded from the uninitialized
weight check. The original exclusion was added as a workaround but is
no longer needed after the upstream fix.
* Add transformers 5.0 compat: rope_theta helper, config-as-dim detection, BatchEncoding guard, try/except for TRL trainer source, push_to_hub_token compiler fix
- llama.py: Add _get_rope_theta() helper handling both config.rope_theta and rope_parameters dict
- llama.py: Handle BatchEncoding in unsloth_fast_generate (transformers 5.0+ returns BatchEncoding from apply_chat_template)
- gemma.py: Detect config passed as dim arg in GemmaFixedRotaryEmbedding
- tokenizer_utils.py: Add try/except for TRL trainer getsource in patch_sft_trainer_tokenizer
- rl_replacements.py: Add compiler fix replacing bare pop("push_to_hub_token") with pop(..., None)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use trl.experimental string check instead of char-count heuristic for thin wrapper detection
The <1000 / >1000 char threshold was fragile -- XPOConfig's parent is only
994 chars and would be skipped. All thin wrappers in TRL 0.26+ contain
"trl.experimental" in their deprecation warning, while no real trainer or
config class does, making it a reliable detection marker.
* Move DISABLE_SDPA_MODEL_NAMES import to module level in sentence_transformer
The function-level import was redundant since loader.py is already imported
at module level. Move it to the existing loader import line.
---------
Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Add `inputs_embeds` parameter to `_fast_prepare_inputs_for_generation` so
`model.generate(inputs_embeds=...)` works with Unsloth-patched models.
Changes:
- Add `inputs_embeds=None` to function signature (fixes HF inspect check)
- Track `use_inputs_embeds` flag: True when inputs_embeds provided and no cache
- Conditionally return inputs_embeds on first step, input_ids on subsequent steps
- Handle input_ids being None/empty for batch size and device extraction
- Add attention_mask None-guard before slicing
Fixes: https://github.com/unslothai/unsloth/issues/3798
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: siddhudonda <siddhudonda@users.noreply.github.com>
When using torchrun with quantized models (4bit/8bit/fp8), each rank
must load the model directly onto its own GPU. The default device_map
("sequential") places everything on GPU 0, causing illegal memory
access errors when Accelerate tries to relocate quantized weights.
Use the existing prepare_device_map() utility from loader_utils to
detect distributed training via LOCAL_RANK/WORLD_SIZE env vars and
override device_map to target each rank's local GPU. This is applied
in both FastLanguageModel.from_pretrained and FastModel.from_pretrained,
covering text, vision, and audio model paths.
Fixes#3914
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
* Refactor Ollama template wiring and harden packing helpers
Signed-off-by: Mohammad Miadh Angkad <MAngkad.BSDSBA2027@aim.edu>
* Fix Qwen3 and Gemma3n template bindings and tidy packing test helper
* Fix gptoss Ollama comment and tinyllama stop parameter
- Fix wrong comment referencing gemma3n for gptoss_ollama in chat_templates.py
- Add missing stop keyword to tinyllama PARAMETER in ollama_template_mappers.py
* Fix _DummyTrainer compatibility across TRL versions
The try/except only handled the removal of return_position_ids
(TRL v0.24+) but not the absence of padding_free (TRL v0.18.2).
Gracefully degrade through all optional collator flags so the
test works from trl>=0.18.2 through v0.27+.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Signed-off-by: Mohammad Miadh Angkad <MAngkad.BSDSBA2027@aim.edu>
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* seperate gguf
* fix Modelfile log
* ollama Modelfile create
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix GGUF file placement: move initial conversion to _gguf dir, fix cleanup
- Move initial GGUF files (from convert_to_gguf) into {model_directory}_gguf/
immediately after conversion, so all GGUF outputs live in the dedicated
directory regardless of quantization method (fixes bf16-only case where
quant == first_conversion skipped the loop and _gguf dir was never created)
- Remove redundant gguf_directory/makedirs from inside the re-quant loop
since the directory is now created before the loop
- Use Path.unlink(missing_ok=True) for base GGUF cleanup robustness
- Unify Modelfile location to {save_directory}_gguf/Modelfile for both
VLM and non-VLM models
- Fix print message to show actual modelfile_location path
- Add gguf_directory key to return dict
- Clean up {save_directory}_gguf in push_to_hub_gguf error/finally blocks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
* Implement GGUF upload method for SentenceTransformer
Added a method to convert and upload SentenceTransformer models to GGUF format, including handling of tokenizer, quantization methods, and repository management on Hugging Face Hub.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
On Windows and macOS (Python 3.8+), multiprocessing uses the spawn
start method. When datasets .map(num_proc=N) is called, it creates a
Pool(N) which re-imports __main__ in each worker, causing infinite
recursion and a RuntimeError during bootstrapping.
Guard the auto-computed dataset_num_proc in the generated Config
__init__ by checking multiprocessing.get_start_method() != 'fork'.
When the start method is not fork (spawn/forkserver), force
dataset_num_proc = None so datasets takes the single-process path.
Linux fork behavior is unchanged.
Also replace the fixed memory threshold logic with the simpler
adaptive approach: cap at 64, then min(num_proc, int(available_gb)),
with a safety floor of 1 when available memory is at or below 2GB.
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
* Disable torchcodec in transformers when FFmpeg is missing
When torchcodec is installed but FFmpeg libraries are unavailable,
transformers still thinks torchcodec is available (via find_spec check)
and tries to use it for audio loading, causing RuntimeError.
This adds disable_torchcodec_if_broken() which tests if torchcodec can
actually load its native libraries, and if not, patches transformers'
_torchcodec_available to False so it falls back to librosa instead.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The cuda.cutlass_epilogue_fusion_enabled and cuda.cutlass_tma_only
inductor config options were added in PyTorch 2.8.0. Using these
options on older PyTorch versions causes a RuntimeError during
GRPOTrainer initialization.
This fix adds a version check to only include these options when
running PyTorch 2.8.0 or later, allowing GRPO training to work on
older PyTorch versions (e.g., Colab environments with PyTorch 2.5-2.7).
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
When datasets library has torchcodec installed but FFmpeg libraries
are missing, torchcodec raises a RuntimeError during import. The
exception handler only caught ImportError and AttributeError, causing
the error to propagate and crash Unsloth imports in environments
like Colab where FFmpeg may not be installed.
Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com>
* Improve MoE performance
* small changes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix imports
* disable autotune
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* LoRA for MoE
* Make autotune default
* make dy contiguous
* use non lora model as base for RL
* Revert "use non lora model as base for RL"
This reverts commit bc8f15629d060593b2eaf436f158ff5ac9df0d5d.
* fixup derp
* non TMA [T4]
* Revert "non TMA [T4]"
This reverts commit 35304566690e7c9ab9632899920c85bff322409a.
* Fixes for VL MoE and v5 transformers
* [transformers] [v5] remove unused hybridcache (#3910)
* remote unused hybridcache
* cleanup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* No double compile for qwen3moe
* Fix top_k on trl GRPO
* Recognise GLM as MoE
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix missing RotaryEmbeddingConfigMixin
* Licensing for autotuning cache
* Cleanup
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Erland366 <erland.pg366@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
_patch_trl_rl_trainers enumerates all trainer modules from dir(trl.trainer)
and attempts to import each one. Modules like alignprop_trainer fail because
they depend on optional packages (diffusers) that may not be installed. The
failure is harmless but the print() call produces noise on every import.
Change print() to logger.info() so these messages only appear when
UNSLOTH_ENABLE_LOGGING=1.
Co-authored-by: Daniel Han <danielhanchen@users.noreply.github.com>
GPT-OSS models use eager attention during inference because flex
attention returns incorrect results (likely due to left padding).
However, when _attn_implementation is set to "flex_attention",
transformers creates BlockMask objects which cause a TypeError
when passed to the eager attention path:
TypeError: unsupported operand type(s) for +=: 'Tensor' and 'BlockMask'
This fix excludes GPT-OSS from using flex_attention, keeping it on
the eager path to avoid the BlockMask/Tensor type mismatch.
* Enable flex attention by default
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Avoid dropping flex attention when SDPA unsupported
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Update rl_replacements.py
* Update rl_replacements.py
* Update rl.py
* Update rl_replacements.py
* Update rl_replacements.py
* Update rl.py
* Update rl.py
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update rl_replacements.py
* Update rl.py
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update rl_replacements.py, remove chat template from codexes commits
* Update rl.py, got rid of gradient checkpointing code that did not work
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
