Now that the upstream patch fixes have landed (#5319 for the three
patch_* helpers, unsloth-zoo#628 for the MoE coverage canary), every
observed cell-level red was one of those two things. Both are fixed,
so re-run the matrix in strict mode:
- Removed every per-step `continue-on-error: true`. A failing test step
fails the cell. The previous green-with-fail-prints lie is gone.
- Runtime patch ledger: was `assert REQUIRED helpers exist by name`
(an inventory walk). Now also `assert len(fail) == 0` -- any
zero-arg patch that raises is a real regression. NEEDS_PRECONDITION
still skips the three patches that legitimately need real CUDA /
runtime args.
- patch_tiled_mlp shim: bumped seq_len from 4 to 192 with hidden=64 so
divmod(192, 64) = (3, 0) and the tiled path actually runs 3 shards
instead of degenerating to n_shards=1 (which is bit-exact and only
confirms patching installed something). Added an explicit
pre-assertion that we are exercising multi-shard.
- openenv graceful-skip warning: previous text said "Weight reload
still functional" which over-promised. Replaced with the literal
consequence: duplicate `collective_rpc("reload_weights")` is not
stripped and `wake_up(tags=["kv_cache"])` is not retagged. Most
users are unaffected; openenv GRPO users on this TRL build may see
redundant reload_weights or partial wake_up.
Includes a merge of main into this branch so the consolidated cells
pip-install the post-#5319 unsloth tree.
* fix: import fast_lora_forward inside patch_fast_lora
patch_fast_lora has referenced an unbound `fast_lora_forward` since
ddf118a8f (2024-11-21). The function is defined at
unsloth/kernels/fast_lora.py:652 and re-exported through
unsloth/kernels/__init__.py:45, but it was never imported into
unsloth/models/_utils.py, so calling patch_fast_lora() raises
NameError: name 'fast_lora_forward' is not defined.
The bug went unnoticed because no production code path calls
patch_fast_lora() unconditionally. Surfaced by a new CPU-CI check that
invokes every zero-arg patch_* helper across unsloth + unsloth_zoo
(consolidated-tests-ci.yml on PR #5312).
Importing inside the function (rather than at module top) keeps the
import surface narrow and avoids a circular-import risk if
unsloth.kernels.fast_lora ever needs to import from
unsloth.models._utils.
* fix: inject typing imports into patch_sft_trainer_tokenizer's exec namespace
patch_sft_trainer_tokenizer rewrites the source of TRL's SFTTrainer
methods (_prepare_non_packed_dataloader, _prepare_dataset) and re-execs
them. With TRL 1.x, those methods carry `Union[...]` type hints in
their signatures. The current rewrite only injects identifiers found by
`dir(trl.trainer.sft_trainer)` into the exec namespace, which does not
include `Union`, so exec(function, ...) raises NameError: name 'Union'
is not defined.
Fix: import Union, Optional, List, Any, Callable, Tuple, Dict, Iterator
inside the function. exec receives `locals()` as its globals dict, so
those names are visible to the executed source body. Same pattern as
unsloth/models/_utils.py:patch_linear_scaling, which already injects
`from typing import Union, Optional, List, Any, Callable, Tuple` into
its own exec_code.
Surfaced by the consolidated CPU-CI runtime patch_* check on PR #5312
in the matrix cell `transformers>=5,<6 + trl>=1,<2`.
* fix: guard openenv_vllm_reload_weights against OSError from inspect.getsource
TRL 0.29.1 and the 1.x line ship some openenv helpers as compiled
bytecode without accessible source on disk. inspect.getsource(patch_target)
raises OSError("could not get source code") in that case, which surfaces
as a hard failure in patch_trl_openenv() and aborts the rest of the
RL_ADDITIONAL_FUNCTIONS["openenv"] iteration.
Wrap the getsource call in a try/except OSError and log a warning
instead. The wake_up(tags=...) rewrite is the only thing skipped; the
core weight-reload patch path stays functional.
Surfaced by the consolidated CPU-CI runtime patch_* check on PR #5312
in matrix cells running TRL 0.29.1 (latest <1.0.0) and TRL 1.3.0
(latest 1.x). The pyproject pin (TRL 0.18.2-0.24.0) still gets source
for this function so the original code path runs unchanged there.
Matches the release cut from the pip branch that ships PR #5296: Studio
chat history and attachments work again with newer @assistant-ui/react,
plus the pinned assistant-ui surface and frontend package-lock.json so
future installs cannot drift back onto a broken bundle.
* Fix FastSentenceTransformer compatibility with sentence-transformers 5.4
* Support varied Transformer init signatures
Detect Transformer.__init__ parameters and build init kwargs accordingly so trust_remote_code and other args are passed using the correct names. Instead of unconditionally using model_args/config_args, the code now inspects the constructor to decide between model_kwargs/config_kwargs vs model_args/config_args and also sets processor_kwargs or tokenizer_args when present. Initializes Transformer with constructed transformer_kwargs (including max_seq_length) to improve compatibility with different Transformer implementations.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden SentenceTransformer path and module checks
* Scrub .github/workflows for staging push (matches staging base)
* Guard auto_model write in FastSentenceTransformer._apply_torch_compile
On sentence-transformers >=5.4 Transformer.auto_model is a read-only
@property backed by self.model, so a direct assignment raises
AttributeError. The two get_peft_model paths already guard the write
with isinstance(getattr(type(...), "auto_model", None), property);
the auto-compile path missed the same guard, which broke the default
trainer path whenever max_steps >= _compile_threshold.
* Add tests for FastSentenceTransformer property guards
* Tighten FastSentenceTransformer redirect lifecycle tests
Drop a duplicate assertion-less case, remove dead AST extraction helper,
and trim unused imports. The remaining six tests cover substitution on
match, restoration on constructor exception, passthrough for unrelated
names, pathlib.Path normalisation, trailing slash handling, and the
no-identifier guard.
* Sync .github/workflows with upstream author branch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Avoid sharing trust_remote_code kwargs dict across constructor buckets
In FastSentenceTransformer._create_transformer_module, the same
trust_remote_code_kwargs dict was being assigned to model_kwargs,
config_kwargs, and processor_kwargs (or model_args / config_args /
tokenizer_args) on the Transformer constructor. transformers'
from_pretrained code paths (configuration_utils, auto_factory,
processing_auto, etc.) call kwargs.pop("trust_remote_code", ...) on
the dict they receive, which would drain the shared object and silently
strip trust_remote_code from the other buckets. Pass an independent
copy to each bucket so subsequent buckets and any pass-through
auxiliary loads still see trust_remote_code.
* Wire do_lower_case and return_dict through Transformer init for ST 5.4
In FastSentenceTransformer._create_transformer_module:
- When Transformer.__init__ accepts do_lower_case (ST 5.4+), pass
the unsloth tokenizer's do_lower_case as a constructor kwarg. The
existing post-init attribute assignment alone is too late: ST 5.4's
__init__ uses do_lower_case to install a Lowercase normalizer on
tokenizer.backend_tokenizer.normalizer, which is not re-applied if
we only set the attribute after construction. The post-init line
is preserved untouched for older ST versions.
- Add return_dict to the manually completed model_forward_params set
so wrapped models with forward(*args, **kwargs) signatures keep ST's
forced dict-like output safety net. ST 5.4's own __init__ unions the
forward signature with the same set plus return_dict; the previous
override silently dropped it.
* Preserve flash-attention forward keys when wrapping ST 5.4 Transformer
Sentence-transformers 5.4's Transformer.__init__ calls
_can_flatten_inputs() during construction, which augments
self.model_forward_params with cu_seq_lens_q, cu_seq_lens_k,
max_length_q, max_length_k, seq_idx whenever feature-extraction with
text modality, the torch backend, flash-attention 2, and varlen
flash-attn support are all available. The post-init override of
transformer_module.model_forward_params used to replace the attribute
outright, silently dropping those keys so ST's preprocess() filter
stripped flash-attn kwargs before reaching model.forward.
Snapshot the constructor-populated set first, leave the existing
overwrite intact for the forward-signature plus tokenizer keys, and
union the snapshot back in so flash-attn forwarding keeps working on
ST 5.4. For older sentence-transformers releases the attribute is
absent and getattr returns an empty set, leaving behavior unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* MROPE for VLM GRPO
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: pluesclues <136766175+pluesclues@users.noreply.github.com>
* Fix DPO trainer multi process hang
* Fix datacollator error
* further dpo vision changes
* cleanup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden DPO vision row processing and source rewrites
- dpo_trainer_vision_signature_columns: also match TRL 0.22.x layout
(image_sizes followed by ref_chosen_logps), so vision keys are not
stripped via remove_unused_columns on the originally-affected version.
- dpo_trainer_concatenated_inputs: fall back to inserting after the
image_sizes block when no token_type_ids anchor follows it.
- Apply the same vision model_kwargs forwarding rewrite to
_compute_loss_liger via dpo_trainer_compute_loss_liger so the Liger DPO
path does not drop pixel_position_ids/image_position_ids/
mm_token_type_ids when args.use_liger_loss is true.
- dpo_trainer_vision_process_row:
- guard chosen/rejected EOS append with tokenizer.eos_token_id is not None
- use features.get("images") and features.get("prompt") to match the
existing get on line 164 and avoid KeyError on rows without those keys
- drop the torch.is_tensor gate so list-form pixel_position_ids/
image_position_ids returned without return_tensors are still aliased
- skip the loop entry for image_position_ids when it was already
promoted to pixel_position_ids, so the output dict no longer carries
both keys with identical data
- dpo_trainer_data_collator_vision_keys: switch from pad_sequence to
trl.trainer.utils.pad with padding_side='left' (matches the DPO
collator's prompt left-pad) and padding_value=-1 for *_position_ids
keys (sentinel for padded patches), 0 otherwise. Skip the key when not
every example carries it. Falls back to pad_sequence if trl.pad is
unavailable or the tensor rank is too high.
- dpo_trainer_prepare_dataset: keep TRL's writer_batch_size=10 when
popping num_proc; removing it defaults to 1000 and reintroduces the
vision OOM risk that writer_batch_size=10 was set to avoid.
* DPO vision row: keep upstream-facing keys and fix patch padding
- dpo_trainer_vision_process_row: no longer aliases image_position_ids
to pixel_position_ids. Each upstream-emitted vision key is forwarded
under its own name. Gemma4 ForConditionalGeneration.forward accepts
image_position_ids directly and renames it to pixel_position_ids only
at the vision-tower call site, so aliasing in the row helper hid the
kwarg the model actually consumes.
- dpo_trainer_vision_process_row: extract pixel_values via "in"
membership instead of unconditional indexing. With the missing-images
path returning [] to the processor, modern processors no longer emit
a pixel_values key, and the previous indexing raised KeyError.
- dpo_trainer_data_collator_vision_keys: pick padding_side per key
family. *_position_ids tensors are patch-aligned to pixel_values
(TRL's DataCollatorForPreference right-pads pixel_values), so pad
them right with the -1 sentinel; mm_token_type_ids is token-aligned
to prompt_input_ids (left-padded by TRL), so pad it left with 0.
* DPO vision: handle multi-image prompts and arbitrary-rank collator pad
- dpo_trainer_vision_process_row: when a prompt is missing vision
placeholders, insert one placeholder per missing image instead of
always inserting a single token. Multi-image rows now satisfy the
processor's token-vs-image count check rather than under-inserting
and tripping the placeholder/feature mismatch.
- dpo_trainer_data_collator_vision_keys: drop the dim()<=2 gate around
trl.trainer.utils.pad. trl.pad handles arbitrary rank correctly,
while the previous fallback to torch.nn.utils.rnn.pad_sequence
raised RuntimeError on rank-3 patch-position tensors with mismatched
non-leading dimensions. The pad_sequence path remains as a degraded
fallback only when trl.pad is unavailable or raises.
* DPO vision row: support scalar images and align prompt-aligned aux ids
- dpo_trainer_vision_process_row: type-aware normalization of the
features['images'] column instead of a truthiness/len check that
raised on single image objects (PIL.Image has no __len__) and on
numpy ndarrays (truthiness ambiguous). Lists/tuples count as their
length, scalar image objects count as one, None counts as zero, and
the original value is forwarded to the processor.
- dpo_trainer_vision_process_row: when max_prompt_length truncates
prompt_input_ids, also slice token_type_ids and mm_token_type_ids
by the same [-max_prompt_length:] suffix. Those keys are 1:1 token
aligned to prompt_input_ids (Gemma 4 vision attention keys off
mm_token_type_ids per modular_gemma4.py), so leaving them at the
original length silently misaligned the multimodal mask.
* DPO vision row: stop synthesizing vision-token placeholders
Pass features['prompt'] and features['images'] straight to the
processor without inserting any extra placeholder tokens. The previous
helper used processing_class.image_token, which is the right prompt
placeholder for Gemma 4 but the wrong one for Gemma 3 (whose prompt
placeholder is boi_token while image_token is the inner expansion
target). Synthesizing that token also broke multi-image rows: text
ended up with N placeholders while the row helper only forwarded the
first image's pixel_values via the standard [0] indexing that mirrors
upstream TRL process_row, so token vs image-feature counts diverged.
Removing the synthesis matches stock TRL behavior; users provide the
correct placeholders for their processor in the prompt.
* Add tests for DPO vision row processor passthrough
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* fix: avoid PerceptionEncoder ImportError blocking trust_remote_code model loads
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update config class retrieval in _utils.py
Refactor config class retrieval logic to use model mapping.
* resolve_model_class: restore _extra_content fallback
The previous fallback iterated mapping.items(), which transformers'
_LazyAutoMapping defines as _model_mapping entries + _extra_content
entries. The PR's per-key loop covers only _model_mapping, so
subclasses of configs registered via AutoModel.register(cfg, model)
silently resolve to None. Add a safe isinstance pass over
_extra_content (no lazy loads, no crash risk) before giving up.
* Add tests for resolve_model_class fallback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* fix: multi-GPU inference crash for bnb 4-bit/8-bit models
When load_in_4bit or load_in_8bit is used with device_map="sequential"
and max_memory constraints that place weights across multiple GPUs (or
entirely on a non-default GPU like cuda:1), the bitsandbytes loading
path in transformers never calls dispatch_model. No AlignDevicesHook is
installed, and the first forward/generate call crashes with:
RuntimeError: Expected all tensors to be on the same device
This adds _attach_bnb_multidevice_hooks() which is called after
from_pretrained returns. It infers a device map from actual parameter
placements and calls dispatch_model(force_hooks=True) to install the
missing hooks. The function is a complete no-op for the common
single-GPU cuda:0 case.
Call sites: FastBaseModel.from_pretrained (vision.py) and
FastLlamaModel.from_pretrained (llama.py).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: align with PR #5053 final review improvements
- Add hook call to the bnb quantized loading branch in llama.py (the
primary load_in_4bit path), not just the non-fast-inference fallback
- Expand bnb detection: also check model.is_loaded_in_4bit,
model.is_loaded_in_8bit, model.quantization_method
- Pass explicit main_device and skip_keys to dispatch_model
- Use logger.info instead of print for the success message
- Use kwargs.get("load_in_8bit", False) at llama.py call sites
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Restrict flash attn to <=256 head dim. Consolidate attn impl checks
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Consolidate the changes into single function
* safeguard for dict instead of object
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix pass attn implementation
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Reapply "updated models template mappers. added lfm2.5vl450m to transformers 5…" (#4945)
This reverts commit 33503ea248.
* Add missing gemma-4-31B-it bnb-4bit mapper entry and LFM2.5 upstream namespace for PR #4950
- Add unsloth/gemma-4-31B-it-unsloth-bnb-4bit to __INT_TO_FLOAT_MAPPER so
the int-to-float resolution works for this model (already listed in
TEMPLATE_TO_MODEL_MAPPER but had no mapper entry).
- Add LiquidAI/LFM2.5-1.2B-Instruct to lfm-2.5 TEMPLATE_TO_MODEL_MAPPER
entry so the canonical upstream namespace is mapped consistently with lfm-2.
* Add missing gemma-4-31B-it bnb-4bit Ollama mapping and lfm-2.5 chat template alias
- Add unsloth/gemma-4-31B-it-unsloth-bnb-4bit to OLLAMA_TEMPLATE_TO_MODEL_MAPPER
so Ollama export works for this model (E2B-it and E4B-it bnb-4bit variants were
already present, 31B-it was inconsistently omitted)
- Register CHAT_TEMPLATES["lfm-2.5"] as alias of the lfm-2 template to prevent
KeyError when Studio resolves LFM2.5 models through MODEL_TO_TEMPLATE_MAPPER
* Add missing LFM2 bnb-4bit INT_TO_FLOAT_MAPPER entry
unsloth/LFM2-1.2B-unsloth-bnb-4bit is referenced in model_mappings.py
but had no mapper.py entry, so model resolution would fail when users
load that variant with load_in_4bit=False or when the float name is
used with load_in_4bit=True.
* Fix review findings for PR #16
1. ollama_template_mappers.py: Restore dropped Gemma-4 base model IDs
(E2B, E4B, 31B, 26B-A4B) and add missing google/ upstream IDs to
the gemma4 Ollama mapper for consistency with other gemma entries.
2. mapper.py: Remove self-mapping non-bnb-4bit entries from
__INT_TO_FLOAT_MAPPER that were polluting FLOAT_TO_INT_MAPPER with
lowercase 16-bit names, causing load_in_4bit=True to return bad
model names. Add direct MAP_TO_UNSLOTH_16bit entries to preserve
the google->unsloth 16-bit redirects.
3. mapper.py: Add LFM2.5 MAP_TO_UNSLOTH_16bit redirect so
LiquidAI/LFM2.5-1.2B-Instruct resolves to its unsloth mirror.
* Add review tests for PR #4950
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove top-level test files
These test_*.py files were added at the repo root rather than under tests/.
Removing them from this PR; the production mapper changes remain.
* Add gemma-4-26B-A4B-it mapping
Adds unsloth/gemma-4-26B-A4B-it to __INT_TO_FLOAT_MAPPER as a 2-tuple so
google/gemma-4-26B-A4B-it routes to unsloth/gemma-4-26B-A4B-it across
INT_TO_FLOAT_MAPPER, FLOAT_TO_INT_MAPPER, and MAP_TO_UNSLOTH_16bit.
The 26B-A4B (MoE) model has no bnb-4bit variant, so the key uses the
plain unsloth name rather than the -unsloth-bnb-4bit suffix.
Removes the now-redundant standalone _add_with_lower call for the -it
variant; the 16bit mapping is registered via the dict loop.
* Add unsloth-bnb-4bit mappings for gemma-4 base (non-it) models
Adds E2B, E4B, 31B base unsloth-bnb-4bit entries to __INT_TO_FLOAT_MAPPER.
The 26B-A4B (MoE) base has no bnb-4bit variant on HF, so it stays on the
standalone _add_with_lower line for the 16bit-only routing.
Removes the redundant _add_with_lower lines for E2B, E4B, 31B base since
the dict loop now registers the same google->unsloth route through the
2-tuple entries, plus full FLOAT_TO_INT and INT_TO_FLOAT coverage.
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: Add cactus QAT scheme support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test(qat): add tests for cactus QAT scheme and fix missing import
* Fix cactus QAT scheme: correct MappingType import, tighten PerGroup filter
- Drop the broken `from torchao.dtypes import MappingType` import. `MappingType`
lives in `torchao.quantization` (and `torchao.quantization.quant_primitives`);
it is not exported from `torchao.dtypes` in any supported torchao release
(verified on 0.14, 0.16, 0.17). The previous code raised `ImportError` on
every cactus call and was masked as a misleading 'torchao not found' error.
- Since `IntxWeightOnlyConfig` already defaults `mapping_type` to
`MappingType.SYMMETRIC`, drop the explicit kwarg entirely and remove the
import. Behavior is unchanged.
- Introduce a named `group_size = 32` constant (matches the int4 / fp8-int4
pattern in the surrounding branches) and add a `% group_size == 0`
divisibility guard to the filter. `PerGroup(32)` requires
`in_features % 32 == 0` at `quantize_()` time, otherwise torchao raises
`ValueError: in_features (N) % group_size (32) must be == 0`. The old
`in_features >= 32` filter would admit non-aligned widths (e.g. 33, 48, 65,
127) and crash `_prepare_model_for_qat` for those shapes.
* Warn when cactus QAT skips non-divisible Linear layers
Multiple reviewers flagged that the divisibility guard added in the
previous commit can silently leave Linear layers in full precision when
their in_features is not a multiple of 32. For currently supported
Unsloth models (Qwen, Llama, Gemma, Mistral, Phi) every Linear width is
already a multiple of 32/64/128 so this never triggers, but surfacing
the coverage gap is cheap and avoids users assuming 100% QAT coverage
when they bring a custom model with unusual shapes.
Emit a UserWarning listing up to the first 8 skipped layers whenever
the cactus filter excludes any Linear due to the modulo guard. This
keeps the lenient silent-skip behavior (consistent with int4 /
fp8-int4), but stops making it silent.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* feat: Add support for OLMo-3 model in mapping and tests
* Update unsloth/models/mapper.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update tests/test_get_model_name.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Fix casing, add Think variants, and align version gate for OLMo-3 PR 4678
Mapper: switch slugs from OLMo-3 to canonical Olmo-3 mixed case, drop the
non-existent unsloth/Olmo-3-7B-Instruct-bnb-4bit dead alias, and add the
already-published Olmo-3-7B-Think and Olmo-3-32B-Think Unsloth mirrors.
Loader: change the olmo3 transformers version gate from Version("4.57.0")
to Version("4.57.0.dev0") so nightly/source builds that already contain
olmo3 are not blocked, matching the OLMo-2, Gemma 3 and Cohere patterns.
* Use canonical Olmo-3 casing and cover Think variants in OLMo-3 tests
Mirrors the mapper.py fixes on pr-4678-code: HuggingFace canonical slugs
for the OLMo-3 family use mixed-case Olmo-3 (not OLMo-3 like OLMo-2), and
Unsloth already hosts Olmo-3-7B-Think and Olmo-3-32B-Think mirrors, so
the resolution matrix now covers all three published Olmo-3 families.
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
The "Patched trl.models.utils.disable_gradient_checkpointing with a no-op"
warning fires once on every Unsloth import, including from notebooks where
the user did not opt into verbose logging. It is a routine integration
patch, not an anomaly the user needs to know about. Gate it on
UNSLOTH_ENABLE_LOGGING=1 like other diagnostic notices.
* Fix grad-accum model_accepts_loss_kwargs detection for vision wrappers
Replace the source-string rewrite of Trainer.__init__ with an instance-level
accepts_loss_kwargs shadow applied on the loaded model. Covers:
1. Unsloth-compiled forward -> True, so HF Trainer does not double-scale
on top of unsloth_fixed_cross_entropy's num_items_in_batch division.
2. Stock forward on a conditional-generation wrapper (Gemma3n, Gemma3
pre-4.57, Qwen-VL family, etc.) where the outer class has no
accepts_loss_kwargs but the inner .model declares False -> False.
This is the case that reproduces issue #4982 under trust_remote_code
or UNSLOTH_COMPILE_DISABLE, where the previous fix's outer-attr
check walked past the inner model and fell through to signature
inspection.
3. Text LMs without any explicit accepts_loss_kwargs -> leave HF default.
The previous .replace()-based patch silently no-ops on transformers 4.48
through 4.52 (variable named model, not unwrapped_model) and is fragile
against any upstream reformat. The new helper walks the PEFT / HF wrapper
chain, finds the first class that declares accepts_loss_kwargs on its own
class dict (type(m).__dict__, not hasattr, to avoid PEFT __getattr__
forwarding), and setattr-shadows that value at every wrapper level so
HF Trainer's hasattr(unwrapped_model, ...) check picks it up at whichever
level accelerate.unwrap_model returns.
Also adds an unconditional post-init clamp of
accelerator.gradient_accumulation_steps = 1 to work around the
transformers 5.0 through 5.5 GradientAccumulationPlugin regression that
makes accelerator.backward divide loss by GA on top of training_step's
own /GA division. Fixed upstream in 5.6.0.dev0; no-op on 4.x and 5.6+.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments
* Address review: cover PEFT-after-load and custom compile location
Two review findings from 3/20 reviewers:
1. [3 of 20 reviewers] apply_accepts_loss_kwargs_fix was called from the
loaders before get_peft_model wraps the base model, so on transformers
4.48-4.52 (which does hasattr on the outer model) the instance shadow
on the base model was lost after PEFT wrapping. Fix: also call it from
the wrapped Trainer.__init__ so it runs on whatever model the user
actually hands to Trainer, which is always the final wrapped form.
2. [1 of 20 reviewers] _forward_is_unsloth_compiled hard-coded the
substrings "unsloth_compiled" / "unsloth_cache" in the co_filename
check, which misclassifies compiled forwards when
UNSLOTH_COMPILE_LOCATION is set to a custom directory. Fix: new
_unsloth_compile_cache_leaves helper that reads the env var and
matches the basename against path components, honoring both the
default and any user override.
Verified locally:
- PEFT-after-load simulation: HF's hasattr(peft, "accepts_loss_kwargs")
now returns True after our init wrapper runs, and value resolves to
False on Gemma3n-style inner wrappers.
- Custom UNSLOTH_COMPILE_LOCATION simulation: compiled detection returns
True for /tmp/my_custom_cache/compiled.py when the env var is set.
- End-to-end Gemma-3 270m + LoRA SFT unchanged: loss 4.9626, grad-norm
matches prior run, all 4 wrapper levels now carry the shadowed attr.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Respect classification head skip list on pre-quantized 4-bit checkpoints (#5027)
FastLanguageModel.from_pretrained(..., num_labels=N) crashed with
"NotImplementedError: normal_kernel_cuda not implemented for 'Byte'" on
pre-quantized bnb 4-bit checkpoints (e.g. unsloth/Qwen3-4B-bnb-4bit)
when running on transformers 5.x.
Two pieces were needed to close this out:
1. unsloth_zoo PR: add "score", "classifier", "qa_outputs" to
SKIP_QUANTIZATION_MODULES so replace_with_bnb_linear leaves task
heads in the compute dtype.
2. This commit: for pre-quantized checkpoints, transformers reads
llm_int8_skip_modules from the quantization_config baked into
config.json and ignores the runtime BitsAndBytesConfig we pass via
kwargs. Unsloth must merge its skip list into
model_config.quantization_config.llm_int8_skip_modules before the
from_pretrained call, or the checkpoint's frozen list
(e.g. ["lm_head", "multi_modal_projector", "merger",
"modality_projection"]) wins and the `score` head gets converted to
Linear4bit with uint8 storage, then _init_weights calls normal_ on
uint8 and crashes.
Also add a defensive post-load cast on the task head to guard against
any residual path that ends up with a non-floating head dtype.
Verified on transformers 4.57.6 and 5.5.0 with:
- unsloth/Qwen3-4B-bnb-4bit + num_labels=3
- unsloth/Qwen3-4B (non-bnb repo, load_in_4bit=True)
- unsloth/Llama-3.2-1B-Instruct + num_labels=3
- unsloth/ModernBERT-large classifier head (bert_classification notebook)
- Regression: causal LM path unchanged, backbone still 4-bit
- 3-step SFT on num_labels=3 confirms gradient flow and weight updates
on score.weight
Fixesunslothai/unsloth#5027
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Fixes#2393.
- `_utils.py`: `has_internet()` now respects `HF_HUB_OFFLINE` with truthy variant parsing in addition to `TRANSFORMERS_OFFLINE`.
- `_utils.py`: replace uncontrolled `except Exception: stats_check()` retry (which had no time limit and could freeze on Kaggle offline mode) with a logged skip.
- `loader.py`: forward `local_files_only` from kwargs into all `AutoConfig.from_pretrained` and `PeftConfig.from_pretrained` probes in `FastLanguageModel.from_pretrained` and `FastModel.from_pretrained`, including the PEFT base-model reload paths.
* Fix num_items_in_batch GA for Gemma4
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix Gemma-4 GRPO catastrophic KL divergence with TRL 1.0.0+
Two compounding bugs caused Gemma-4 GRPO training to diverge with KL ~10^12
at step 1 against TRL 1.0.0+. Both fixes are runtime patches in the existing
TRL/model patch flow and are no-ops for models and TRL versions that are not
affected.
Fix 1 (rl.py): replace trl.models.utils.disable_gradient_checkpointing with
a no-op context manager. TRL 1.0.0+ wraps generation in
`with torch.no_grad(), disable_gradient_checkpointing(self.model, ...):`
purely to suppress a cosmetic PyTorch warning ("None of the inputs have
requires_grad=True"). Inside torch.no_grad() the gradient checkpointing
state has no functional effect on the forward pass. On context exit, TRL
calls model.gradient_checkpointing_enable() which dispatches to HF's
generic implementation and overwrites Unsloth's custom
`use_gradient_checkpointing="unsloth"` wrapper, corrupting Gemma-4 forward
numerics. Replacing the toggle with a no-op preserves Unsloth's custom GC
wrapper across generation passes. The patch walks sys.modules dynamically
to also rebind the symbol on every trl.* module that already imported it
(grpo_trainer, dpo_trainer, rloo_trainer, dppo_trainer, gfpo_trainer,
grpo_with_replay_buffer_trainer, and any future trainer module).
Fix 2 (vision.py): inject `final_logit_softcapping` from `config.text_config`
into the top-level `model.config` for multimodal models. Unsloth's GRPO
trainer reads `getattr(model.config, "final_logit_softcapping", 0)` but
for Gemma-4 the attribute lives only on the nested `Gemma4TextConfig`,
so the lookup silently defaults to 0 instead of 30.
Backwards compatibility:
- trl 0.22.2: no `disable_gradient_checkpointing` symbol exists, the patch
early-returns via `hasattr` guard.
- trl 0.27.1: same broken pattern as 1.0.0, the noop replacement is correct.
- trl 1.0.0+: end-to-end verified on `unsloth/gemma-4-E2B-it` GRPO with TRL
1.0.0 and transformers 5.5.0. Step 1 loss=2.46e-08, kl=2.92e-05 (machine
zero) vs broken baseline loss=1.37e+06, kl=1.76e+09.
- Llama / non-VLM text models: Fix 2 is a no-op (no `text_config`); Fix 1
is functionally identical (Unsloth's GC wrapper is preserved).
- Qwen3-VL and other VLMs without final_logit_softcapping: Fix 2 is a no-op
(text_config.final_logit_softcapping is None).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Apply loop 1 review fixes for PR #4934
- Move Fix 2 from vision.py to rl_replacements.py:858 and :1110 at the
actual consumer sites. This avoids mutating model.config (which could
leak into save_pretrained output) and covers text-only Gemma-4 paths
that do not flow through FastBaseModel.from_pretrained.
- Revert the vision.py injection block entirely.
- Narrow the bare except blocks in patch_trl_disable_gradient_checkpointing
from `except Exception:` to `(AttributeError, ImportError)` and
`(AttributeError, TypeError)` to avoid masking unrelated bugs.
- Add logger.warning_once when the noop patch is installed, matching
patch_trl_openenv and patch_trl_vllm_generation convention.
- Remove the dead per-module `_unsloth_noop_patched` sentinel check inside
the sys.modules walk. The function-level early return already covers
this case.
- Move `import sys` and `from contextlib import contextmanager` to the
module-level imports instead of inside the function body.
- Rewrite the ordering comment in PatchFastRL to accurately describe
why patch_trl_disable_gradient_checkpointing must run before
patch_trl_rl_trainers.
- Fix keyword default spacing to match surrounding rl.py style.
End-to-end verified: Gemma-4-E2B GRPO on TRL 1.0.0 + transformers 5.5.0
step 1 loss=2.464e-08 kl=2.921e-05, all 5 steps succeed.
* Apply loop 2 review fix for PR #4934
Extract the final_logit_softcapping fallback logic into a shared helper
`_unsloth_get_final_logit_softcapping(config)` defined in rl_replacements.py
and injected into the compiled cache via RL_PRE_ITEMS["grpo_trainer"]. Both
call sites (`grpo_trainer__generate_and_score_completions` and
`grpo_trainer_compute_loss`) now use the helper instead of inlining the
same text_config fallback block twice.
Verified: compiled cache file lists the helper at module scope and both
consumer sites call it. Gemma-4-E2B GRPO step 1 loss=2.464e-08 kl=2.921e-05
(unchanged), all 5 steps pass.
* Apply loop 3 review fix for PR #4934
Extend _unsloth_get_final_logit_softcapping to also fall back to
config.get_text_config() for composite configs such as T5GemmaConfig
where the text sub-config is not exposed via the text_config attribute
but only via the get_text_config() method. Guard against (TypeError,
ValueError) raised by ambiguous composite configs, and skip the
self-referential case where get_text_config() returns self.
This addresses the 6/7 reviewer consensus from the third review loop.
Verified:
- Helper returns 30.0 for Gemma-4, T5Gemma, and Gemma 1/2 configs.
- Helper returns 0 for Llama, Qwen, Mistral, Cohere, Granite, and
ambiguous configs raising ValueError.
- Gemma-4-E2B GRPO step 1 loss=2.464e-08 kl=2.921e-05 (unchanged).
- Llama-3.2-1B GRPO all 5 steps loss=0 kl=0 (no regression).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* updated models template mappers. added lfm2.5vl450m to transformers 5.3.0 whitelist
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix Mistral training crash when xformers is unavailable
* Fix/adjust Mistral DPO training crash fix for PR #4889
- Clarify comment in MistralForCausalLM_fast_forward: the DPO embed-masking
block runs BEFORE attention_mask is nulled out, and it is the consumer that
requires a 2D mask.
- Add defensive attention_mask.ndim == 2 guard to the LlamaModel_fast_forward
DPO embed-masking block so it self-protects if a 4D mask ever reaches it.
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* fix: skip redundant HfFileSystem().glob() calls in loader.py
Guard the SUPPORTS_LLAMA32 glob blocks with `is_model and is_peft` so
the HfFileSystem HTTP call is only made when both configs could actually
exist. This prevents indefinite hangs on slow/unreliable networks since
the glob result is redundant when either AutoConfig or PeftConfig
already failed to load.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove test file from main PR - moved to separate PR
Tests for the glob skip guard belong in their own PR to keep
the loader change minimal and reviewable.
* Harden HfFileSystem glob: fix Windows path splitting, add try/except
- Use str.rsplit("/", 1) instead of os.path.split to extract filenames
from HfFileSystem paths. HfFileSystem always returns POSIX-style paths,
but os.path.split uses the OS separator, so on Windows the entire path
was returned as the "filename" and the config name comparison always
failed.
- Wrap the HfFileSystem().glob() call in try/except to gracefully handle
network failures (offline mode, timeouts, unreachable Hub). On failure
both_exist stays False, which is the safe default.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove redundant HfFileSystem().glob() call for remote repos
When is_model and is_peft are both True, AutoConfig and PeftConfig
have already loaded successfully, proving both config.json and
adapter_config.json exist. The HfFileSystem network call to re-verify
this was redundant and could cause hangs on slow networks.
Replace the glob + try/except block with a direct both_exist = True
assignment.
* Remove unused HfFileSystem import
HfFileSystem was only used for the glob() calls that were replaced
with direct both_exist = True assignments in the previous commit.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Gemma-4 does not need FORCE_FLOAT32. Testing shows that both float16 and
bfloat16 work correctly without the forced float32 override:
- Inference: identical outputs for float16 and bfloat16 (greedy decoding)
- Training (100 steps, 4-bit LoRA, SFT on FineTome-100k):
- float16 final loss: 3.048
- bfloat16 final loss: 3.065
- Losses converge to within 0.02 by step 60
- Grad norms healthy and comparable for both dtypes
The FORCE_FLOAT32 path was actually causing training divergence. With
it enabled, the compiled float32 run diverged at step ~28 with grad norms
collapsing to near zero and loss plateauing at ~12.4. Without it, both
dtypes train normally.
This enables float16 on Tesla T4 and other GPUs without bfloat16 support.
* fix(studio): reuse HF cached repo casing to prevent duplicate downloads
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Move cache case resolution tests to separate PR
Tests for resolve_cached_repo_id_case and get_model_config case resolution
belong in their own PR to keep this change focused on the runtime fix.
* fix(studio): debug-log HF_HUB_CACHE fallback in path_utils
* Fix stale memoization in resolve_cached_repo_id_case
- Check exact-case path before memo to ensure a newly-appeared exact
match always wins over a previously memoized variant
- Validate memoized entries still exist on disk before returning them
to prevent stale results when cache dirs are deleted/recreated
* Minor cleanups for cache case resolution
- Use .is_dir() instead of .exists() for exact-case cache check
(cache entries are always directories)
- Remove redundant fallback in _detect_audio_from_tokenizer since
get_cache_path already handles case resolution and returns None
when the model is not cached
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* fix: patch PEFT for Gemma4ClippableLinear in loader checkpoint path
The same Gemma4ClippableLinear monkey-patch that exists in vision.py
for training is needed in loader.py for loading existing checkpoints
(used by export and inference).
Gemma4ClippableLinear wraps nn.Linear but does not subclass it, so
PEFT's LoRA injection fails with "Target module not supported".
The patch redirects PEFT to target the inner .linear child instead.
Applied only to the vision model PeftModel.from_pretrained path.
Temporary fix until PEFT adds native support (peft#3129).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: wrap ClippableLinear patch in try/finally to always restore
Ensures _create_and_replace is restored even if PeftModel.from_pretrained
raises, preventing leaked global state across subsequent model loads.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Gemma-4 support landed in transformers main
(huggingface/transformers#45192). Update the version pin from
5.5.0.dev0 to 5.5.0 across loader, Studio version switcher,
and the MLX installer. Also fix install_gemma4_mlx.sh which
referenced a non-existent v5.5-release branch -- pin it to
the correct commit (91b1ab1) instead.
* UI Changes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove unrelated test file
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix forward compatibility with transformers 5.x
Tested on transformers 4.57.6, 5.3.0, and 5.4.0. All changes are no-ops
on transformers 4.x.
1. Skip exec-based config patching for transformers >= 5.0
Config classes in v5 use @strict, @auto_docstring, and interval()
which break exec(inspect.getsource(...)). Those configs already use
rope_parameters (the v5 replacement for rope_scaling).
2. Slice position_ids to last token in fast_forward_inference
Transformers 5.x generate() accumulates position_ids as
[batch, full_seq_len] across decode steps instead of [batch, 1].
cos[position_ids] then produces the wrong shape for rotary
embeddings. Fixed in llama, qwen3, falcon_h1, gemma2, cohere,
granite. No-op on 4.x since position_ids is already [batch, 1].
3. Handle @strict config kwargs for sequence classification
num_labels, max_position_embeddings, id2label etc. are set on the
config object and passed via config= instead of as kwargs.
AutoModelForSequenceClassification routing added to FastModel loader.
4. Exclude modernbert from flex_attention
ModernBERT with flex_attention hits CUDA illegal memory access in
create_block_mask. Falls back to eager attention safely.
5. Propagate token_type_ids and mm_token_type_ids through GRPO VLM path
Gemma3 Vision requires token_type_ids during training. Qwen3VL
requires mm_token_type_ids for M-RoPE. Extract from inputs in
compute_loss, pass to grpo_accumulated_loss, and extend
mm_token_type_ids for completion tokens in
_generate_and_score_completions.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add try/except safety net around config exec for pre-release transformers versions
* Pop config-level kwargs in seqclass path and use except Exception
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix missing num_items_in_batch in unsloth_prediction_step
unsloth_prediction_step calls compute_loss without num_items_in_batch
during evaluation. This causes _unsloth_pre_compute_loss to see
num_items_in_batch=None, which triggers a spurious warning for every
model when gradient_accumulation_steps > 1:
"Unsloth: Not an error, but {model} does not accept num_items_in_batch.
Using gradient accumulation will be very slightly less accurate."
The standard transformers prediction_step computes num_items_in_batch
via _get_num_items_in_batch before passing it to compute_loss. This
patch does the same in unsloth_prediction_step.
Tested on Llama-3.2-1B-Instruct and Olmo-3-7B-Instruct with
gradient_accumulation_steps=3 and eval_steps=3. Warning is gone and
eval loss is computed correctly for both.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard _get_num_items_in_batch for older transformers versions
_get_num_items_in_batch was added in transformers 4.46. Wrap the call
in try/except so older versions fall back to num_items_in_batch=None,
which preserves the original behavior of not passing it.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix Gemma3N audio training stride assertion with non-reentrant checkpointing
Gemma3N audio conformer processes variable-length audio tensors
that cause stride mismatches in AOT autograd compiled backward
when non-reentrant gradient checkpointing is used. The error
manifests as:
AssertionError: expected size 2==2, stride 1928==1936 at dim=0
This happens because the audio conformer's conv/norm layers produce
tensors whose strides vary with audio clip duration, but AOT autograd
traces the backward graph assuming fixed strides from the first batch.
The notebook sets gradient_checkpointing_kwargs={"use_reentrant": False}
and TRL 0.27.0+ also forces this. Both override Unsloth's own
use_reentrant=True set during prepare_model_for_training.
Fix: intercept gradient_checkpointing_enable on Gemma3N models to
always force use_reentrant=True, regardless of what the notebook
or TRL passes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix Colab huggingface-hub conflict, ensurepip fallback, bump to 2026.3.14
- colab.py / setup.sh: relax == pins to >= when installing studio.txt
on Colab so huggingface-hub does not clobber Colab's bundled version
(breaks transformers is_offline_mode import)
- install_python_stack.py: when uv is unavailable and pip is missing
(uv-created venvs), bootstrap via ensurepip before attempting upgrade
- Bump version to 2026.3.14
- Bump installer min version pins to 2026.3.14
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
