* Fix left-padding masks and positions in batched decode/prefill
* Fix batched generation with left padding
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix attention mask handling, padding_idx zeroing, and Mistral batched generation
1. attention_dispatch.py: Fall back from flash/xformers to SDPA when an
attention_mask is present, since flash attention only supports causal
masking via flag and cannot consume arbitrary padding masks.
2. gemma2.py: Apply attention_mask during decode inference for bsz > 1.
Guard against boolean SWA/GA flags with isinstance check. Slice mask
to match K/V length when sliding window is active. Remove dead
commented-out SDPA branch (SDPA does not support softcapping).
3. granite.py: Apply attention_mask during decode inference for bsz > 1.
Remove dead commented-out SDPA branch and misleading comment.
4. mistral.py: Fix 2D-to-4D padding mask conversion -- convert 0/1 mask
to additive format (0 for keep, -inf for mask) before combining with
the causal mask. Force SDPA backend when attention_mask is present.
5. llama.py: Skip zeroing embed_tokens.weight[padding_idx] when the
embedding is weight-tied to lm_head, since zeroing the shared weight
forces logit(pad) = 0 which is higher than real token logits in models
like Gemma, causing the decoder to emit pad tokens as gibberish. Also
add eos != pad guard, clean up unused _seq_length variable, and fix
get_max_cache_shape handling.
6. vision.py: Same padding_idx fix as llama.py for the vision model
loading path.
Tested on gemma-2b-it, gemma-2-2b-it, Llama-3.2-1B, Mistral-7B-v0.3,
Qwen2.5-0.5B, Qwen3-0.6B with flash-attn 2.8.3 active. All outputs
coherent, zero crashes, zero resize warnings.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Inference path optimizations: eliminate per-layer GPU-CPU sync, cache inspect.signature, add Granite SDPA split
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* More inference path optimizations across model files
- gemma: hoist rotary_seq_len computation to model level (eliminates N
per-layer GPU-CPU syncs from position_ids.max().item()), pre-convert
attention mask to bool once for all layers, use scalar float multiply
instead of torch.tensor allocation for embedding scaling
- gemma2: use in-place tanh_() for softcap attention, use scalar float
multiply for embedding scaling
- granite: pre-convert attention mask to bool once for all layers
- cohere: use in-place neg_() for rotary embedding (consistent with
all other model files)
- falcon_h1: use in-place mul_() for key_multiplier scaling
- llama: use in-place tanh_() for logit softcapping
* Revert scalar multiply for Gemma/Gemma2 embedding scaling
The original torch.tensor(..., dtype=hidden_states.dtype) is intentional:
sqrt(3072) rounds to 55.5 in bfloat16 vs 55.4256 in float32. A plain
scalar multiply may compute at higher precision internally, producing
different results. Restore the explicit dtype-cast tensor to match the
training path in LlamaModel_fast_forward.
* Fix hardcoded cuda:0 device strings and add Cohere .eq(0) bool mask
Replace 15 hardcoded "cuda:0" with f"{DEVICE_TYPE_TORCH}:0" across
gemma.py, gemma2.py, cohere.py, and falcon_h1.py to support multi-GPU
and non-CUDA devices (XPU, etc.). Add .eq(0) bool mask pre-conversion
in CohereModel_fast_forward_inference for batched inference consistency
with llama.py, granite.py, and gemma.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Disable flex_attention for Mllama (Llama 3.2 Vision)
Mllama's _update_causal_mask uses the deprecated make_flex_block_causal_mask
which creates a BlockMask with Q_LEN=KV_LEN=total_seq_len. During decode
with KV cache, q_len=1 but the block_mask still has Q_LEN=total_seq_len,
causing a ValueError. This is an upstream transformers issue -- newer models
use flex_attention_mask from masking_utils which handles decode correctly
via cache_position, but mllama has not been updated yet.
Add mllama to the exclusion list in prefer_flex_attn_if_supported alongside
gpt_oss so it falls back to sdpa, which works correctly for both training
and inference.
* Fix off-by-one in sliding window K/V slicing for gemma2, qwen3, falcon_h1, cohere
The old formula `slicing_tokens = 1 - sliding_window` uses negative indexing
that keeps `sliding_window - 1` tokens instead of `sliding_window`. For example
with sliding_window=32 and kv_seq_len=100, `1-32 = -31` keeps indices 69..99
(31 tokens) instead of the correct 68..99 (32 tokens).
Replace with `start = kv_seq_len - sliding_window` to match the fix already
applied in llama.py and the canonical definition in transformers masking_utils
(sliding_window_overlay: kv_idx > q_idx - W, which keeps exactly W tokens).
Also add attention_mask slicing after K/V trim in qwen3, falcon_h1, and cohere
to prevent mask/K dimension mismatch during batched SDPA inference, matching
the pattern already used in llama.py.
Currently only gemma2 (sliding_window=4096) is actively affected. The other
three models have sliding_window=None in their configs so the code path is
not triggered, but this keeps it correct for any future models that set it.
* Fix Gemma2 softcapping order: apply mask after softcap, not before
The attention mask must be applied AFTER logit softcapping, not before.
Both the Google DeepMind reference implementation (google-deepmind/gemma,
gm/nn/_modules.py lines 254-277) and transformers' eager_attention_forward
(gemma2/modeling_gemma2.py lines 187-193) use this order:
1. logits = Q @ K^T * scale
2. logits = tanh(logits / softcap) * softcap # softcap first
3. logits = logits + mask # mask after
4. probs = softmax(logits)
The PR had the mask addition before softcapping, which causes tanh to
clamp the -inf mask values to -softcap instead of preserving them as -inf
for softmax. While the practical impact is small (masked positions get
~1e-23 probability instead of exact zero), this should match upstream.
* Clarify GQA condition precedence and remove stale comments
Add explicit parentheses to grouped query attention conditions in
llama.py, qwen3.py, granite.py to make operator precedence clear.
The expression `bsz == 1 or not X and Y` relies on Python binding
`not` > `and` > `or` which is correct but easy to misread.
Remove dead commented-out code (`# else: # Knn, Vnn = Knn, Vnn`)
and stale mask comments (`# if attention_mask ...`) from the bsz==1
fast path in llama, qwen3, cohere, falcon_h1, gemma2 inference
functions. These were leftover from the pre-batched-inference
structure and no longer apply.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Allow fp8 for non fast inference
* Extensive fp8 alow and quantizer patch
* Clean up commented-out code, duplicate import, and revert unnecessary Version() changes
- Delete commented-out FP8 fast_inference guard in FastModel (loader.py)
instead of leaving it commented -- matches FastLanguageModel which was
properly deleted
- Delete commented-out fast_inference guard in loader_utils.py
- Remove duplicate `from transformers import GenerationConfig, CompileConfig`
in vision.py (line 112 already imports both plus AutoConfig)
- Revert Version(trl.__version__) back to Version(trl) in trainer.py --
trainer.py imports Version from unsloth_zoo.utils which already handles
module objects
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Add resilience to TRL internal API reclassification
TRL is moving toward v1.0 and will reclassify several
currently-importable symbols as internal with no stability
guarantees. This adds try/except cascading imports with local
fallbacks so Unsloth keeps working regardless of whether TRL
removes, moves, or restructures these symbols.
Changes:
- rl.py: Add try/except cascade for unwrap_model_for_generation
with local contextmanager fallback. Wire sanitize_logprob from
RL_REPLACEMENTS into the compiled trainer template (same pipeline
as selective_log_softmax and other global functions). Add import
math and import logging to the template header.
- rl_replacements.py: Remove inline import of sanitize_logprob
from trl.scripts.vllm_serve in the regex replacement. The
function is now a module-level global in the compiled file.
- tokenizer_utils.py: Wrap dynamic exec import with per-item
fallback so a single removed symbol does not break the entire
bulk import.
Depends on unslothai/unsloth-zoo#516.
Tested across all TRL versions from 0.22.2 through 0.29.0.dev0
(git main). Training losses and grad norms are bit-identical
to unpatched runs.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Warn when save_pretrained_gguf overrides quantization to MXFP4 for GPT-OSS
GPT-OSS only supports MXFP4 format. If the user passes a different
quantization_method, log a warning via logger.warning_once before
overriding. Pass quantization_method=None to suppress the warning.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix Nemotron-H and Nemotron-VL model support
- Add Mamba kernel precision settings for Nemotron-H hybrid models
- Fix VL model auto_model selection for models that only register
AutoModelForCausalLM in their auto_map
- Skip quantization of out_proj for Nemotron-H Mamba layers
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Simplify VLM auto_model selection logic
Reduce three branches to two since the first and third both assign
AutoModelForVision2Seq. The simplified condition checks whether the
auto_map exclusively registers AutoModelForCausalLM without the VLM
class, and defaults to AutoModelForVision2Seq otherwise.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix FP8 model loading for BNB/16-bit: redirect to BF16 sibling
Models like Ministral-3-3B-Instruct-2512 ship with FP8 weights and an FP8
quantization_config in their config.json. Loading these with BNB 4-bit/8-bit
fails because BNB cannot quantize FP8 tensors. Loading with 16-bit also fails
because the FP8 quantization config has activation_scheme=static which is
unsupported by transformers' FineGrainedFP8Config.
When an FP8 model is detected and the user is not explicitly requesting FP8
loading, check if a BF16 sibling repo exists (model_name + "-BF16") and
redirect to it. This happens early in the loading flow before any quantization
config processing.
Also pass the modified model_config to auto_model.from_pretrained to avoid
transformers re-reading the original config from the model repo.
Tested with Ministral-3-3B in 4-bit and 16-bit modes. Both now load and
train correctly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Simplify FP8 condition and narrow exception handling
Simplify the load_in_fp8 check (works for bool and string values).
Narrow inner except to KeyError and add comment for outer except.
* Warn user when FP8 model has no BF16 sibling for redirect
Previously the except block silently fell through with `pass`,
so users would get a confusing BNB dtype error later. Now prints
a clear message explaining the FP8 situation and suggesting
load_in_fp8=True or uploading a BF16 version.
* Fix FP8 redirect state corruption and add fbgemm_fp8 support
- Fix state corruption: model_name was reassigned before
AutoConfig.from_pretrained, so if config fetch failed,
model_name pointed to BF16 repo while auto_config still
had FP8. Now only updates state after both checks succeed.
- Save original model_name so warning message is correct
even on failure.
- Handle fbgemm_fp8 quant method in addition to fp8.
* Extract FP8 redirect to shared _redirect_fp8_to_bf16() in _utils.py
Addresses reviewer feedback:
- Move FP8 redirect logic to a shared function callable from both
vision.py (FastBaseModel) and llama.py (FastLlamaModel)
- Raise RuntimeError instead of warning when BF16 sibling not found
- Add FP8 redirect to llama.py for text-only model loading path
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add Ministral 3B/8B/14B mapper entries
Adds all 9 Ministral model variants to the mapper:
- Instruct (3B, 8B, 14B) with FP8 variant mappings
- Base (3B, 8B, 14B)
- Reasoning (3B, 8B, 14B)
This routes mistralai/Ministral-* to unsloth/Ministral-* repos
(BF16 weights), which also avoids the FP8 config issue for the
standard loading path through loader.py.
* Add FP8 mapper entries for Mistral-Small-3.2 and Magistral-Small-2509
---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-253.us-east-2.compute.internal>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Suppress FBGEMM CUTLASS "Arch conditional MMA" stdout spam on Blackwell GPUs
On Blackwell GPUs (B200/B100, SM100), FBGEMM's f8f8bf16_blockwise kernel
is hardcoded to cutlass::arch::Sm90 with no SM100 code path. When
test_has_fbgemm() probes this kernel, it fires 2304 "ERROR : Arch
conditional MMA instruction used without targeting appropriate compute
capability" lines before aborting and returning zeros.
The existing HidePrintMessage filter on sys.stderr (line 109) does not
catch these because CUDA device-side printf writes to stdout fd 1 at the
C level, bypassing Python's sys.stdout/sys.stderr entirely.
Fix: add suppress_cuda_printf() context manager in import_fixes.py that
redirects fd 1 and fd 2 to /dev/null at the OS level, with
torch.cuda.synchronize() and libc fflush before restoring. Wrap the
test_has_fbgemm() call in fp8.py with this context manager.
Tested on B200 with fbgemm-gpu-genai 1.4.0+cu130 and 1.5.0+cu130:
- Before: 2304 warning lines on every import
- After: 0 warning lines
- UNSLOTH_HAS_FBGEMM correctly set to 0 (Triton fallback works)
- Works with both UNSLOTH_ENABLE_LOGGING=0 and =1
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard _libc init and fflush to prevent fd leak on failure
---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-253.us-east-2.compute.internal>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix VLM processor load degradation and vLLM CUDA version detection
vision.py - Fix VLM processor load for issue #4085:
- Before loading the processor, scan local config files and strip the
_Unsloth_Patched_ prefix. AutoProcessor.from_pretrained silently
degrades to a text-only tokenizer instead of raising an exception
when it encounters the unrecognized class name, so the existing
get_auto_processor fallback never triggers. Sanitizing the configs
before loading fixes backwards compat for old corrupted saves.
- After loading, detect when AutoProcessor returned a text-only
tokenizer for a VLM model (has no image_processor attribute) and
trigger the manual fallback constructor.
import_fixes.py - Fix vLLM CUDA version mismatch detection:
- _is_broken_vllm_error now also matches CUDA shared library errors
(libcudart, libcublas, libnvrtc) with "cannot open shared object
file". Previously it only matched errors containing "vllm._c" in
the message text, which missed cases where the error message was
about the missing CUDA library itself (e.g. vllm built for CUDA 12
on a CUDA 13 system).
- New _get_vllm_cuda_mismatch_message function extracts the CUDA
version from the error, compares to the system CUDA version via
torch.version.cuda, and returns a targeted install command using
the correct GitHub releases wheel URL.
- disable_broken_vllm uses the targeted message when a CUDA mismatch
is detected, falling back to the existing generic message otherwise.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-253.us-east-2.compute.internal>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add `datasets` metadata support to model cards
Add an optional `datasets` parameter to all save/push functions so users
can specify which datasets were used for training. The metadata is set
via `ModelCard.data.datasets` for standard paths and via
`metadata_update` for GGUF and generic save paths.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix datasets metadata for existing repos, add token, improve errors
- Add metadata_update fallback in create_huggingface_repo and
upload_to_huggingface so datasets metadata is set even when the
repo already exists (previously only worked on first creation).
- Pass token=token to all metadata_update calls so they work
without a global HF login.
- Replace silent except:pass with logger.warning_once for
metadata failures so users know if something went wrong.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix generic datasets metadata repo resolution for PR #4076
* Fix create_huggingface_repo username resolution for PR #4076
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
* FP8 per tensor quant support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix security-regression fallout in chat templates and PDL patching
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop security regression test files from PR scope
* Apply suggestion from @danielhanchen
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Guard optional vLLM imports when extension is broken
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove vLLM import guard tests from PR scope
* Block broken vLLM imports like causal_conv1d
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Raise ImportError for stable torchvision mismatches
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove torchvision compatibility tests from PR scope
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
When using device_map='balanced' with multiple GPUs, the labels tensor
may reside on a different device than the logits/losses tensors. This
causes a RuntimeError at the masked_fill_ call in the chunked
cross-entropy forward path.
Fix: explicitly move labels to the same device as logits at the start
of Fast_CrossEntropyLoss.forward(). This is a no-op on single-GPU
setups.
Fixes#4041
* Auto-configure AMDGPU_ASIC_ID_TABLE_PATH on ROCm startup
* Remove ROCm fd2 amdgpu.ids noise filter wrappers
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use PyPI bitsandbytes for amd extra to avoid malformed wheel URL
* Add amd-preview extra for bitsandbytes continuous wheel channel
* Keep amd extra on bitsandbytes>=0.49.1 and remove amd-preview
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Wrap unsloth_zoo import with HIP amdgpu.ids filter
* Refactor ROCm ids filter helpers for readability
* Rename ROCm ids filter helper and annotate call sites
* Remove obsolete amdgpu ids filter alias
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
MI355X (gfx950) has the same 1024-thread workgroup limit as MI300X (gfx942),
but was missing from is_cdna(), causing all Triton kernels to use num_warps=32
(2048 threads) instead of 16 (1024 threads), resulting in OutOfResources crash.
Tested on: 8x AMD Instinct MI355X (gfx950), ROCm 7.1
* Suppress HIP libdrm stderr noise in causal_conv1d probe
* Broaden HIP libdrm stderr suppression for early ROCm startup
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
* Improve HIP GPU name reporting in startup banner
* Drop MI300X arch suffix in banner name
* Normalize _utils.py file mode
* Simplify FA2 fallback text and filter AMD ids noise
* Strip trailing GPU arch suffix via regex
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use gfx lookup default and normalize Ryzen AI naming
* Remove name-path Ryzen AI normalization
* Expand ROCm gfx map to full documented GPU name aliases
* Simplify HIP fallback naming to AMD gfx token
* Remove Ryzen Al torch_name normalization
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Improve HIP GPU name reporting in startup banner
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Handle broken causal_conv1d import at runtime
Add a startup import-time probe for causal_conv1d and disable the fast path when the shared library is ABI broken. This keeps Falcon H1/model loading resilient without requiring env flags.
- Add disable_broken_causal_conv1d in import_fixes.
- Invoke it early from unsloth/__init__ during package init.
- Make Falcon H1 optional imports in loader and models/__init__ soft-fail instead of failing hard.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Enforce unavailable semantics for broken causal_conv1d
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove Falcon H1 import swallowing
* Restore optional Falcon H1 import guard
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove causal_conv1d regression tests
* Trim FA2 fallback messaging
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* convert print to logger
* Print but cleaner
* Hide model on multiple devices
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix typo
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix typo transfomers -> transformers, revert MoE message change
* Update MoE detection message to show num_experts and target_modules
* Fix llama-cli path in save info message
* target_parameters warning for moe
* fix should_convert_module for llm_int8_skip_modules
* fix should_convert_module for llm_int8_skip_modules
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Logging filters
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* negation
* remove should_convert_module patch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
* Fix warmup_ratio deprecation warning for transformers >= 5.0
In transformers 5.0, warmup_ratio is deprecated in favor of
warmup_steps which now accepts float values (< 1 = ratio,
>= 1 = absolute steps).
The compiler now conditionally sets warmup_steps=0.1 on
transformers >= 5.0 (same semantics as warmup_ratio=0.1) and
keeps warmup_ratio=0.1 on older versions where warmup_steps
only accepts int.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Inject token_type_ids for Gemma3 multimodal training on transformers 5.x
In transformers 5.x, create_causal_mask_mapping() raises ValueError when
is_training=True and token_type_ids is None. When doing text-only SFT on
Gemma3 4B (a multimodal model), the dataset_utils detection for
_needs_token_type_ids can miss because:
- The model is wrapped in PeftModel, so type(model).__module__ points to
peft.peft_model instead of transformers
- The processing_class is a tokenizer (not Gemma3Processor), so the
fallback MRO check resolves to a module without create_causal_mask_mapping
This adds a fallback in _unsloth_pre_compute_loss that injects
token_type_ids=zeros when:
1. token_type_ids is not already in inputs
2. The inner model config has model_type "gemma3"
3. The model's module has create_causal_mask_mapping (transformers 5.x)
4. The model is in training mode
On transformers 4.x, create_causal_mask_mapping does not exist so this
check is inert.
Depends on: unslothai/unsloth-zoo#488
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* FP8: Load model on-the-fly in vLLM
**Summary:** Existing support for `load_in_fp8=True` performs
an offline quantization when loading the initial model.
This is no longer necessary as of vllm==0.12.0 (after
https://github.com/vllm-project/vllm/pull/23014), where we
can quantize the model on-the-fly when we load it:
```
llm = LLM(
...
hf_overrides={
"quantization_config_dict_str": json.dumps(torchao_config),
},
)
```
**Note:** Needs https://github.com/unslothai/unsloth-zoo/pull/380
**Test Plan:**
https://gist.github.com/andrewor14/5b85119fae46845d07b608d420907423
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix on-the-fly FP8: always check mapper first, fallback to on-the-fly
The original implementation bypasses the FP8 mapper entirely for
vllm >= 0.12.0, meaning models like Llama-3.2-1B-Instruct and Qwen3-8B
that have pre-quantized FP8-Block/FP8 checkpoints would never use them.
This fixes the priority order:
1. Mapper has a pre-quantized model -> use it (always)
2. Mapper has no match + vllm >= 0.12.0 -> on-the-fly FP8 via torchao
3. Mapper has no match + vllm < 0.12.0 -> offline quantization
Changes:
- loader_utils.py: Move vllm >= 0.12.0 check after mapper lookups
- loader.py: Set load_in_fp8=False when mapper resolves to a
pre-quantized model to prevent double quantization
Tested on B200 with Llama-3.2-1B-Instruct and Qwen3-8B. Corrected code
produces results matching baseline (pre-quantized path preserved).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
* convert print to logger
* Print but cleaner
* Hide model on multiple devices
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix typo
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix typo transfomers -> transformers, revert MoE message change
* Update MoE detection message to show num_experts and target_modules
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
* Fix#3397: Prevent trainer tokenization hang with safe num_proc
* Fix#3397: Add missing import sys for Windows-safe tokenization
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Consolidate with existing num_proc guard in dataset_utils.py
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
* Fix EmbeddingGemma float16 NaN by adding gemma3_text to FORCE_FLOAT32 and SDPA lists
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Inject model reference for dynamic token_type_ids detection in SFTTrainer
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Suppress vLLM v1 executor sleep/wake log messages
Add HideLoggingMessage filters for vllm.v1.executor.abstract logger to
suppress repetitive sleep/wake INFO and WARNING messages that spam training
output when UNSLOTH_VLLM_STANDBY is enabled. The existing filter at line 275
handles the legacy vllm.executor.executor_base path; this adds coverage for
the v1 engine path used by vllm 0.11+.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
