* Refactor loss computation to include completion_mask
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fixes for trl 0.28 and above
Remove sync/reload weights calls , remove vllm.LLM instantiation
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Refactor loss computation to include completion_mask
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fixes for trl 0.28 and above
Remove sync/reload weights calls , remove vllm.LLM instantiation
* patch rpc in openenv for newer trl
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pluesclues <136766175+pluesclues@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix gpt temporary patch for grpo to happen after compile
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Refactor loss computation to include completion_mask
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
trl/trainer/callbacks.py imports is_wandb_available from
accelerate.utils, not from transformers. The original fix in #4147
only patched the transformers version, so `from trl import GRPOTrainer`
still crashed via the callbacks.py -> accelerate -> wandb path.
Must patch both the source module (accelerate.utils.imports) AND the
re-export namespace (accelerate.utils) since Python's
`from accelerate.utils import X` reads from the latter, which holds
its own cached reference.
* Fix broken wandb import crashing unsloth startup
When wandb is installed but broken (e.g., wandb < 0.19.11 with
protobuf >= 6.0), the import chain unsloth -> trl -> transformers ->
is_wandb_available() -> import wandb crashes with:
ImportError: cannot import name 'Imports' from
'wandb.proto.wandb_telemetry_pb2'
This happens because transformers' is_wandb_available() has no
try/except around `import wandb`. The error propagates up and kills
`from unsloth import FastLanguageModel` even though wandb is optional.
Add disable_broken_wandb() following the same pattern as
disable_torchcodec_if_broken(). It proactively tries importing wandb
during early init, and if the import fails, patches
is_wandb_available() to return False and sets WANDB_DISABLED=true.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fixup mapper issues and resolve properly
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix: update GGUF save paths to use ~/.unsloth/llama.cpp with Windows support
* fix: quote LLAMA_CPP_DEFAULT_DIR in fallback shell commands to handle paths with spaces
* refactor: deduplicate platform-specific build instructions in quantization error message
* chore: remove accidentally committed PR description file
* Fix import safety and f-string bugs in save.py
- H4: Add defensive try/except for LLAMA_CPP_DEFAULT_DIR and IS_WINDOWS imports
with fallback defaults, so save.py works even if zoo PR #526 is not merged yet
- H5: Fix Kaggle error path using plain "Error: {e}" instead of f"Error: {e}",
so the actual exception is shown to users
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Datta Nimmaturi <venkatadattasainimmaturi@gmail.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix lm_head lora save
* Fix _need_to_train_embeddings guard for lm_head LoRA targets
When lm_head is already in final_modules as a LoRA target, the
_need_to_train_embeddings block should not also add it to
modules_to_save. This prevents dual-wrapping (LoRA + modules_to_save
on the same module) which causes assertion failures downstream.
Check if embed_tokens/lm_head are already being trained as LoRA
targets before adding them to modules_to_save. Also prevents
duplicate entries with elif guards.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Current arch.startswith("gfx1") incorrectly matches:
- RDNA1 (gfx10xx) and RDNA2 (gfx103x): not ROCm supported
- gfx1102 (RX 7600), gfx1103 (Phoenix APU): not in ROCm support matrix
- gfx1150/1151/1152 (RDNA3.5 APUs): not in ROCm support matrix
Replace with explicit whitelist aligned to the ROCm Linux support matrix:
https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html
gfx1100 - RDNA3 discrete (RX 7900 series, PRO W7900/W7800)
gfx1101 - RDNA3 discrete (RX 7800/7700 series, PRO W7700)
gfx1200 - RDNA4 discrete (RX 9060 series)
gfx1201 - RDNA4 discrete (RX 9070 series, AI PRO R9700)
Mirrors the existing is_cdna() pattern. Avoids silently applying
unverified Triton kernel tuning to unsupported hardware.
Fix global dequantize buffer dtype mismatch when loading multiple 4-bit models with different dtypes in the same process. Adds dtype check alongside existing None check for WEIGHT_BUFFER in both CUDA/HIP and XPU paths.
Use 16 warps for RDNA in the chunked cross-entropy forward kernel
(large vocab > 65536), matching the existing CDNA optimization.
Benchmarked on W7900 (gfx1100) with actual unsloth kernels (5 trials, median):
- Chunked CE forward (BS=65536): 16 warps = 2.4-2.6x faster than 32
- All other kernels (LayerNorm, RoPE, SwiGLU): default heuristic is
already optimal for RDNA; no modification needed.
Depends on: #4109 (provides is_rdna() detection)
TMA (Tensor Memory Accelerator) is an NVIDIA Hopper+ feature that does
not exist on AMD GPUs. However, _check_tma_support() incorrectly
returns True on ROCm because:
1. torch.cuda.get_device_capability() returns (11, 0) for gfx1100,
satisfying the >= 9 check intended for Hopper (sm_90).
2. ROCm Triton exports tl.make_tensor_descriptor (the symbol exists
even though the hardware does not support TMA).
This would cause MoE grouped_gemm to attempt TMA operations on AMD
GPUs, leading to runtime failures.
Fix: early-return False for HIP devices, matching the existing XPU
guard.
* fix(Triton): ensure float32 eps in RMS LayerNorm rsqrt for HIP/ROCm
On HIP (AMD ROCm), Triton constexpr eps may not promote to float32
in rsqrt, causing numerical instability (NaN/Inf) on RDNA GPUs
(gfx1100, gfx1151 Strix Halo, etc.).
Use tl.full((), eps, tl.float32) to explicitly create a float32
scalar before adding to row_var in rsqrt. Applied to both standard
and Gemma RMS LayerNorm forward kernels.
Tested on W7900 (gfx1100): full test suite passed (dim 512-2048,
bf16/fp16, various seqlen).
Related: #3385, #3588
* Apply same float32 eps fix to layernorm.py for PR #4110
layernorm.py has the identical tl.constexpr eps pattern in
layernorm_forward that can misfire on HIP/ROCm. Apply the same
tl.full((), eps, tl.float32) fix for consistency.
Both testing_suite_layernorm (standard LayerNorm) and
testing_suite_layernorm (RMS LayerNorm) pass on NVIDIA after
this change.
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* fix(ROCm): comprehensive RDNA GPU support - fix Gemma3 NaN & add is_rdna()
- Add is_rdna() detection for RDNA3/3.5/RDNA4 consumer GPUs (gfx11xx, gfx1151, gfx12xx)
- Disable torch.compile for Gemma3 on HIP to fix NaN loss (fixes#3385, #4029)
- Export is_cdna/is_rdna from kernels for downstream use
- Import is_rdna into cross_entropy_loss for future RDNA-specific tuning
Tested on AMD Radeon PRO W7900 (gfx1100) with ROCm 7.1:
✓ Gemma3-1B: loss 3.37→3.25 (no NaN)
✓ Llama-3.2-1B: loss 2.44→2.37 (no NaN)
✓ Qwen2.5-1.5B: loss 1.89→1.85 (no NaN)
✓ RMS LayerNorm Triton kernel: bf16/fp16 PASSED
✓ Cross Entropy Loss Triton kernel: 32K/256K vocab PASSED
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: scope compile disable to RDNA only, use partial mode, remove unused import
Changes based on Daniel's review:
1. (HIGH) Replace DEVICE_TYPE=='hip' with is_rdna() to avoid disabling
torch.compile on CDNA GPUs (MI250X/MI300X/MI350) where it works fine
2. (MEDIUM) Use 'partial' instead of '1' for UNSLOTH_COMPILE_DISABLE to
only disable model forward compilation while keeping loss compilation,
matching the existing Sesame pattern
3. (LOW) Remove unused is_rdna import from cross_entropy_loss.py (F401)
* Remove redundant is_cdna/is_rdna exports from kernels/__init__.py
These functions are imported directly from .utils where needed
(e.g. cross_entropy_loss.py, loader.py). No external code imports
them from the unsloth.kernels namespace.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The function (introduced in #3923) assumed that the absence of
`triton.runtime.triton_key` on ROCm means torch.compile will crash.
Investigation shows this is incorrect:
1. `triton.runtime.triton_key` was renamed/removed in the ROCm Triton
fork — it does not exist at that path. However,
`triton.compiler.compiler.triton_key` (the path torch._inductor
actually imports) EXISTS and works correctly on ROCm.
2. Both call-sites in torch._inductor (codecache.py and
async_compile.py) already wrap the import in try/except, so even a
genuinely missing triton_key would be handled gracefully.
3. Comprehensive testing on ROCm 7.1 + Triton 3.4.0 + gfx1100 confirms
torch.compile works correctly for matmul, cross-entropy, RMSNorm,
multi-layer transformer forward+backward, and LoRA — all without
triton.runtime.triton_key.
The original code was also ineffective (environment variables set after
torch import have no effect on torch._dynamo config), so removing it
has zero behavioral change on existing installations.
Supersedes the compile-disable portion of #3923.
* Fix transformers v5 RoPE inv_freq corruption during model loading
Transformers v5 initializes models on the meta device, then
_move_missing_keys_from_meta_to_device() replaces all non-persistent
buffers with torch.empty_like() (uninitialized memory). Vanilla
transformers restores inv_freq via _init_weights() checking for
original_inv_freq, but Unsloth's LlamaRotaryEmbedding subclasses
lack this attribute, so inv_freq stays corrupted with garbage values.
This caused 5-11x higher training loss on transformers v5 for all
models using Unsloth's rope (Llama 3.x, Qwen3, Mistral, TinyLlama,
Granite). Models using native transformers rope (Gemma, Phi-4,
Falcon-H1) were unaffected.
The fix recomputes inv_freq from the stored base/dim after model
loading, applies model-specific scaling via _apply_inv_freq_scaling(),
and rebuilds cos/sin caches. Also handles LongRopeRotaryEmbedding
(Phi-3.5 style short/long inv_freq). Guarded by transformers >= 5.0.0
so it is a no-op on v4.
Tested on: Llama 3.1 8B, Llama 3.2 3B, Qwen3 14B, Qwen3 4B, Phi-4,
TinyLlama, Mistral 7B, Gemma2 2B, Falcon-H1 -- all v5 losses now
match v4 baselines to < 0.004 absolute difference.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Unpack BatchEncoding in generate() for v4/v5 backwards compatibility
Old notebooks pass the full tokenizer output as input_ids:
inputs = tokenizer(..., return_tensors="pt").to("cuda")
model.generate(input_ids=inputs, ...)
This worked on transformers v4 because generate() internally
extracted the tensor. Transformers v5 calls .shape on input_ids
directly, which crashes since BatchEncoding has no .shape attribute.
Fix: in unsloth_fast_generate(), detect when input_ids is a dict-like
object (BatchEncoding) and unpack its contents into separate kwargs
before forwarding to the underlying generate(). This makes both old
and new notebook patterns work on both v4 and v5.
* Remove redundant seen_ids dedup in _fix_rope_inv_freq
named_modules() already deduplicates with remove_duplicate=True (default).
Also clarify that native v5 rotary classes (Gemma3 etc.) have original_inv_freq
which transformers v5's _init_weights() uses to restore inv_freq, so they do
not need this fix.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix left-padding masks and positions in batched decode/prefill
* Fix batched generation with left padding
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix attention mask handling, padding_idx zeroing, and Mistral batched generation
1. attention_dispatch.py: Fall back from flash/xformers to SDPA when an
attention_mask is present, since flash attention only supports causal
masking via flag and cannot consume arbitrary padding masks.
2. gemma2.py: Apply attention_mask during decode inference for bsz > 1.
Guard against boolean SWA/GA flags with isinstance check. Slice mask
to match K/V length when sliding window is active. Remove dead
commented-out SDPA branch (SDPA does not support softcapping).
3. granite.py: Apply attention_mask during decode inference for bsz > 1.
Remove dead commented-out SDPA branch and misleading comment.
4. mistral.py: Fix 2D-to-4D padding mask conversion -- convert 0/1 mask
to additive format (0 for keep, -inf for mask) before combining with
the causal mask. Force SDPA backend when attention_mask is present.
5. llama.py: Skip zeroing embed_tokens.weight[padding_idx] when the
embedding is weight-tied to lm_head, since zeroing the shared weight
forces logit(pad) = 0 which is higher than real token logits in models
like Gemma, causing the decoder to emit pad tokens as gibberish. Also
add eos != pad guard, clean up unused _seq_length variable, and fix
get_max_cache_shape handling.
6. vision.py: Same padding_idx fix as llama.py for the vision model
loading path.
Tested on gemma-2b-it, gemma-2-2b-it, Llama-3.2-1B, Mistral-7B-v0.3,
Qwen2.5-0.5B, Qwen3-0.6B with flash-attn 2.8.3 active. All outputs
coherent, zero crashes, zero resize warnings.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Inference path optimizations: eliminate per-layer GPU-CPU sync, cache inspect.signature, add Granite SDPA split
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* More inference path optimizations across model files
- gemma: hoist rotary_seq_len computation to model level (eliminates N
per-layer GPU-CPU syncs from position_ids.max().item()), pre-convert
attention mask to bool once for all layers, use scalar float multiply
instead of torch.tensor allocation for embedding scaling
- gemma2: use in-place tanh_() for softcap attention, use scalar float
multiply for embedding scaling
- granite: pre-convert attention mask to bool once for all layers
- cohere: use in-place neg_() for rotary embedding (consistent with
all other model files)
- falcon_h1: use in-place mul_() for key_multiplier scaling
- llama: use in-place tanh_() for logit softcapping
* Revert scalar multiply for Gemma/Gemma2 embedding scaling
The original torch.tensor(..., dtype=hidden_states.dtype) is intentional:
sqrt(3072) rounds to 55.5 in bfloat16 vs 55.4256 in float32. A plain
scalar multiply may compute at higher precision internally, producing
different results. Restore the explicit dtype-cast tensor to match the
training path in LlamaModel_fast_forward.
* Fix hardcoded cuda:0 device strings and add Cohere .eq(0) bool mask
Replace 15 hardcoded "cuda:0" with f"{DEVICE_TYPE_TORCH}:0" across
gemma.py, gemma2.py, cohere.py, and falcon_h1.py to support multi-GPU
and non-CUDA devices (XPU, etc.). Add .eq(0) bool mask pre-conversion
in CohereModel_fast_forward_inference for batched inference consistency
with llama.py, granite.py, and gemma.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Disable flex_attention for Mllama (Llama 3.2 Vision)
Mllama's _update_causal_mask uses the deprecated make_flex_block_causal_mask
which creates a BlockMask with Q_LEN=KV_LEN=total_seq_len. During decode
with KV cache, q_len=1 but the block_mask still has Q_LEN=total_seq_len,
causing a ValueError. This is an upstream transformers issue -- newer models
use flex_attention_mask from masking_utils which handles decode correctly
via cache_position, but mllama has not been updated yet.
Add mllama to the exclusion list in prefer_flex_attn_if_supported alongside
gpt_oss so it falls back to sdpa, which works correctly for both training
and inference.
* Fix off-by-one in sliding window K/V slicing for gemma2, qwen3, falcon_h1, cohere
The old formula `slicing_tokens = 1 - sliding_window` uses negative indexing
that keeps `sliding_window - 1` tokens instead of `sliding_window`. For example
with sliding_window=32 and kv_seq_len=100, `1-32 = -31` keeps indices 69..99
(31 tokens) instead of the correct 68..99 (32 tokens).
Replace with `start = kv_seq_len - sliding_window` to match the fix already
applied in llama.py and the canonical definition in transformers masking_utils
(sliding_window_overlay: kv_idx > q_idx - W, which keeps exactly W tokens).
Also add attention_mask slicing after K/V trim in qwen3, falcon_h1, and cohere
to prevent mask/K dimension mismatch during batched SDPA inference, matching
the pattern already used in llama.py.
Currently only gemma2 (sliding_window=4096) is actively affected. The other
three models have sliding_window=None in their configs so the code path is
not triggered, but this keeps it correct for any future models that set it.
* Fix Gemma2 softcapping order: apply mask after softcap, not before
The attention mask must be applied AFTER logit softcapping, not before.
Both the Google DeepMind reference implementation (google-deepmind/gemma,
gm/nn/_modules.py lines 254-277) and transformers' eager_attention_forward
(gemma2/modeling_gemma2.py lines 187-193) use this order:
1. logits = Q @ K^T * scale
2. logits = tanh(logits / softcap) * softcap # softcap first
3. logits = logits + mask # mask after
4. probs = softmax(logits)
The PR had the mask addition before softcapping, which causes tanh to
clamp the -inf mask values to -softcap instead of preserving them as -inf
for softmax. While the practical impact is small (masked positions get
~1e-23 probability instead of exact zero), this should match upstream.
* Clarify GQA condition precedence and remove stale comments
Add explicit parentheses to grouped query attention conditions in
llama.py, qwen3.py, granite.py to make operator precedence clear.
The expression `bsz == 1 or not X and Y` relies on Python binding
`not` > `and` > `or` which is correct but easy to misread.
Remove dead commented-out code (`# else: # Knn, Vnn = Knn, Vnn`)
and stale mask comments (`# if attention_mask ...`) from the bsz==1
fast path in llama, qwen3, cohere, falcon_h1, gemma2 inference
functions. These were leftover from the pre-batched-inference
structure and no longer apply.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Allow fp8 for non fast inference
* Extensive fp8 alow and quantizer patch
* Clean up commented-out code, duplicate import, and revert unnecessary Version() changes
- Delete commented-out FP8 fast_inference guard in FastModel (loader.py)
instead of leaving it commented -- matches FastLanguageModel which was
properly deleted
- Delete commented-out fast_inference guard in loader_utils.py
- Remove duplicate `from transformers import GenerationConfig, CompileConfig`
in vision.py (line 112 already imports both plus AutoConfig)
- Revert Version(trl.__version__) back to Version(trl) in trainer.py --
trainer.py imports Version from unsloth_zoo.utils which already handles
module objects
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Add resilience to TRL internal API reclassification
TRL is moving toward v1.0 and will reclassify several
currently-importable symbols as internal with no stability
guarantees. This adds try/except cascading imports with local
fallbacks so Unsloth keeps working regardless of whether TRL
removes, moves, or restructures these symbols.
Changes:
- rl.py: Add try/except cascade for unwrap_model_for_generation
with local contextmanager fallback. Wire sanitize_logprob from
RL_REPLACEMENTS into the compiled trainer template (same pipeline
as selective_log_softmax and other global functions). Add import
math and import logging to the template header.
- rl_replacements.py: Remove inline import of sanitize_logprob
from trl.scripts.vllm_serve in the regex replacement. The
function is now a module-level global in the compiled file.
- tokenizer_utils.py: Wrap dynamic exec import with per-item
fallback so a single removed symbol does not break the entire
bulk import.
Depends on unslothai/unsloth-zoo#516.
Tested across all TRL versions from 0.22.2 through 0.29.0.dev0
(git main). Training losses and grad norms are bit-identical
to unpatched runs.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Warn when save_pretrained_gguf overrides quantization to MXFP4 for GPT-OSS
GPT-OSS only supports MXFP4 format. If the user passes a different
quantization_method, log a warning via logger.warning_once before
overriding. Pass quantization_method=None to suppress the warning.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix Nemotron-H and Nemotron-VL model support
- Add Mamba kernel precision settings for Nemotron-H hybrid models
- Fix VL model auto_model selection for models that only register
AutoModelForCausalLM in their auto_map
- Skip quantization of out_proj for Nemotron-H Mamba layers
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Simplify VLM auto_model selection logic
Reduce three branches to two since the first and third both assign
AutoModelForVision2Seq. The simplified condition checks whether the
auto_map exclusively registers AutoModelForCausalLM without the VLM
class, and defaults to AutoModelForVision2Seq otherwise.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix FP8 model loading for BNB/16-bit: redirect to BF16 sibling
Models like Ministral-3-3B-Instruct-2512 ship with FP8 weights and an FP8
quantization_config in their config.json. Loading these with BNB 4-bit/8-bit
fails because BNB cannot quantize FP8 tensors. Loading with 16-bit also fails
because the FP8 quantization config has activation_scheme=static which is
unsupported by transformers' FineGrainedFP8Config.
When an FP8 model is detected and the user is not explicitly requesting FP8
loading, check if a BF16 sibling repo exists (model_name + "-BF16") and
redirect to it. This happens early in the loading flow before any quantization
config processing.
Also pass the modified model_config to auto_model.from_pretrained to avoid
transformers re-reading the original config from the model repo.
Tested with Ministral-3-3B in 4-bit and 16-bit modes. Both now load and
train correctly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Simplify FP8 condition and narrow exception handling
Simplify the load_in_fp8 check (works for bool and string values).
Narrow inner except to KeyError and add comment for outer except.
* Warn user when FP8 model has no BF16 sibling for redirect
Previously the except block silently fell through with `pass`,
so users would get a confusing BNB dtype error later. Now prints
a clear message explaining the FP8 situation and suggesting
load_in_fp8=True or uploading a BF16 version.
* Fix FP8 redirect state corruption and add fbgemm_fp8 support
- Fix state corruption: model_name was reassigned before
AutoConfig.from_pretrained, so if config fetch failed,
model_name pointed to BF16 repo while auto_config still
had FP8. Now only updates state after both checks succeed.
- Save original model_name so warning message is correct
even on failure.
- Handle fbgemm_fp8 quant method in addition to fp8.
* Extract FP8 redirect to shared _redirect_fp8_to_bf16() in _utils.py
Addresses reviewer feedback:
- Move FP8 redirect logic to a shared function callable from both
vision.py (FastBaseModel) and llama.py (FastLlamaModel)
- Raise RuntimeError instead of warning when BF16 sibling not found
- Add FP8 redirect to llama.py for text-only model loading path
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add Ministral 3B/8B/14B mapper entries
Adds all 9 Ministral model variants to the mapper:
- Instruct (3B, 8B, 14B) with FP8 variant mappings
- Base (3B, 8B, 14B)
- Reasoning (3B, 8B, 14B)
This routes mistralai/Ministral-* to unsloth/Ministral-* repos
(BF16 weights), which also avoids the FP8 config issue for the
standard loading path through loader.py.
* Add FP8 mapper entries for Mistral-Small-3.2 and Magistral-Small-2509
---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-253.us-east-2.compute.internal>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Suppress FBGEMM CUTLASS "Arch conditional MMA" stdout spam on Blackwell GPUs
On Blackwell GPUs (B200/B100, SM100), FBGEMM's f8f8bf16_blockwise kernel
is hardcoded to cutlass::arch::Sm90 with no SM100 code path. When
test_has_fbgemm() probes this kernel, it fires 2304 "ERROR : Arch
conditional MMA instruction used without targeting appropriate compute
capability" lines before aborting and returning zeros.
The existing HidePrintMessage filter on sys.stderr (line 109) does not
catch these because CUDA device-side printf writes to stdout fd 1 at the
C level, bypassing Python's sys.stdout/sys.stderr entirely.
Fix: add suppress_cuda_printf() context manager in import_fixes.py that
redirects fd 1 and fd 2 to /dev/null at the OS level, with
torch.cuda.synchronize() and libc fflush before restoring. Wrap the
test_has_fbgemm() call in fp8.py with this context manager.
Tested on B200 with fbgemm-gpu-genai 1.4.0+cu130 and 1.5.0+cu130:
- Before: 2304 warning lines on every import
- After: 0 warning lines
- UNSLOTH_HAS_FBGEMM correctly set to 0 (Triton fallback works)
- Works with both UNSLOTH_ENABLE_LOGGING=0 and =1
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard _libc init and fflush to prevent fd leak on failure
---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-253.us-east-2.compute.internal>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix VLM processor load degradation and vLLM CUDA version detection
vision.py - Fix VLM processor load for issue #4085:
- Before loading the processor, scan local config files and strip the
_Unsloth_Patched_ prefix. AutoProcessor.from_pretrained silently
degrades to a text-only tokenizer instead of raising an exception
when it encounters the unrecognized class name, so the existing
get_auto_processor fallback never triggers. Sanitizing the configs
before loading fixes backwards compat for old corrupted saves.
- After loading, detect when AutoProcessor returned a text-only
tokenizer for a VLM model (has no image_processor attribute) and
trigger the manual fallback constructor.
import_fixes.py - Fix vLLM CUDA version mismatch detection:
- _is_broken_vllm_error now also matches CUDA shared library errors
(libcudart, libcublas, libnvrtc) with "cannot open shared object
file". Previously it only matched errors containing "vllm._c" in
the message text, which missed cases where the error message was
about the missing CUDA library itself (e.g. vllm built for CUDA 12
on a CUDA 13 system).
- New _get_vllm_cuda_mismatch_message function extracts the CUDA
version from the error, compares to the system CUDA version via
torch.version.cuda, and returns a targeted install command using
the correct GitHub releases wheel URL.
- disable_broken_vllm uses the targeted message when a CUDA mismatch
is detected, falling back to the existing generic message otherwise.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Ubuntu <ubuntu@ip-172-31-16-253.us-east-2.compute.internal>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add `datasets` metadata support to model cards
Add an optional `datasets` parameter to all save/push functions so users
can specify which datasets were used for training. The metadata is set
via `ModelCard.data.datasets` for standard paths and via
`metadata_update` for GGUF and generic save paths.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix datasets metadata for existing repos, add token, improve errors
- Add metadata_update fallback in create_huggingface_repo and
upload_to_huggingface so datasets metadata is set even when the
repo already exists (previously only worked on first creation).
- Pass token=token to all metadata_update calls so they work
without a global HF login.
- Replace silent except:pass with logger.warning_once for
metadata failures so users know if something went wrong.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix generic datasets metadata repo resolution for PR #4076
* Fix create_huggingface_repo username resolution for PR #4076
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Hanchen <danielhanchen@users.noreply.github.com>
