* Fix TrainingArguments silently disabling unsloth gradient checkpointing
* Cover loaded adapters and preserve explicit None in GC restore
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: expose full compressed-tensors scheme set in an export formats dropdown
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: multi-select export formats, portable torchao FP8/INT8, GGUF LoRA, source parity
Export page overhaul on top of the formats dropdown:
- Unify merged precision into one sorted multi-select list (16-bit first, then
8-bit, then 4-bit). Drop "vLLM" from labels, add INT8 (W8A8), INT8 (W8A16),
INT4 (W4A16), MXFP4, MXFP8. Quick formats render as toggle pills; the rest live
in a multi-select "More formats" dropdown, so several formats export in one run.
- Add a portable torchao FP8/INT8 save path (Float8WeightOnlyConfig /
Int8WeightOnlyConfig) that needs no NVIDIA GPU to produce and loads in vLLM.
FP8 serializes to safetensors, INT8 to .bin. Wired into save_pretrained_merged
and push_to_hub_merged via a TORCHAO_EXPORT_SCHEMES registry and
_unsloth_save_torchao, parallel to the compressed-tensors path.
- Hide NVIDIA-only compressed-tensors formats when no NVIDIA GPU is present; keep
16-bit and portable FP8/INT8. The backend also rejects a compressed request on
non-NVIDIA hardware so it stays authoritative.
- Relax merged export to non-PEFT models so Local Model and Hugging Face sources
get the same 16-bit / compressed / portable options.
- GGUF: send the whole quant list in one call (merge once, quantize many).
- LoRA: add a GGUF adapter option (convert_lora_to_gguf.py) with an outtype
select (f16/bf16/f32/q8_0/auto), alongside the safetensors adapter.
- Thread the new fields through models, routes, orchestrator, and worker; extend
the export tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate export by accelerator with a torch-aware reason; fix export save dir naming
Export runs through Unsloth, which requires a compute accelerator (NVIDIA/AMD/Intel
GPU or Apple MLX) and has no CPU code path, so a bare-CPU host cannot export even
with PyTorch installed. Add export_capability() in utils/hardware that reports
export_supported plus a precise reason so the UI stops showing a generic "no GPU":
- pytorch_not_installed: a --no-torch install (even a physical GPU is unusable)
- no_accelerator: PyTorch present but no supported accelerator (bare CPU)
- mlx_unavailable: Apple Silicon where the MLX stack is missing or too old
Expose the fields on /api/system/hardware and /api/system, and guard the mutating
export routes (load-checkpoint, export/merged|base|gguf|lora) with HTTP 400 and the
reason, leaving read-only endpoints usable so the Export page still renders.
Make core/export/export.py import without PyTorch and without a usable accelerator
(the Unsloth import is caught) so the export worker degrades to a clear message
instead of crashing at import.
Frontend: keep /export reachable on chat-only hosts and gray out the method and
format options with the backend reason (Alert plus disabled MethodPicker) instead
of silently redirecting to /chat, so users see why export is unavailable.
Also fix the export save directory producing "model/null" for Local Model and
Hugging Face sources that have no run/checkpoint, naming the folder from the model id.
* CI: validate Studio export capability gating on Linux, Windows and macOS
Add a small pytest matrix that runs studio/backend/tests/test_export_capability.py
on ubuntu-latest, windows-latest and macos-latest. It confirms, on each real OS,
that hardware.export_capability() reports the right decision and reason
(pytorch_not_installed, no_accelerator, or mlx_unavailable) and that the export
backend imports without PyTorch and degrades to a clear message instead of crashing.
Hosted runners have no GPU/MLX, so this covers the "export unavailable, here is why"
path a Mac/Windows user without an accelerator sees; a real accelerator export is
validated separately. The job installs only a CPU PyTorch plus the backend import
deps (no unsloth, triton, or llama.cpp), so it runs in seconds with no GPU.
* Studio export: address Codex review (source-aware gating, GGUF LoRA token/MLX/guard)
Frontend (export-page):
- Gate LoRA and quantized-model restrictions on the active source. isAdapter /
isQuantized come from the selected checkpoint; in Local Model / Hugging Face
("model") source mode they were stale, so LoRA stayed wrongly enabled for a
direct base model (backend then rejects "No adapter to export") and a stale
"quantized" flag disabled every method for an unrelated, exportable model. Add
effectiveIsAdapter / effectiveIsQuantized (false outside checkpoint mode) and use
them in the method-reset effect and the MethodPicker disabled state.
- Hide the GGUF LoRA option on a macOS/MLX host (the backend rejects GGUF LoRA on
MLX), so users no longer pick it, wait through the load, and always fail. Disable
the "GGUF adapter" button on a Mac host and never send loraGguf there.
Backend (core/export/export.py):
- Pass the HF token into the GGUF LoRA conversion (save_pretrained_gguf), so a
gated/private base model's config fetch in convert_lora_to_gguf.py is
authenticated; without it the load can succeed but the conversion fails.
- Guard the save_pretrained_gguf capability check with getattr so an older Unsloth
model that lacks the method returns the clean "not supported" message instead of
an AttributeError that surfaces as a generic 500.
* Studio export: address 2nd Codex review (CI index, empty merged, test import)
- studio-export-capability-ci.yml: add --extra-index-url https://pypi.org/simple to
the torch install so torch's transitive deps still resolve; --index-url alone
replaces PyPI with only the CPU wheel index, which does not serve all of them.
- export-page handleStart: reject an empty merged selection (mirrors canExport), so
clicking the panel's Start button with every precision pill deselected no longer
submits mergedSelections: [] and launches an unintended default 16-bit export.
- test_export_imatrix_compressed: the torchao-registry test now reads unsloth/save.py
as text (like the other ast/string checks) instead of `import unsloth.save`, which
raised ModuleNotFoundError in the CPU studio-backend suite that has no unsloth
installed.
* Studio export: make comments succinct across the export changes
* Studio export: use load token for local GGUF LoRA export of gated bases
* Studio export: harden portable torchao path and gate multi-format Hub push
torchao (_unsloth_save_torchao):
- merge to an isolated temp staging dir so a co-selected 16-bit output at save_directory is not deleted
- narrow VLM detection to vision_config / ForVisionText2Text so T5/BART/Whisper are not misrouted
- forward trust_remote_code (from auto_map) to the reload so custom-code models export
Export UI:
- hide portable torchao formats on macOS/MLX (backend rejects quantized export there)
- restrict a Hub merged export to a single format (each writes to the repo root)
* Studio export: torchao tokenizer remote-code + XPU offload, scale GGUF timeout
torchao (_unsloth_save_torchao):
- honor auto_map in the staged tokenizer/processor configs (not just model.config) when
deriving trust_remote_code, so custom-code tokenizers reload after the merge
- offload single-device XPU models to CPU (and empty the XPU cache) before the reload, matching
the CUDA path, so an Intel GPU that fits the model once does not OOM on the second copy
Export orchestrator:
- scale the GGUF wait timeout by the number of requested quants so a multi-quant list export of a
large model does not time out at a flat 3600s
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio export: show portable torchao formats only on non-NVIDIA (CPU) hosts
Portable torchao FP8/INT8 is the fallback for hosts without the NVIDIA compressed-tensors path.
On an NVIDIA GPU the compressed-tensors FP8/FP4/INT formats are the intended path (llm-compressor
auto-installs), so hide the portable duplicates there; keep them on CPU / non-NVIDIA hosts and
continue hiding them on macOS/MLX.
* Studio export: report all output folders and the exported formats
- Multi-format merged export now collects every sibling output directory (one per selected
precision) instead of only the last; the success banner lists them all.
- Show the selected precision formats in the run summary (a Formats row, like GGUF Quantizations),
so the panel says what is being exported rather than just 'Merged Model'.
- Persist the selected formats in the run summary and seed them on mount, so navigating away and
back (or toggling the export method) restores the selection instead of resetting to 16-bit.
* Studio export: list all output formats, add GGUF LoRA target, default Q8_0, auto-select newest checkpoint
- Progress/summary panel now shows a Formats row with the selected merged
formats, and the success banner lists every output folder a multi-format
merged run creates (one line per format) instead of only the last one.
- Merged format selection is seeded from the active run, so navigating away
and back (or switching method cards) no longer resets it to 16-bit.
- GGUF / Llama.cpp now offers an Export target toggle (Full model or LoRA
adapter) for adapter checkpoints, reusing the LoRA GGUF export path.
- Removed the Auto GGUF LoRA output type and defaulted to Q8_0 in the UI,
the request model, and the backend defaults; the outtype list is now
Q8_0/F16/BF16/F32. Core save.py still accepts auto for external callers.
- When a finetune has no checkpoint selected, auto-select the newest one.
* Studio torchao export: robust reload class + optional VLM import
Two fixes to the portable torchao FP8/INT8 export reload, from review of the
narrowed VLM detection:
- Encoder-decoder seq2seq checkpoints (T5/BART/Whisper) are not causal LMs.
With the narrowed is_vlm test they now correctly skip the image-text class,
but fell through to AutoModelForCausalLM and failed to reload after the merge.
Reload them with their own architecture class from the config instead.
- AutoModelForImageTextToText was imported unconditionally at the top of the
torchao path, so on Transformers builds without that class the import aborted
every torchao export (even text-only). Import it lazily only for a VLM, with
the AutoModelForVision2Seq fallback used elsewhere in Unsloth.
* Studio: enable FP8/FP4 compressed export for newer-transformers models
The shipped llm-compressor 0.10.x pins transformers<=4.57.6, so FP8/FP4 export failed
for models needing a transformers 5.x sidecar (Qwen3.5, Gemma-4, Qwen3-Next): the
quantization subprocess crashed importing the removed TORCH_INIT_FUNCTIONS.
Run the quantization against a dedicated llm-compressor-main "shadow": a --target
package dir (transformers 5.10.2 + llm-compressor main + compressed-tensors) layered
over the existing torch. It installs --no-deps so torch is never touched (works on any
Studio torch build), is provisioned lazily and fingerprint-cached, and can be turned
off with UNSLOTH_DISABLE_LLMCOMPRESSOR_MAIN.
- transformers_version.py: provision + validate .venv_llmcompressor.
- export.py: route all compressed exports through the shadow when available; else keep
the workspace 0.10.x path and fail fast past its transformers ceiling.
- save.py: launch _compressed_quantize.py with a clean PYTHONPATH = shadow.
- _compressed_quantize.py: skip linear_attn / vision tower / MTP modules (matches the
RedHatAI and NVIDIA reference quants, and is required by the grouped schemes).
Verified all four schemes (fp8, w8a8, w4a16, mxfp4) on Qwen3.5-9B and Llama-3.2-1B, and
fp8 on Gemma-4, end to end through Studio.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix GGUF LoRA export tests
* Fix export CI expectations
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <112766706+wasimysaid@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* fast_generate: clear error for vLLM-style inputs when fast_inference=False
When fast_inference=False, fast_generate falls back to HuggingFace
generate, and the wrapper already rejects vLLM-only usage (a
sampling_params or lora_request kwarg, or a string prompt). A vLLM prompt
dict ({'prompt':..., 'multi_modal_data':...}) or a SamplingParams passed
positionally slipped through and hit transformers.generate, raising a
cryptic 'SamplingParams object has no attribute update'. Detect both and
raise the same clear 'only supported with fast_inference=True' error.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate: also reject positional list of SamplingParams and list of vLLM prompt dicts
Address review feedback: the slow-mode guard missed SamplingParams passed inside a
positional list and a list of {"prompt": ...} dicts, both valid vLLM batched shapes
that leaked into transformers.generate. Fold the checks into small predicates and
extend the GPU-free test (now 7 reject + 3 pass).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test_fast_generate_slow_guard: expose assertions via a test_ function so pytest collects them
The assertions lived in run(), only called from __main__, so pytest reported no tests
collected and CI skipped the coverage. Rename to test_fast_generate_slow_guard; the
standalone script entrypoint still works.
* fast_generate: reject vLLM tokenized/embeds prompt dicts in the slow-mode guard
vLLM also accepts prompt dicts keyed by prompt_token_ids or prompt_embeds, not just
prompt/multi_modal_data. Those slipped past the slow-mode guard and fell through to
HuggingFace generate with a cryptic error. Recognize all vLLM prompt-dict keys and
add a TokensPrompt test case (now 8 reject + 3 pass).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate slow-mode guard: catch vLLM prompts= keyword form
vLLM's generate names its first argument `prompts`, so a slow-mode call
like fast_generate(prompts="hi") or prompts=[{"prompt": ...}] bypassed the
guard and leaked into HuggingFace generate as an unexpected kwarg. Check
kwargs["prompts"] with the same _is_vllm_prompt predicate and add two test
cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate slow-mode guard: reject vLLM tokenized prompt kwargs
vLLM's legacy call shape passes tokens as prompt_token_ids= (and prompt_embeds=),
which are not HuggingFace generate arguments. In slow mode these bypassed the
guard and leaked into HF generate as unexpected kwargs. Reject their presence
with the same tokenize-first message and add a test case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate slow-mode guard: treat prompts= as vLLM-only
prompts is a vLLM keyword, not a HuggingFace generate argument, so any value
passed as prompts= (including a bare token-id list, which _is_vllm_prompt
deliberately ignores for positional HF token ids) is a vLLM-style call. Reject
prompts= / prompt_token_ids= / prompt_embeds= on presence, and keep the
conservative _is_vllm_prompt check only for the positional arg.
* fast_generate slow-mode guard: reject vLLM prompt kwargs on presence
prompts / prompt_token_ids / prompt_embeds are vLLM-only keyword names that
HuggingFace generate does not accept, so a defaulted call like prompts=None
should raise the actionable slow-mode error instead of leaking a None kwarg
into HF generate. Check membership in kwargs rather than a non-None value.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* feat: add mlx public trainer api
* test: cover mlx public trainer api
* fix: preserve mlx epoch trainer configs
* fix: pass mlx warmup ratio through config
* fix: align mlx trainer dataset order
* fix: keep mlx chat templates import-light
* fix: infer mlx trainer context length
* fix: mirror cuda mlx context defaults
* fix: align mlx notebook trainer defaults
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: keep mlx public helpers import-light
* refactor: reuse mlx optimizer normalization
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: address mlx review feedback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: tighten mlx training argument parity
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: align mlx trainer eos default
* Fix MLX trainer to accept DataCollatorForSeq2Seq and handle TokenizerWrapper in get_chat_template
* Trim redundant docstrings on internal MLX helpers
* MLX review fixes: Studio optimizer import-safe on non-MLX hosts, preserve explicit max_length, skip MLX tests before import
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* MLX review round 2: defer max_length to model context, optimizer alias fallback for older zoo, skip non-MLX test on missing GPU deps
* MLX review round 3: keep chat_templates importable without torch on MLX
* fix: preserve MLX trainer notebook shims
* fix: ignore CUDA tokenizer moves on MLX
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: harden MLX trainer shims
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: unwrap MLX scheduler enum args
* fix: coerce integral MLX epoch counts
* fix: spoof CUDA compatibility APIs on MLX
* fix: harden MLX notebook compatibility shims
* MLX: add torch.cuda.mem_get_info to the compatibility shim
Notebook memory cells call torch.cuda.mem_get_info()[0] directly (not gated by
is_available), so on MLX it raises without a shim. Return (free, total) bytes
from the MLX device stats, consistent with the other torch.cuda compat helpers,
and add a matching assertion to the compat-API test.
* MLX: use active memory for mem_get_info; fix BatchEncoding.to keyword device
Address review on the MLX compatibility shim:
- torch.cuda.mem_get_info() now derives free bytes from current active MLX
memory instead of the peak high-water mark, so a capacity check stays
accurate after a transient spike or a prior run.
- BatchEncoding.to(device=...) passed by keyword no longer forwards a positional
None alongside the keyword (which raised "multiple values for 'device'"), so
non-CUDA keyword moves like .to(device="cpu") delegate correctly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* MLX: accept preserve_dataset_order; stub RL trainers with a clear error
Two fixes so unmigrated notebooks behave predictably on MLX (torch present):
- preserve_dataset_order is a real MLXTrainingConfig field but was missing from
the extra-argument allowlist, so passing it (as a config or trainer kwarg)
could be rejected as unknown on a zoo without the field. Add it to
_MLX_IMPLEMENTED_EXTRA_ARGUMENTS so the documented no-shuffle path is reachable.
- GRPO/DPO/ORPO (and KTO/PPO/Reward) have no MLX trainer yet. Retarget the ones
the installed trl exposes to a stub that raises a clear 'not supported on MLX'
error instead of importing the real torch/CUDA trainer and crashing deep
inside it. Only existing trainers are retargeted (no invented attributes),
idempotent across re-imports.
* MLX: make RL-trainer stubbing import-safe; back current-memory APIs with active memory
Address review on the MLX shims:
- The RL-trainer stub loop probed trl with getattr(_trl, name), which triggers
trl's lazy trainer import and pulls torch -- that can crash import unsloth on a
torch-free MLX install just to check existence. Decide what to stub from
trl.__all__ + already-materialized attrs (vars) instead; never resolve the real
trainer. All trl trainer names are in __all__, so they are still stubbed (even
torch-free), and the probe no longer imports torch.
- torch.cuda.memory_reserved / memory_allocated (the current, non-max APIs) were
aliased to peak max_memory_reserved. Back them with current active MLX memory so
cleanup / capacity checks see live usage; max_* keep the peak high-water mark.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* MLX: keep TRL's SFTConfig epoch default under the trl.SFTConfig alias
Unmigrated notebooks import SFTConfig from trl, which the MLX build aliases to
the public training-args class. TRL/HF SFTConfig defaults to num_train_epochs=3
(max_steps=-1); the native MLX config defaults to max_steps=60. So an SFTConfig
built without an explicit length silently ran 60 MLX steps instead of TRL's 3
epochs under the alias. Alias trl.SFTConfig to a thin subclass that seeds the
TRL epoch default only when neither max_steps nor num_train_epochs is given;
explicit lengths pass through untouched, and the native public args class keeps
its MLX default. Epoch mode is supported by the MLX trainer.
* MLX CI: keep the GGUF reload smoke under the job timeout
The RELOAD-GGUF-via-llama-cli step timed out at 300s. BF16 GGUF decode is
CPU-bound on the macOS runner (~10s+/token), so generating 24 tokens landed
right on the 300s cliff and killed the process. This step is a save/reload
integrity smoke (it only needs a few chars of output), so the token count is
incidental: generate 8 tokens with explicit threads and a small headroom on the
subprocess timeout, all env-tunable (UNSLOTH_GGUF_RELOAD_N / _THREADS /
_TIMEOUT). Cuts the reload well under the 25 minute job budget.
* MLX: broaden trainer stubs, real peak-memory reset, fix shim tests
Address review on the MLX public API:
- The SFTConfig identity tests asserted trl.SFTConfig is UnslothTrainingArguments,
but the alias now points at the _MLXSFTConfig subclass that preserves TRL's
epoch default, so the MLX suite failed before testing the shim. Assert
issubclass instead.
- torch.cuda.reset_peak_memory_stats was a no-op, so max_memory_reserved kept
earlier model-load peaks across a scoped run. Wire it to mx.reset_peak_memory
with the same core/metal fallback used for the reads.
- The unsupported-trainer stubs were a fixed list, so trainers outside it (a
newer RLOOTrainer) still routed to the real torch trainer. Derive the set from
trl.__all__ (every non-SFT *Trainer) so all non-SFT surfaces fail with a clear
MLX message; names come from __all__ so trl is never resolved.
- The non-MLX export smoke skipped only on missing bitsandbytes/triton; other
absent GPU deps (numpy/torch/unsloth-zoo, or _gpu_init re-raising ImportError)
made it fail on CPU hosts. Skip on any ImportError.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: keep MLX notebook compatibility minimal
* MLX CI: force CPU + small context for the GGUF reload smoke
The RELOAD-GGUF-via-llama-cli step timed out even at 8 tokens (>420s), so it is a
fixed hang, not per-token cost: on the paravirtual macOS runner GPU llama.cpp's
Metal backend stalls, and the gemma3 GGUF advertises a 32768 context that llama-cli
would otherwise fully allocate. Run llama-cli CPU-only (-ngl 0) with a small context
(-c 256); keep generation short. All env-tunable (UNSLOTH_GGUF_RELOAD_NGL / _CTX /
_N / _THREADS / _TIMEOUT). Also print llama.cpp's partial stdout/stderr on timeout so
a future hang is diagnosable instead of an opaque TimeoutExpired.
* MLX CI: export the reload-smoke GGUF as q8_0, not bf16
The GGUF reload via llama-cli timed out on the runner even CPU-only with a tiny
context and 8 tokens. Root cause is the format, not the flags: the smoke exported
quantization_method='not_quantized', which maps to a bf16 GGUF, and llama.cpp's
bf16 CPU decode is unusably slow on the paravirtual macOS runner. Export q8_0
(fast_quantized, the exporter default and what users deploy) instead -- llama.cpp
has optimized q8_0 CPU kernels, so the fresh-process reload loads and generates in
seconds. The reload stays CPU-only (-ngl 0) with a small context.
* test: clear TRL shim before availability check
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Fix gpt-oss offload_embedding and generate() logits_to_keep on fused models
offload_embedding=True moved embed_tokens to CPU but left the input/output device-shuffling forward hooks commented out ('[TODO] Doesn't seem to work!'), so an eager forward/generate with CUDA input_ids hit the CPU embedding and raised a device-mismatch RuntimeError. Re-implement them in a testable helper _install_offload_embedding_hooks that saves the origin device on the module (the pre-hook returns a new tensor, so a device stashed on the original input is lost) and runs the lookup on the embedding weight's CURRENT device. Reading the weight device at call time (not a hard-coded cpu) also handles a non-quantized (bf16) embedding that a later model.to(...) pulls back onto the GPU, which the hard-coded version broke in the opposite direction.
unsloth_base_fast_generate injected logits_to_keep/num_logits_to_keep whenever an inner submodule forward accepted it, but transformers validates generate kwargs against the top-level prepare_inputs_for_generation (plus forward when it takes kwargs). On fused/PEFT-wrapped gpt-oss this raised 'model_kwargs are not used by the model: [logits_to_keep]'. Only inject when the top level would accept it, mirroring transformers _validate_model_kwargs. Behavior is unchanged for every model that works today.
Adds tests/test_offload_embedding_hooks.py and tests/test_generate_kwarg_gate.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload hooks: store origin device on the tensor, not the shared module
The pre-hook stashed the input device on embed_tokens itself, which races when
concurrent forwards share the module (serving). Ride it on the moved tensor and
read it from the post-hook args instead: stateless and thread-safe.
* Also strip mm_token_type_ids that generate() rejects (Qwen3-VL vision GRPO)
The vision processor (Transformers 5.x path) emits mm_token_type_ids, which
Qwen3-VL's generate() then rejects in _validate_model_kwargs on transformers
4.x, so vision GRPO fails at the first rollout:
ValueError: The following `model_kwargs` are not used by the model:
['mm_token_type_ids']
Unlike logits_to_keep this is an incoming kwarg rather than one we inject, so
drop it in unsloth_base_fast_generate when the top level generate does not
accept it, reusing the same _unsloth_generate_accepts_kwarg gate. Extends the
GPU-free gate test with the accept/reject mm_token_type_ids cases (7/7 pass).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim mm_token_type_ids comment
* Trim comments in gpt-oss offload/logits fix (comment-only)
* gpt-oss offload: return embedding output to the decoder device, not the input's
When offload_embedding moves the embedding to CPU, model.device can become CPU and
inputs then arrive on CPU, so returning the output to the input device left it on CPU
and the CUDA decoder hit a device mismatch. Capture the decoder device before offload
and always return there. This also drops the per-request tensor state (stateless, so
concurrent forwards stay correct).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: refuse offload_embedding for tied word embeddings
Tied models share embed_tokens.weight with lm_head, so offloading the weight
to CPU strands the output projection there (device mismatch at generate) and
saves no VRAM since lm_head still needs it on GPU. Detect the shared weight via
get_output_embeddings and raise NotImplementedError instead of loading into a
crash. Untied models (gpt-oss, Llama-3.1-8B) offload unchanged.
Adds tests/test_offload_tied_guard.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: skip embedding offload on fast_inference (vLLM)
vLLM manages its own weights, so offload_embedding cannot apply on the
fast_inference path (previously it was silently ignored). Disable it with a
notice, mirroring the WSL and Windows skips.
* Trim offload embedding comments (comment-only)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: track decoder device live so it survives model.to()
The post-hook returned the embedding output to a device captured at load time.
If a model is loaded on CPU then moved with model.to(cuda), that device is
stale and the output lands on the wrong device. Read the decoder device live
from the (untied) output embeddings, keeping the captured device as a fallback.
Adds a stale-fallback regression test.
* Make generate-kwarg-gate cases pytest-collectable
Cases lived in run(), which pytest does not collect, so CI never exercised the
gate. Expose them as test_generate_kwarg_gate; still runnable via __main__.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: skip a meta (disk-offloaded) lm_head as the return device
A device_map that disk-offloads an untied lm_head leaves its weight on the meta
device until that module's own hook runs, so reading it as the decoder device
would move real hidden states to meta. Skip meta (and a missing weight) and fall
back to the captured device. Adds a regression test.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Pin llm-compressor auto-install to a vetted version range
install_llm_compressor() auto-installs llm-compressor on first use of an FP8/FP4
compressed export when it is not already present. The install command used the bare
package name, so pip resolved to whatever the configured index served; a compromised,
dependency-confused, or inflated-version ("999.0.0") release could then run under the
Unsloth process at install and import time.
Bound the automatic install to a vetted range
(_LLM_COMPRESSOR_SPEC = "llmcompressor>=0.8.0,<0.13"), which the oneshot /
QuantizationModifier API this uses supports, so pip can no longer jump to an arbitrary
future or inflated version. An already-installed newer llm-compressor is still used
as-is (the import short-circuits), so this only constrains the auto-install, never a
user's own install.
Add UNSLOTH_DISABLE_LLM_COMPRESSOR_AUTOINSTALL=1 to forbid the automatic install
entirely and require a manual, vetted install, for locked-down or air-gapped
environments. Update the manual-install hints to the pinned spec.
Add tests/saving/test_llm_compressor_install_pin.py: static (ast) guards that the spec
stays a bounded pin, that the install command never passes an unpinned llmcompressor
literal, and that the opt-out env gate is evaluated before any install runs.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Loosen llm-compressor auto-install ceiling to <1.0 so new models still export
The earlier <0.13 ceiling was too tight: brand-new architectures (for example
Qwen3_5ForConditionalGeneration / qwen3_5, gemma-4 MoE) can require a newer
llm-compressor, and _unsloth_save_compressed_tensors already fails with
"requires a newer llm-compressor" when a scheme is unavailable. Capping the
auto-install at 0.12 would block getting that newer release and break compressed
export for new models.
Widen to llmcompressor>=0.8.0,<1.0. pip still auto-installs the latest 0.x
(where new-architecture support lands), while the <1.0 ceiling continues to block
a jump to an inflated-version ("999.0.0") or 1.0+ dependency-confusion release.
An already-installed newer llm-compressor is still used as-is (the import
short-circuits), and UNSLOTH_DISABLE_LLM_COMPRESSOR_AUTOINSTALL still forbids the
automatic install entirely for locked-down environments.
* Lower llm-compressor floor to 0.6.0 so supported old torch still resolves
The >=0.8.0 floor conflicts with the torch this install pins in its constraints
file. Unsloth supports torch>=2.4, but llm-compressor 0.7.0+ require torch>=2.7
(0.10+ need >=2.9, 0.12+ need >=2.10). On a supported torch 2.4-2.6 box pip then
has no candidate in [0.8.0, 1.0) and FP8/FP4 export fails before quantization.
Lower the floor to 0.6.0 (its metadata only needs torch>=1.7), which never
conflicts with any supported torch. pip still prefers the newest compatible
release, so modern torch continues to get the latest 0.x (0.12.0). The <1.0
ceiling that blocks an inflated-version supply-chain jump is unchanged.
Add a regression test asserting the floor stays <= 0.6.0.
* Cap llm-compressor auto-install ceiling to a vetted minor (<0.13)
A bare <1.0 ceiling still admits any 0.x, so an inflated "0.999.0" served by a
compromised or misconfigured index would win pip's highest-version selection --
the same dependency-confusion this pin is meant to block. Cap the ceiling to the
current vetted minor (<0.13) so that jump is blocked; bump it deliberately, after
vetting, when a newer llm-compressor is needed (e.g. for a brand-new architecture
scheme). Current new models are unaffected: 0.12.0 is < 0.13 and supports them.
The 0.6.0 floor (torch>=1.7 compatible) is unchanged, so resolution still works
across Unsloth's whole supported torch range (2.4 -> 0.6.0 ... 2.12 -> 0.12.0).
Add a regression test asserting the ceiling admits the current vetted release but
blocks an inflated 0.x and the next major.
* Trim comments in the llm-compressor pin (comment-only, no code change)
* Cap llm-compressor auto-install to the exact vetted patch (<=0.12.0)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Condense the verbose comments and docstrings added by the recent
chat template, GPT-OSS detection, PEFT tensor-parallel, and Studio
inference proxy fixes. Comments and whitespace only; no code changes.
* fix: keep LoRA reloads working with PEFT 0.19
* test: exercise the PEFT tensor-parallel symbol extractor
* test: prove the full PEFT tensor-parallel seam
* fix: harden PEFT tensor-parallel shims
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: fall back when PEFT tensor-parallel source inspection fails
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Add FP8/FP4 compressed export to save_pretrained_merged
Adds compressed-tensors export (for vLLM) to save_pretrained_merged /
push_to_hub_merged via llm-compressor, alongside the existing lora /
merged_16bit / merged_4bit / gguf / torchao paths:
model.save_pretrained_merged("model", tokenizer, save_method="fp8")
Supported save_method values: fp8 (FP8_DYNAMIC), mxfp4, nvfp4 (W4A4) and
mxfp8. The LoRA is merged to 16bit at save_directory, then a quantized
checkpoint is written to save_directory + "-<fmt>". nvfp4 needs a small
calibration set (defaults to ultrachat, overridable via calibration_dataset).
Notes:
- llm-compressor is installed lazily on first use, pinning the current torch
and transformers via a constraints file so they are not upgraded (a plain
install pulls transformers>=5 and breaks Unsloth).
- Quantization runs in a separate process (unsloth/_compressed_quantize.py,
launched by file path) so Unsloth's transformers attention patches do not
interfere with the forward llm-compressor runs during calibration, mirroring
how GGUF export shells out to llama.cpp.
- mxfp8 needs a newer llm-compressor (transformers>=5); it is recognised and
raises a clear error until that stack is available.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: main-process guard, calibration subsampling, tokenizer + dtype handling
- Route the 16bit merge through unsloth_generic_save for both LoRA and full
finetuned models, so non-PEFT models are written in 16bit consistently
instead of saving the original (possibly quantized) weights directly.
- Honor is_main_process: only the main process quantizes and writes the
compressed output, so distributed ranks do not race on the same dirs.
- Subsample an in-memory calibration Dataset before save_to_disk so large
training sets are not fully copied to a temp dir.
- Tolerate a missing tokenizer in the converter (data-free exports); still
require one for calibration based schemes.
- Open config.json via a context manager in both files.
- Drop the redundant nvfp4 entry from the unsupported-name check (fp4 covers it).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add direct LoRA to GGUF export and harden FP8/FP4 compressed export
- Run llm-compressor install and scheme check before the 16bit merge so
unsupported schemes (e.g. mxfp8) fail fast without writing a checkpoint
- Only the main process installs, merges, quantizes and uploads; isolate
hub pushes to a temp dir and clean all temp dirs in a finally
- Forward standard save kwargs (state_dict, max_shard_size, ...) to the merge
- Fall back to the first dataset split for Hub calibration ids
- Export LoRA adapters to GGUF via convert_lora_to_gguf.py: modernize
save_pretrained_ggml/push_to_hub_ggml and add save_method="lora" to
save_pretrained_gguf/push_to_hub_gguf; resolve base from the adapter config
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix LoRA GGUF shell-injection test and compressed export trailing-slash path
- Update tests/saving/test_save_shell_injection.py for the new delegation: the
LoRA to GGUF conversion now lives in _unsloth_save_lora_gguf, so assert it
passes argv as a list with no shell=True and that the legacy ggml wrappers
delegate to it instead of calling subprocess.Popen directly
- Normalize the local save_directory before building the "<dir>-<fmt>" sibling
so a trailing slash no longer nests the compressed output inside the 16bit dir
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Polish FP8/FP4 and LoRA GGUF export after review
- Warn (not silently downgrade) when an explicit quantization_method is not a
valid LoRA GGUF outtype; default stays f16
- Correct the inference hardware note: MXFP8 is 8-bit (cc >= 8.9), only FP4
needs Blackwell for full activation quantization
- Document that a local fp8/fp4 save keeps the 16bit merge at save_directory
and writes the quantized checkpoint to save_directory + "-<fmt>"
* Use sequential calibration pipeline and validate Hub access early
- nvfp4 calibration no longer forces the memory-hungry "basic" pipeline. The
quantization runs in a clean subprocess, so llm-compressor's default
sequential pipeline (layer-by-layer onloading) works and lets large models
that do not fit at once still calibrate; fall back to "basic" only if tracing
fails
- For push_to_hub compressed exports, create/validate the repo up front so a bad
token or denied repo fails before the merge and quantization instead of after
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden compressed export: explicit sequential pipeline, base-tokenizer calibration, GPU memory
- nvfp4 calibration now passes pipeline="sequential" explicitly (layer-by-layer
onloading) instead of relying on the inferred default, with a "basic" fallback
- Calibration datasets with a messages column no longer require a chat template:
base / non-chat tokenizers fall back to concatenating message contents
- Free the in-memory model's CUDA memory before the quantize subprocess loads its
own copy from disk (best-effort, single-device non-quantized only; restored
afterward), so a single GPU need not hold two copies at once
- Create the calibration temp dir in the system temp location instead of next to
the save directory, avoiding stray dirs in the workspace
* Free the failed calibration model before the basic-pipeline retry
In the sequential -> basic NVFP4 fallback, release the partially-processed model
and clear the CUDA cache before loading a fresh copy, so the retry does not
transiently hold two model copies on the GPU.
* Harden calibration data handling and compressed-export edge cases
- Calibration messages without a chat template now handle multimodal (list)
content, None content, and null message rows instead of crashing on join
- Raise a clear error when the calibration dataset is empty after subsampling
- Reset llm-compressor's global session before freeing the model in the
sequential -> basic NVFP4 fallback, so the old model is actually released
- LoRA GGUF export accepts a single-element list quantization_method
- Attach datasets metadata to the pushed repo on compressed hub exports
- Warn (instead of silently) if the model cannot be restored to its device
- Raise a clear error if the LoRA base model id cannot be determined
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Handle DatasetDict calibration, MoE routers, and MTP models in compressed export
- Reduce an in-memory DatasetDict calibration set to a single split before row
subsampling, so save_to_disk does not copy every split to the temp dir
- For MoE models, keep the router/gate unquantized and pass
moe_calibrate_all_experts so every expert is calibrated
- Warn when a model carries MTP / speculative-decoding tensors that the
compressed export does not include
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Support many more compressed-tensors schemes and address review
- Expand save_method to cover the full set of compressed-tensors preset schemes:
FP8 (dynamic/static/block), INT8, W8A8, W8A16, W4A16(+asym), W4A8, W4AFP8,
MXFP4(+A16), NVFP4(+A16), plus the gated MXFP8; calibration is used only for
the static-activation schemes (FP8 static, NVFP4)
- Broaden the near-miss save_method error to cover int/w-prefixed names
- MoE: also keep the Qwen shared-expert gate unquantized
- Strip non-model-input columns from already-tokenized calibration data so the
collator does not choke on a leftover messages column
- Forward the Hub token to the LoRA converter and the quantize subprocess so
gated/private base models and calibration datasets work without a global login
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Collapse compressed-tensors export help line so ruff-format converges
The print line in print_quantization_methods needed two ruff-format passes to
reach a fixpoint (merge implicit string concat, then collapse the single-arg
print). pre-commit.ci applies one pass per run, so it kept reformatting. Land
the converged single-line form directly.
* Add CPU-only regression tests for the export API
Cover all export paths without a GPU, for slow CPU-only CI:
- pure-function checks of the compressed-tensors scheme registry and save_method
normalization (aliases, calibration flags, near-miss errors)
- AST checks that every merged saver dispatches compressed export, the GGUF savers
expose the lora branch, torchao routes PTQ/QAT, the public methods stay attached,
and the export subprocesses remain shell-safe (argv list, sys.executable, no shell)
- monkeypatched dispatch checks that fp8/nvfp4/merged_16bit, the LoRA-GGUF outtype
resolution, and torchao PTQ/QAT reach the right helper with the right arguments
* Run the CPU-only export tests in consolidated CI
tests/saving is --ignored by the Repo tests (CPU) job, so the new GPU-free export
tests are added by path to consolidated-tests-ci.yml (collection sanity + Bucket-A run),
alongside the existing CPU saving tests, so they actually execute on CPU CI.
* Add GPU GGUF export + llama-cli inference smoke test
tests/saving/test_gguf_export_and_inference.py: skipif no CUDA. Trains a tiny
phrase-imprinting LoRA, exports a full-model q8_0 GGUF (merge -> convert_hf_to_gguf
-> llama-quantize), asserts a valid GGUF (magic + size), and - when a llama-cli
binary is available - runs one bounded generation (byte cap + watchdog kill) and
asserts the trained phrase round-trips through HF -> GGUF -> quantize -> inference.
The llama-cli step skips gracefully since the export only builds llama-quantize.
* Fix variant mismatch in compressed (FP8/FP4) export
save_pretrained_merged(..., save_method=fp8/nvfp4, variant=...) forwarded
the variant into the intermediate 16bit merge, so Transformers wrote
variant-named shards (model.<variant>.safetensors). The converter
subprocess then reloaded that directory with the default weight filenames,
so the compressed export failed after doing the merge.
Pop the variant out of the intermediate merge (internal staging that the
subprocess reloads with default names) and forward it via --variant so it
is applied to the final compressed checkpoint instead. Add a CPU AST guard
for the contract.
* Harden export paths from review
- install_llm_compressor: fall back to uv pip when this interpreter has no
pip seeded (uv-created/relocatable venvs), instead of failing with
No module named pip.
- LoRA GGUF export: if convert_lora_to_gguf.py is missing (a prebuilt or
reused CWD llama.cpp install carries binaries but not the converter
script), force a dedicated source checkout that ships it.
- push_to_hub_gguf(save_method=lora): return on non-main ranks, matching the
local save_pretrained_gguf lora branch, so only rank 0 converts/uploads.
- compressed export VLM detection: require a vision_config or a
ForVisionText2Text architecture; a bare *ForConditionalGeneration also
matches text seq2seq models (T5/BART/Whisper) and is no longer treated as
a VLM on its own.
- GGUF GPU smoke test: drop SFTConfig(max_length=1024), which raises under
newer TRL padding-free training; length enforcement is not needed here.
* Add imatrix option to GGUF export, enabling IQ low-bit quants
save_pretrained_gguf / push_to_hub_gguf gain imatrix_file:
None -> no imatrix (unchanged)
'/path' -> pass to llama-quantize --imatrix (a *.gguf_file is renamed to *.gguf)
True -> download the upstream unsloth/<base>-GGUF imatrix (imatrix_unsloth.dat or
.gguf_file), raising a clear error if none exists
An importance matrix unlocks the IQ low-bit quants (iq2_xxs, iq4_xs, ...), which were hard
disabled before. They are gated: requesting one without an imatrix raises a clear error.
- _resolve_imatrix_file resolves path/True (PEFT base first, normalized via get_model_name,
derives unsloth/<base>-GGUF, copies out of the HF cache before renaming *.gguf_file).
- IMATRIX_QUANTS registry replaces the old commented-out IQ entries; save_to_gguf accepts a
resolved imatrix and threads it into the quantize calls.
- The --imatrix flag is emitted by unsloth_zoo's quantize_gguf (companion change). save.py
fails fast with an upgrade hint if the installed unsloth_zoo lacks the imatrix kwarg.
Tests: tests/saving/test_imatrix_export.py (CPU: resolution, repo derivation, IQ gate,
--imatrix wiring) wired into CI; tests/saving/test_gguf_export_and_inference.py extended with
GPU iq2_xxs/iq4_xs export + inference. Verified end to end on Llama-3.2-1B: imatrix
auto-downloaded, iq2_xxs/iq4_xs exported and run via llama.cpp.
Note: requires the companion unsloth_zoo quantize_gguf imatrix change.
* Address imatrix/compressed review feedback: unsloth org GGUF repo, fail-fast, calibration split
- imatrix auto-resolve (imatrix_file=True): derive the upstream repo as unsloth/<base>-GGUF
instead of <org>/<base>-GGUF, so official bases (e.g. meta-llama/Llama-3.1-8B-Instruct) find
the matching Unsloth GGUF imatrix repo rather than failing on a nonexistent meta-llama/...-GGUF.
- Resolve/validate the imatrix before the 16-bit merge in save_pretrained_gguf, so a bad path or
an unavailable upstream imatrix fails fast instead of after a long, multi-GB merge.
- Compressed calibration: when a Hub dataset has no "train" split, resolve the first split name
and slice it, instead of materializing the whole dataset just to take num_samples rows. Keeps
the original materialize-then-subselect path as a last resort.
Tests: add unsloth/<base>-GGUF mapping for an official base id, and create the imatrix file in the
quantize_gguf flag test (quantize_gguf now validates the imatrix exists).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Improve code quality & performance: fix typos, compile regex & cache fields
- Fix typos across core files (repeatted → repeated, splitted → split, etc.)
- Compile regex patterns once as class attributes in TextPreprocessor
- Cache text fields/columns in RawTextDataLoader
- Improve comments (re-use → reuse)
* Use immutable raw text field constants
---------
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
FalconH1RMSNormGated is imported from transformers but never referenced in
unsloth/models/falcon_h1.py. The unused hoist trips the import-hoist lint gate
on the merge commit of every open PR (the gate lints PR-head merged into main),
so clearing it here unblocks those PRs.
* Fix fast_inference crash on ABI-broken vLLM: force-load compiled extensions in the broken-vLLM probe
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Broaden broken-vLLM probe: catch non-libcudart .so failures and _moe_C_stable_libtorch
* Revert stray reformat of the PDL fix log line
* Trim verbose comments in the broken-vLLM probe
* Drop non-existent vllm._moe_C_stable_libtorch from the broken-vLLM probe
* Shorten comments in broken vLLM extension detection
Condense the docstrings and inline comments for the lazy-loaded vLLM probe
and the new regression test while keeping the rationale. Comments only, no
code changes (verified with an AST signature check and the existing tests).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
On torch >= 2.11 torchao tries to dlopen each prebuilt _C*.so and logs a
per-file "Failed to load .../_C*.so" WARNING via the torchao logger when one
cannot load. This happens on an ABI tag mismatch in the prebuilt wheel (for
example a cp310 .so under a cp312 runtime, as on Colab) or when the kernel
targets an arch the GPU does not have (mxfp8 needs FP8 hardware, _C_cutlass_90a
is Hopper/SM90 only). torchao falls back to its non-cpp paths and Unsloth's
bnb-4bit / Triton kernels do not use these, so the warning is cosmetic.
Add a HideLoggingMessage filter on the same torchao logger that already filters
the torch < 2.11 "Skipping import of cpp extensions" message, so only these
records are dropped rather than raising the whole logger to ERROR.
* Patch FalconH1RMSNorm to fix float64 compilation on Intel Arc DG2
Fixesunslothai/unsloth#6555
Root cause: FalconH1RMSNorm.forward() does hidden_states.pow(2).mean().rsqrt()
with self.variance_epsilon being a Python float64. When torch.compile fuses
this pattern into the auto-generated Triton kernel
'triton_per_fused__to_copy_mean_mul_pow_rsqrt_*', the float64 epsilon causes
type promotion to double. Intel Arc DG2 GPUs do not support double precision
(Double type is not supported on this platform).
The existing patch_rms_layernorm() only patches LlamaRMSNorm, not the
separate FalconH1RMSNorm class in transformers.models.falcon_h1.
Fix: add Unsloth_FalconH1RMSNorm that delegates to fast_rms_layernorm
(@torch.compiler.disable, handles epsilon as tl.float32), and call the
patch in FastFalconH1Model.pre_patch() before model creation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Condense FalconH1RMSNorm patch comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Fix DDP crash from CPU-resident rotary inv_freq buffer
DistributedDataParallel broadcasts all named buffers regardless of
persistence or device, but Unsloth's RoPE inv_freq buffer is kept on
CPU on purpose (per-GPU cos/sin caches are precomputed instead). That
mismatch crashed multi-GPU DDP training with "No backend type
associated with device type cpu" during _sync_module_states.
Mark inv_freq/short_inv_freq/long_inv_freq buffers as DDP-ignored
instead of moving them to GPU, so they're skipped during the buffer
broadcast without disabling broadcast_buffers for the rest of the
model.
Fixes#6656
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: harden DDP-ignore against private API drift, re-apply after PEFT wrap
- Wrap the private DistributedDataParallel._set_params_and_buffers_to_ignore_for_model
call in try/except, falling back to setting _ddp_params_and_buffers_to_ignore
directly so a future PyTorch API change can't block model loading.
- Move _exclude_rope_inv_freq_from_ddp to loader_utils.py (shared by loader.py,
llama.py, vision.py without circular imports) and call it again after
get_peft_model wraps the model in a PeftModel, since the rotary buffers'
fully qualified names change once nested under "base_model.model...".
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Fix offline checkpoint load/export failing with "tokenizer is weirdly not loaded"
Loading a fine-tuned checkpoint with no internet (e.g. a Studio export) crashed
with "Unsloth: The tokenizer is weirdly not loaded? Please check if there is one."
For a LoRA adapter the loader reassigns model_name to the base model repo id and
only keeps the local checkpoint dir as tokenizer_name when it contains
tokenizer_config.json, tokenizer.json AND special_tokens_map.json. Modern
tokenizers (e.g. Gemma) store special tokens inside tokenizer_config.json and
omit special_tokens_map.json, so tokenizer_name fell back to the base repo id.
The tokenizer/processor loads in vision.py then hit the Hub with no
local_files_only, so with no network they failed (AutoProcessor) or hung for
minutes (AutoTokenizer) even though every file was already cached.
loader.py: keep the local checkpoint dir as tokenizer_name when it has a
tokenizer config plus the actual tokenizer files (tokenizer.json / tokenizer.model
/ vocab files); special_tokens_map.json is no longer required.
vision.py: compute an effective local_files_only (explicit kwarg plus the
HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE env vars, mirroring loader.py and
diffusion.py) and thread it through every AutoConfig, AutoProcessor,
AutoTokenizer and the manual VLM processor fallback, including the
hf_hub_download in that fallback (which now prefers a local file). When a load
fails and no offline env var is set, retry against the local cache. The retry
forces HF offline mode because local_files_only alone does not stop
AutoProcessor / AutoTokenizer from issuing a /api/models request during class
resolution. The final error now explains the offline/cache cause instead of the
misleading "weirdly not loaded" message.
studio export: probe Hub reachability once per checkpoint load and pass
local_files_only when offline so exports use the local checkpoint dir / cache
instead of hanging or crashing with no internet.
Online behavior is unchanged: the new flags default to off and the retry only
runs after a network related failure.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: safer offline forcing, cached fallback config, proxy-aware probe
Follow-up to the offline checkpoint load fix, addressing review feedback:
- vision.py: only flip the process-wide HF offline flag when offline is actually
requested (local_files_only / env) or after a real network failure, never
pre-emptively while we might be online. The flip is now guarded by a lock +
depth counter so nested or concurrent windows restore the flag correctly
(no stale value).
- vision.py: guard the get_auto_processor fallback so a network error there
returns None and the local-cache retry still runs instead of escaping.
- vision.py: in the manual VLM processor fallback, read tokenizer_config.json
via hf_hub_download(..., local_files_only=...) so a cached repo-id config is
still resolved offline and the model-specific image/video tokens are restored.
- studio export: make the reachability probe proxy aware (probe the configured
HTTP(S) proxy egress, honour NO_PROXY, use the endpoint port) so a proxy-only
setup is not wrongly marked offline; allow UNSLOTH_OFFLINE_PROBE=0 to disable.
- studio export: run the audio/vision type-detection probes inside the
forced-offline window when offline, so their config/tokenizer reads hit the
local cache instead of waiting out connection timeouts.
Online behavior remains unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: gate offline retry, safer tokenizer_name pop, skip audio net probe offline
- vision.py: only force the process-wide HF offline flag on the tokenizer
retry when offline was requested or the captured primary error is actually
network related, so a permanent tokenizer error no longer toggles global
offline mode for other concurrent loads.
- loader.py: always pop tokenizer_name out of kwargs and let a caller-supplied
value win, avoiding a "multiple values for keyword argument 'tokenizer_name'"
TypeError when it is also passed explicitly downstream.
- model_config.py / export.py: add local_files_only to detect_audio_type so the
raw requests.get tokenizer_config fetch is skipped offline (it ignores the HF
offline flag), and pass it from the export probe.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: classify LocalEntryNotFoundError as offline-related
huggingface_hub's LocalEntryNotFoundError subclasses FileNotFoundError, so the
"not isinstance(cur, FileNotFoundError)" guard in _is_offline_related_error was
swallowing it and it could never be recognised as offline, despite being listed
in the network error types. It means "not in cache and the Hub is unreachable",
which is genuinely offline. Capture the class into an isinstance-checkable tuple
(empty, hence a no-op, if the import is unavailable) and exclude it from the
FileNotFoundError guard, so a real offline failure now triggers the local-cache
retry while a plain missing-file error still propagates.
* Address review: require merges.txt for BPE, status-gate HTTP errors, isolate local-only audio cache
- loader.py: a local dir with vocab.json but no merges.txt (and no tokenizer.json)
is not a loadable BPE tokenizer, so do not treat it as self-sufficient; require
merges.txt alongside vocab.json in both gate blocks, otherwise fall back to the
base model tokenizer as before.
- vision.py: _is_offline_related_error no longer buckets every HfHubHTTPError /
requests HTTPError as offline. HTTP errors are judged by status code: only a
transient 5xx triggers the forced local-cache retry, while 401/403 (auth/gated)
and 404 (missing) propagate as the real error instead of being masked. Hard
signals (connection/timeout/OfflineModeIsEnabled/LocalEntryNotFoundError) still
classify as offline.
- model_config.py: include local_files_only in the audio-detection cache key so a
local-only (offline) negative result cannot be reused by a later online probe,
which would otherwise route an audio model through the text loader until restart.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address re-review: fix studio test stubs, force offline env in probe window, drop redundant retry
- studio/backend/tests/test_vision_cache.py: the three _detect_audio_from_tokenizer
stubs were called with the new local_files_only kwarg and raised TypeError, failing
Backend CI. Add local_files_only to the stub signatures and add a test that a
local-only negative does not poison a later online audio probe.
- export.py: the type-detection probe window now also sets HF_HUB_OFFLINE /
TRANSFORMERS_OFFLINE env vars (saved/restored), not just the in-process flag.
transformers_version._load_config_json / _check_tokenizer_config_needs_v5 gate
their urllib fetches on the env vars, and is_vision_model may spawn a subprocess
that inherits os.environ but not the in-process flag; without the env vars a
probe-detected offline export could still block on a network timeout.
- vision.py: only retry the processor load when the first attempt was online and
failed with a network error. When local_files_only was already requested the first
attempt was forced offline, so the previous retry just repeated identical failing
work before the last-resort path.
- model_config.py: correct the _audio_detection_cache type annotation to the 3-tuple
key (name, token_fingerprint, local_files_only).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address review: thread-safe probe-offline env window, clear error for local dir without config
- export.py: guard the HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE mutation in
_force_offline_probe_window with a lock + depth counter (mirrors _force_hf_offline),
so concurrent / nested export probes only flip on first entry and restore on last
exit. This prevents overlapping export requests from permanently poisoning those
env vars or restoring a stale value.
- vision.py: in the VLM processor fallback, when tokenizer_name is a local directory,
read its tokenizer_config.json directly and raise a clear FileNotFoundError if it is
absent, instead of handing the local path to hf_hub_download (which would treat it as
a repo id and raise a confusing HFValidationError / RepositoryNotFoundError).
hf_hub_download is now only used for actual repo ids.
* Address review: classify raw socket.gaierror DNS failures as offline
Add the platform-specific getaddrinfo / DNS-resolution wording to the offline
detection list in _is_offline_related_error so a bare socket.gaierror (an OSError
subclass) is recovered from the local cache: "Name or service not known" and
"Temporary failure in name resolution" (Linux) and "nodename nor servname
provided" (macOS). Genuine non-network OSErrors (disk full, permission denied)
and plain FileNotFoundError still propagate.
* Address review: retry degraded VLM offline, force offline for text export + patch-tokenizer fallback
- vision.py: a degraded VLM processor (text-only, no image_processor) whose manual
fallback fails offline used to be kept, so image inputs broke even with cached
files. _construct_vlm_processor_fallback now returns its failure error;
_acquire_processor surfaces it, and the caller retries forced-offline when the
result is None OR a degraded VLM and the failure was network related, keeping the
original result if the retry is not strictly better (never regress). The retry is
still gated on an online first attempt + offline-related error so a permanent
error never flips the global offline flag.
- vision.py: wrap the patch_tokenizer except-branch AutoTokenizer.from_pretrained in
the same forced-offline-on-network-error pattern as the primary / last-resort
loads, so an offline export where patch_tokenizer raises does not hang or fail.
- export.py: force HF offline around the two FastLanguageModel loads (text and SNAC)
when the probe detected offline. Their text tokenizer path (load_correct_tokenizer
-> AutoTokenizer) does not forward local_files_only, so without this a text export
could still contact the Hub. Added a small _offline_window_if helper reused by the
probe and load windows.
* Consolidate offline loading into one entry-point decision
Decide offline once per entry point instead of at every HF call site. The
prior approach threaded local_files_only into ~15 scattered config / tokenizer
/ processor / weight loads, each wrapped in its own try-online, classify-error,
retry-forced-offline dance, which is what kept surfacing "another call site you
missed", "another error shape misclassified", and global-flag thread-safety in
review.
FastLanguageModel / FastModel / FastBaseModel.from_pretrained now share an
@_offline_aware_load decorator: when offline (explicit local_files_only kwarg or
HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE env) it sets local_files_only and runs the
whole load inside one _force_hf_offline() window so every nested HF call inherits
it; when online it runs normally and, only if the load fails with a genuinely
network-related error, retries once forced-offline. The online path is unchanged
(no probe added) and 401 / 403 / 404 / permanent errors still propagate.
Centralise the offline helpers in loader_utils.py as the single source of truth
(shared by loader.py, re-exported from vision.py, and reused by the Studio
exporter):
- _force_hf_offline now sets the HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE env vars
AND the in-process huggingface_hub / transformers flags, refcounted under one
lock so nested / concurrent windows restore correctly. Setting the env vars
covers env-gated urllib probes and spawned subprocesses too.
- _get_effective_local_files_only, _is_offline_related_error (unchanged
classifier, retains the 5xx-vs-4xx, LocalEntryNotFound and gaierror handling),
_offline_aware_load, and _resolve_checkpoint_tokenizer_name.
loader.py: wrap both entry points; drop the two duplicated env-var fallback
blocks and the two byte-identical local-tokenizer-gate blocks (now
_resolve_checkpoint_tokenizer_name).
vision.py: drop the per-site force_offline params and the three retry gates
(processor, patch_tokenizer fallback, last-resort). They now just surface the
underlying error so the single entry-point safety net retries forced-offline. A
network fallback error now takes precedence over a permanent primary error so the
offline retry still fires when the manual VLM fallback needs cached repo files.
studio/backend export.py: reuse the unified core _force_hf_offline (env + flags)
and drop the duplicate probe-window primitive; the snac / text branches no longer
need their own window. model_config.py: also gate the raw requests.get audio
fallback on the HF offline env vars so it is covered even without the kwarg.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address 10-reviewer P1 findings: vision cache split, PEFT offline, retry OOM
Split the Studio vision-detection cache by local_files_only, mirroring the audio
cache fix. is_vision_model / _is_vision_model_uncached / _raw_config_has_vision_config
/ load_model_config now thread local_files_only, the cache key includes it, and the
exporter passes it. Offline detection also skips the transformers-5 network
subprocess and stays on the local cache, so an offline negative can no longer be
keyed under the online entry and poison a later online probe. Adds a regression
test mirroring the audio poison test.
Forward local_files_only to both PeftModel.from_pretrained adapter-attach sites in
loader.py so a cached remote LoRA adapter resolves from the local cache under
explicit local-only / offline loads (defence-in-depth alongside the forced-offline
window).
_offline_aware_load: run the forced-offline retry OUTSIDE the except block and
collect + empty the device cache first. An except-scoped exception keeps its
__traceback__, which pins the failed attempt's frame locals (a partially loaded
model) until the block exits; loading the model again while that copy is still
alive could OOM a large VLM. Letting the except block close drops the traceback so
the partial load is freed before the retry reallocates.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address Codex review: env-offline cache key + rebuild HF sessions in offline window
Key the Studio audio and vision detection caches on the EFFECTIVE offline state
(local_files_only OR the HF offline env vars), not just the kwarg. detect_audio_type
and is_vision_model both skip the remote fetch / network subprocess when
HF_HUB_OFFLINE / TRANSFORMERS_OFFLINE is set even with the default
local_files_only=False, so the result reflects offline; storing it under the online
(False) key let an env-offline negative poison a later online lookup once the env var
was cleared. Both now compute effective_offline once and use it for the cache key and
the downstream call. Adds a regression test for the env-offline dimension.
_force_hf_offline now rebuilds huggingface_hub's cached sessions on enter and exit
(best-effort _reset_hf_sessions). On hub 0.x the offline adapter is baked into the
per-thread requests.Session at creation, so flipping the constant alone leaves an
already-cached online session able to hit the network inside the window (and an
offline one stuck offline after restore); resetting forces the next get_session() to
match the current flag. On hub 1.x offline is checked dynamically per request, so
reset_sessions does not exist and the helper is a safe no-op.
The third review point (release the failed load before retrying) was already fixed in
af0f58a: the forced-offline retry now runs outside the except block and frees the
device cache first, so the failed attempt's traceback-pinned partial model is
released before the retry reallocates.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Align Studio _env_offline parsing with the canonical offline helper
model_config._env_offline gates the raw requests.get tokenizer-config fallback in
detect_audio_type and the audio/vision detection cache keys, but it only accepted
unstripped "1"/"true"/"yes". unsloth's offline helpers (loader_utils._env_says_offline
and the from_pretrained env fallback) accept the canonical set {1,true,yes,on} after
strip + lowercase, so HF_HUB_OFFLINE=on or HF_HUB_OFFLINE=" 1 " was treated as offline
by the loaders but online here, leaving the raw network fetch reachable while
"offline". Use the same strip + lowercase {1,true,yes,on} set. Adds parsing tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix lint: drop dead offline-helper re-exports from vision.py
The import-hoist verifier (scripts/verify_import_hoist.py) flagged vision.py's
re-export block as HOISTED-IMPORT-UNUSED blockers: it imported eight offline
helpers from loader_utils but only used three internally
(_get_effective_local_files_only, _is_offline_related_error, _offline_aware_load).
The other five were imported purely to preserve `from unsloth.models.vision import
X`, but nothing imports four of them from vision, and loader.py already imports
_resolve_checkpoint_tokenizer_name straight from loader_utils.
Import only the three names vision.py actually uses, and point the Studio exporter
at the canonical source (from unsloth.models.loader_utils import _force_hf_offline)
instead of re-exporting it through vision. loader_utils stays the single source of
truth; no behaviour change.
* Address Opus review: chain probe errors, unify env-offline, status-less HTTP
Chain the original AutoConfig/PeftConfig probe exception into the combined
RuntimeError in both FastLanguageModel.from_pretrained and FastModel.from_pretrained
(`raise RuntimeError(combined_error) from (autoconfig_exc or peft_exc)`). The probes
caught every Exception and stringified it, so the re-raised RuntimeError had no
__cause__/__context__ and _is_offline_related_error could not classify it -- the
network-down-but-cached auto-retry never fired for these entry points. With the
cause chained, the decorator sees a ConnectionError/LocalEntryNotFoundError/5xx and
retries forced-offline from cache; a permanent cause (404 / bad config) is still not
offline-classified and propagates without a wasted retry.
Unify the third offline-env parser: studio/backend/utils/transformers_version._env_offline
now uses the canonical {1,true,yes,on} + strip + lowercase set (matching
loader_utils._env_says_offline and model_config._env_offline), so HF_HUB_OFFLINE=on
or " 1 " no longer leaks the direct urllib metadata fetches to the network.
_is_offline_related_error: a status-less HTTP error (no response / unparseable code)
now falls back to the network-wording check instead of being dropped, so a transient
HTTP failure with clear "couldn't connect" wording is treated as offline. HTTP errors
with a real status code still decide by code (4xx propagates, 5xx is offline).
* Condense offline-loading code comments, drop dead helper, dedupe import for PR #6554
* Add unit tests for offline-loading helpers for PR #6554
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Guard load cleanup with try/finally and add retry-contract tests for PR #6554
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add gc.collect retry-step test for PR #6554
* Tighten offline-loading comments and docstrings for PR #6554
* Raise the both-config-failed error before model-type lookup so offline retry fires for PR #6554
* Prefer offline cause for retry and bound export reachability probe for PR #6554
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Skip remote mapper while offline, harden text-load cleanup, and stop stacked offline retries for PR #6554
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Surface VLM fallback offline errors, probe offline before export version activation, and restore progress bars across retries for PR #6554
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Restore offline env after export version activation so the persistent worker re-decides per load for PR #6554
* Classify socket.gaierror and urllib URLError as offline by type for PR #6554
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Probe offline around export load preflights and never offline-retry TLS failures for PR #6554
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Force in-process offline for export preflights, verify proxy egress in probe, and skip caching offline version negatives for PR #6554
* Snapshot offline constants before forcing env and require local processor files for VLM checkpoints for PR #6554
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Keep pad-named pad_tokens; defer pad repair to shared unsloth_zoo.pad_token
A pad-named token (e.g. <|vision_pad|>) is a valid pad. The narrow fallback that
stripped vision pad tokens on text-only models is now a no-op; the active path
delegates to the shared fix_pad_token in unsloth_zoo, which keeps pad-named tokens
and only heals missing / eos-collision / out-of-range pads.
This fixes the Qwen3-4B-Base load crash (its config ships pad_token=<|vision_pad|>):
the old swap could not find a safe text pad (eos is <|endoftext|>, no unk_token) and
left the tokenizer broken. Removes the unused _VISION_PAD_TOKENS / _SAFE_TEXT_PAD_TOKENS
sets. Tests updated.
Pairs with unslothai/unsloth-zoo#831.
* Remove _fix_vision_pad_token; inline the no-op fallback
A pad-named token (e.g. <|vision_pad|>) is a valid pad, so the old vision-pad swap
helper has no purpose. _fix_pad_token now returns the tokenizer unchanged when the
shared unsloth_zoo.pad_token module is unavailable, instead of routing through a
no-op helper. Test WANTED set updated.
* Fix construct_chat_template leaking {INPUT}/{OUTPUT} sentinel into the template
In construct_chat_template's inner process() helper, the branch handling a
section that starts with the {INPUT}/{OUTPUT} sentinel sliced the part from
part.find(which) (which is 0 in that branch), so the literal sentinel was
re-included in the generated Jinja chat template. The endswith branch already
slices correctly with part[:part.find(which)]; this slices past the sentinel
with part[len(which):], so a template whose input or output section begins
with the sentinel (for example a user turn that starts with {INPUT}) renders
correctly instead of emitting a literal {INPUT}/{OUTPUT}.
Added a regression test covering {INPUT}-leading and {OUTPUT}-leading sections.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix SyntheticDataKit.chunk_data dropping single-chunk documents
chunk_data turns the n boundary points from np.linspace into n-1 ranges
via the boundaries[:-1] / [1:] pairing. When a document fits in a single
chunk (n_chunks == 1) that produces zero ranges, so the loop writes no
files and the whole document is silently dropped. Emit the full
[0, length] range when n_chunks <= 1; the multi-chunk path is unchanged.
Added a regression test covering the single-chunk and multi-chunk cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* chunk_data: emit nothing for an empty document (no empty chunk file)
Addresses review feedback: when the input document is empty (length == 0),
return no chunks instead of writing a single empty chunk file. Added a
regression test for the empty-document case.
* chunk_data: reject overlap >= chunk size (non-positive stride)
Per review feedback: when overlap >= max_tokens the chunk stride is
non-positive, which would divide by zero or silently emit one oversized
chunk. Raise a clear RuntimeError for that unusable configuration. Added
a regression test.
* Broaden single-chunk guard to length <= max_tokens (also fixes sub-overlap docs); expand tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
* Fix Qwen3 NaN: self-heal vision pad_token in load_correct_tokenizer
Text-only Qwen3 (and Qwen2.5) models share Qwen3-VL's vocab, so their
Hub tokenizer configs ship <|vision_pad|> as pad_token. Padding text-only
training with a vision token corrupts attention/loss and produces NaN
losses and gradients on affected stacks.
patch_tokenizer already heals this, but only when a model with config is
passed. The standalone load_correct_tokenizer path (and custom training
loops) still returned <|vision_pad|>. This adds a model-independent guard
in load_correct_tokenizer that replaces a vision pad_token on text-only
tokenizers with the first safe text token (<|endoftext|>, <pad>, [PAD],
<unk>), falling back to eos_token only if it differs from pad_token.
The result now matches upstream Qwen configs (pad_token <|endoftext|>,
id 151643) with no new token added. Vision processors (image_processor
present) and non-vision pad tokens (Llama, Qwen2) are left untouched.
Fixes#3155
* Format pad_token helper for ruff kwarg-spacing hook (pre-commit)
* Harden vision pad_token fix: drop unk candidate, guard get_vocab, skip vision eos
* Tighten code comments (no logic change)
* Delegate pad_token fix to shared unsloth_zoo.pad_token
Generalize the narrow _fix_vision_pad_token by delegating to unsloth_zoo's
shared fix_pad_token (single source of truth, AGPL-3.0), which scans the
reserved-token families instead of only the vision-pad case. A guarded import
keeps this working against an older unsloth_zoo that has not shipped the module
yet: on ImportError it falls back to _fix_vision_pad_token.
allow_add=False is passed so the early load_correct_tokenizer call stays
side-effect free (no model here to resize embeddings); the later model-aware
patch_tokenizer call finishes the job and is idempotent.
Adds tests/python/test_pad_token_fix.py covering both dispatch paths offline.
* Fix CPOTrainer crash with multimodal processors
CPOTrainer shares build_tokenized_answer/tokenize_row and __init__ with
ORPOTrainer, but the ORPO replacement functions that route tokenization
through the underlying text tokenizer and resolve pad_token_id were only
registered for orpo_trainer. With a multimodal processing class (e.g.
Gemma4Processor) the positional self.processing_class(prompt, ...) call binds
prompt to images=, leaving text=None and raising
TypeError: 'NoneType' object is not subscriptable.
Register the existing orpo_trainer_text_tokenizer and
orpo_trainer_processor_pad_token under cpo_trainer as well so CPO/SimPO
fine-tuning of multimodal models works. No change for plain tokenizers.
* Add CPO processor tokenizer regression test
Static, CPU-only checks that cpo_trainer registers the same
orpo_trainer_text_tokenizer and orpo_trainer_processor_pad_token rewriters as
orpo_trainer, and that the rewriter drops the broken positional
self.processing_class(prompt, ...) call. Guards against issue #4952 regressing.
* Format CPO test assert for ruff line length (pre-commit)
* Bind CPO __init__ pad/eos token reads to underlying tokenizer
TRL 0.28+ CPOTrainer.__init__ reads bare processing_class.pad_token and
processing_class.eos_token before pad_token_id, which raises AttributeError
for multimodal processors (e.g. Gemma) where those live on .tokenizer.
Extend orpo_trainer_processor_pad_token to route that block through the
underlying tokenizer, and add a regression test.
* Tighten code comments (no logic change)
* Make CPO/ORPO rewriters reach the trainer on TRL 1.x
TRL 1.x moved CPOTrainer and ORPOTrainer out of trl.trainer into
trl.experimental.<algo> and dropped the trl.trainer.<algo>_trainer shim that
older TRL (0.26 - 0.28) kept. patch_trl_rl_trainers() discovers trainers via
dir(trl.trainer), so on TRL 1.x cpo_trainer and orpo_trainer are never found and
the multimodal-processor tokenization fix (#4952) silently stops applying, even
though the rewriters themselves still match the source.
Re-expose experimental-only trainers that Unsloth has rewriters for (RL_FUNCTIONS
keys) under trl.trainer before discovery, so the existing patch machinery and its
thin-wrapper resolution work unchanged. The alias is a no-op on older TRL where
trl.trainer.<algo>_trainer already exists.
Also rebind the patched Trainer/Config into every already-imported trl.* module
that holds the original class so the fix is visible at the experimental import
site (from trl.experimental.cpo import CPOTrainer), not only via trl.trainer.
Verified on transformers 4.57.6 + trl 0.22.2, transformers 4.57.6 + trl 0.27.1,
and transformers 5.12.1 + trl 1.6.0: CPOTrainer with a multimodal processor
tokenizes through the underlying text tokenizer with no crash on all three, and
the SFT/GRPO/DPO patch paths are unchanged.
* Format for ruff (pre-commit)
* Simplify CPO fix to mirror ORPO registrations (#4952)
Register the existing ORPO row-tokenizer/pad-token rewriters for cpo_trainer.
Under the trl<=0.24.0 pin CPOTrainer lives in trl.trainer.cpo_trainer (found by
dir(trl.trainer)), shares ORPO's build_tokenized_answer and uses
processing_class.pad_token_id, so the two registrations are sufficient.
Drop the trl 1.x experimental aliasing/rebind machinery in rl.py and the
bare-pad_token rewriter: trl 1.x (CPO in trl.experimental) and the bare
pad_token pattern (trl>=0.28) are not installable under the pin.
* CPO: route bare pad_token/eos_token default through inner tokenizer
TRL 1.x CPO/ORPO __init__ (the trl.experimental source unsloth resolves on TRL
0.26+) defaults processing_class.pad_token from processing_class.eos_token
before tokenizing. Multimodal processors (Gemma3/Gemma4 Processor) expose those
attributes on .tokenizer, not on the processor, so that bare access raises
AttributeError during __init__ even with the pad_token_id fallback registered.
Extend orpo_trainer_processor_pad_token to rewrite that defaulting block to run
on the inner tokenizer. The pinned TRL range (<=0.24.0) has no such block, so
the regex is a no-op there and only the existing pad_token_id fallback applies.
Verified the rewrite against the real trl 1.6.0 experimental CPOTrainer.__init__
(bare access removed, result compiles, a processor without pad_token no longer
raises) and added offline regression tests for both the rewrite and its no-op.
---------
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
* vision/llama: thread finetune_audio_layers through get_peft_model
Pass finetune_audio_layers to get_peft_regex so FastModel / FastVisionModel can
adapt the Gemma 4 / Gemma 3N audio encoder and the audio/vision embedder
projectors. Enabled automatically under target_modules="all-linear", included in
the explicit-target layer-scope filter, and added to the UNSLOTH_USE_NEW_MODEL
kwargs defaults. Requires the matching unsloth-zoo get_peft_regex change.
* vision: address review feedback on finetune_audio_layers threading
- Drop finetune_audio_layers from the explicit-target_modules filter condition: it
defaults False, so including it forced every explicit list through get_peft_regex
and printed the scope warning even for text / vision models that never opted in.
- Guard the kwarg by get_peft_regex's signature so an older unsloth_zoo without the
parameter does not raise TypeError; warn once if requested but unsupported.
* vision: honor finetune_audio_layers with explicit targets; fail fast on old zoo
- Route an explicit target_modules list through get_peft_regex when audio is opted
in (positive `or finetune_audio_layers`), so finetune_audio_layers=True is no
longer a no-op for explicit lists. The scope warning is printed only when a layer
family is actually disabled, so default text/vision calls are not spammed.
- Raise (instead of warn) when finetune_audio_layers=True but the installed
unsloth_zoo lacks the parameter, rather than silently training a language-only
adapter.
* vision: keyword-safe audio flag, graceful all-linear on old zoo
- Move finetune_audio_layers to the end of get_peft_model's signature so
positional callers of finetune_last_n_layers (and the params after it) keep
their slots.
- Track whether the caller EXPLICITLY requested audio before all-linear flips
the flag on, and only raise the 'needs newer unsloth_zoo' error for explicit
requests. A plain all-linear (text/vision) run now degrades gracefully on an
old unsloth_zoo instead of failing.
* Fix FlashAttention fp32 crash with DoRA (use_dora=True)
DoRA upcasts lora_magnitude_vector to fp32 for the optimizer, which promotes
the q/k/v_proj output to fp32. FlashAttention only accepts fp16/bf16, so the
fp32 q/k/v raised 'FlashAttention only support fp16 and bf16 data type'.
Downcast q/k/v to the compute dtype before the flash kernels.
Fixes#1013
* Apply kwarg-spacing format hook to DoRA dtype test (pre-commit)
* DoRA+FA2: downcast any fp32 among Q/K/V and clamp to a flash-supported dtype
* Tighten code comments (no logic change)
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* Fix misleading 'only for image models' error for Qwen3-VL when torchvision is missing
transformers >= 5.4 hard-requires torchvision for VLM image/video processors and
no longer falls back to a slow processor. Without torchvision the processor load
raises ImportError, unsloth degrades to a text-only tokenizer, and the vision data
collator later fails with 'UnslothVisionDataCollator is only for image models!'.
Detect this case at load time and raise a clear, actionable error pointing at the
missing torchvision dependency instead.
Fixesunslothai/unsloth#4202
* Apply kwarg-spacing format hook to vision torchvision guard (pre-commit)
* Make torchvision-missing detection precise: check availability first, match specific error text
* Tighten code comments (no logic change)
* Make missing-torchvision VLM error version-agnostic
The raise also fires on transformers 4.57.x for VLMs with a video processor
(Qwen2.5-VL, Qwen3-VL), where AutoVideoProcessor requires torchvision. The old
message claimed 'transformers >= 5.4 requires torchvision', which is inaccurate
on 4.57.x. Reword to state torchvision is required for this model's vision
processors without a version-specific claim.
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* Fix GRPOTrainer evaluate() crash when called without prior training
GRPOTrainer.compute_loss read self.current_gradient_accumulation_steps
directly. That attribute is only set by the transformers training loop, so
calling trainer.evaluate() standalone (no prior trainer.train()) raised
AttributeError. Read it via getattr with a fallback to
args.gradient_accumulation_steps so standalone evaluation works.
* GRPO eval: fall back accumulation steps to 1 so standalone eval_loss is not underreported
* Tighten code comments (no logic change)
* fix: use EMPTY_LOGITS on the fused-CE not-return_dict path (#2068)
CausalLM_fast_forward's fused cross-entropy path (small batch, labels set,
UNSLOTH_RETURN_LOGITS off) computes the loss straight from hidden_states
via unsloth_fused_ce_loss and never materializes `logits`. The
return_dict=True branch returns EMPTY_LOGITS, but the `not return_dict`
branch returned `(logits,) + outputs[1:]`, raising
"UnboundLocalError: cannot access local variable 'logits'" whenever it ran
(e.g. training with return_dict=False). Same bug in the llama and mistral
fast-forward paths.
Return EMPTY_LOGITS on that branch too, matching the adjacent return_dict
output. Verified on GPU: a forward(return_dict=False, labels=...) that
raised UnboundLocalError now returns (loss, EMPTY_LOGITS, ...) and
backward() succeeds.
Adds tests/test_fused_ce_not_return_dict_logits.py, a CPU source-drift guard
(the fused path itself is GPU/triton only) asserting both fast-forward paths
keep using EMPTY_LOGITS there.
* Address review: parse the fused-CE drift line with whitespace-tolerant regexes
The drift detector sliced the source with exact string matching
(source.index("output = (") + the next newline), so a formatter respacing or
rewrapping the assignment would break the parse. Switch to anchored regexes that
tolerate whitespace and line wrapping, keeping the match anchored after the
fused guard so it targets the fused-CE branch and not the normal
output = (logits,) path. Behavior and the two drift assertions are unchanged.
* Tighten code comments (no logic change)
---------
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
* Reset torch.compile cache poisoned by a stray forward before trainer.train()
A manual forward / forward+backward run under model.train() before
trainer.train() (for example a pre-train grad-norm probe like
out = model(**batch); out.loss.backward()) silently poisons training when
torch.compile is enabled. The stray training-mode pass is the first one in the
process, so it compiles and caches the model forward and, via AOTAutograd, its
backward graph in a one-off context that does not match the real training loop.
When trainer.train() reuses that cached graph the gradients come out NaN/Inf,
the loss never moves, and the run looks like it trains but never learns.
Observed on gpt-oss-20b (loss frozen at ~4.25, grad_norm NaN from step 1) with
both use_gradient_checkpointing="unsloth" and =True. It does not reproduce when
the probe runs under torch.no_grad(), nor with UNSLOTH_COMPILE_DISABLE=1, and a
single torch._dynamo.reset() before training fully cures it (loss 4.29 -> 0.0002,
identical to a run with no probe). Resetting the gradient-checkpointing buffers,
zero_grad, empty_cache, or for_training does not help, confirming the corruption
lives in the torch._dynamo / torch.compile cache.
get_peft_model now attaches a one-shot forward pre-hook that records whether a
forward ran before train(). prepare_for_training_mode checks it at the start of
train() and, if a pre-train forward was seen and torch.compile is enabled, calls
torch._dynamo.reset() (plus a pristine gradient-checkpoint reset and zero_grad)
and warns once. On the normal path (no pre-train forward) it is a strict no-op:
no dynamo reset, no recompilation, identical loss curve.
* Ignore no-grad pre-train probes and detect probes across the wrapper chain
A no-grad forward (with torch.no_grad(): model(**batch)) builds no AOTAutograd
backward graph, so it cannot poison the compiled training graph. Gate the marker
on torch.is_grad_enabled() so such probes no longer trigger a needless dynamo
reset, recompile and warning on an otherwise clean run.
Also walk the model wrapper chain (PeftModel / DDP / base model) when resetting
so a probe that ran on a different wrapper than self.model is still detected, and
tear down every detector hook in the chain. Re-installing the detector is now
idempotent and only re-registers when a prior hook was already removed.
* Walk DDP/FSDP .module when scanning for the pre-train marker
The chain walk followed only .model and .base_model, so a probe that fired on the
model below a DDP/FSDP wrapper (which exposes it via .module) left the marker
undetected and the poisoned compile cache un-reset. Add .module to the walk.
* Install pre-train detector on the full-finetuning path too
get_peft_model returns early when UNSLOTH_ENABLE_FULL_FINETUNING=1, before the
detector was installed, so full-finetuning runs (which still use torch.compile) did
not drop a graph cache poisoned by a stray pre-train forward. Install the detector
before both full-finetuning early returns (FastLlamaModel and FastBaseModel). The
detector is idempotent, so this never stacks duplicate hooks when get_peft_model is
also called on a LoRA model.
* torch.compile stray-forward reset: tighten comments (no code change)
* Wire stray-forward compile-cache reset into SFT path and PEFT pass-through
The pre-train forward detector is installed for plain LoRA/vision models in
get_peft_model, but only RL trainers ran the reset via prepare_for_training_mode.
A grad-enabled probe before SFTTrainer.train() therefore left the poisoned Dynamo
cache in place and the detector hook running on every training forward.
- trainer.py: wrap SFTTrainer.train to run _unsloth_reset_stray_compile_cache,
which both drops the poisoned cache and tears down the detector hook. For
UnslothSFTTrainer the later prepare_for_training_mode assignment supersedes it.
- llama.py: arm the detector before the 'Already have LoRA adapters' early return
so pre-wrapped PEFT models keep the reset capability.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Preserve detector evidence on reinstall + wire reset into plain Trainer path
P2 (_utils.py): _unsloth_install_pretrain_detector cleared marker['seen'] before the
live-hook early return, so a re-entrant get_peft_model/patch_peft_model after a
grad-enabled probe erased the recorded poisoning while leaving the hook installed,
and train() then skipped the Dynamo reset. Only reset seen when (re)installing a
fresh hook; keep it when a live hook is already recording.
P2 (llama.py): the detector is armed for every LoRA model, but only TRL SFT/RL train
wrappers consumed it. Inject _unsloth_reset_stray_compile_cache(self) at the start of
the generated _fast_inner_training_loop so a bare transformers.Trainer.train() also
drops a poisoned cache and tears down the hook. Idempotent with the TRL-wrapper reset.
* Make _unsloth_reset_stray_compile_cache an importable module-level helper
The reset was only defined inside the RLTrainer_replacement template string, so
'from unsloth.models.rl import _unsloth_reset_stray_compile_cache' raised ImportError
(swallowed) on the SFT auto-packing wrapper and the injected plain-Trainer loop -
both paths kept the poisoned Dynamo cache and the dangling detector hook.
Move the canonical implementation to unsloth.models._utils (next to the detector,
exported in __all__). The RL trainer template now imports it (no-op fallback if the
import ever fails), and trainer.py / llama.py import it from _utils too, so every
training entry point actually runs the reset.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The gpt-oss dtype branch selected the BnB UNSLOTH_FORCE_CUSTOM_DTYPE path on the
raw load_in_4bit flag. A native MXFP4 checkpoint loaded by exact name with the
default load_in_4bit=True (e.g. openai/gpt-oss-20b) keeps the flag set until
check_and_disable_bitsandbytes_loading runs inside FastBaseModel, which is after
this branch sets the env var. So MXFP4 gpt-oss got the BnB down_proj/router
compute-dtype path (which also calls dequantize_module_weight on a non-bnb module)
instead of the MXFP4 bias upcast. Gate on the effective bnb state
(load_in_4bit and _bnb_compatible_quant), mirroring the _load_in_4bit_ token gate
introduced in #6504. BnB-4bit and -BF16 gpt-oss are unchanged.
* Windows installer: fix DiskPart UAC, drive-root cache, spurious rename warning, CPU-base messaging
amd-smi gate (DiskPart UAC mid-install): the AMD torch wheel ships hipInfo.exe
inside the venv, and the bitsandbytes fix prepends that venv Scripts dir to PATH.
shutil.which("hipinfo") then found it and flipped _amd_smi_allowed() to True, so
the post-install AMD probe fell through to `amd-smi list` (the venv hipInfo failed
to report gcnArchName, which is why the arch came from the GPU-name table) and
amd-smi elevated, popping the DiskPart UAC. Fix: a hipinfo resolved inside the
active venv (sys.prefix) is the torch-wheel binary, not a HIP SDK, and must not
open the gate. Mirrored in install_python_stack.py, install_llama_prebuilt.py, and
backend utils/hardware/amd.py (the runtime VRAM poller had the same latent prompt).
TORCHINDUCTOR_CACHE_DIR: move from C:\tc to <StudioHome>\TORCHINDUCTOR_CACHE_DIR so
the inductor/Triton cache lives under the user's Studio home, not the system drive
root. Long paths are already enabled above so deep inductor paths still fit.
unsloth.exe rename: skip the rename (and its "pip may fail with WinError 32"
warning) when SKIP_STUDIO_BASE=1. In the install.ps1 flow base packages are not
reinstalled, so unsloth.exe is never rewritten; the self-rename only failed because
setup runs via unsloth.exe (the running launcher holds its own file). The
'studio update' flow still attempts it.
CPU PyTorch messaging: clarify that the CPU base is temporary and setup replaces it
with GPU ROCm wheels, and print an explicit "GPU ROCm PyTorch installed" line after
the AMD wheels land, so the log makes clear the final install is GPU-accelerated.
Adds two regression tests covering the venv-internal vs external hipInfo gate.
Verified end-to-end on a Strix Halo box (Radeon 8060S / gfx1151): install.ps1
--local from this branch completed exit 0 with no DiskPart prompt, no rename
warning, the cache under the Studio home, and "GPU ROCm PyTorch installed
(gfx1151)"; Studio then booted and detected "ROCm (HIP 7.13.99004) -- AMD Radeon
8060S Graphics".
* Windows installer: drop the unreliable unsloth.exe rename and its WinError 32 warning
setup.ps1 used to rename the running unsloth.exe out of the way before the
base-package upgrade so pip could replace it. That rename never actually
worked: setup runs *via* unsloth.exe, so renaming our own running
uv-trampoline launcher failed with a sharing violation (WinError 32) and only
printed a scary 'could not rename unsloth.exe; pip may fail with WinError 32'
warning on every Windows install and update.
It also was not needed. pip tolerates a running/locked console-script .exe: it
moves the old one aside and writes the new one. The base upgrade routes through
pip on Windows, so the upgrade succeeds (or, in the install.ps1 flow with
SKIP_STUDIO_BASE=1, the base is not touched at all) and unsloth.exe is left
intact either way.
Removing the rename block and its failed-install restore block removes the
false warning for all Windows devices in both the install and update flows.
* Windows installer: gate venv-internal hipInfo.exe in PowerShell amd-smi probe; harden venv path checks
Follow-up to PR #6296.
- install.ps1 and setup.ps1: ignore the AMD torch wheel hipInfo.exe that lives
inside the Studio venv when probing for a HIP SDK, so amd-smi no longer reopens
the DiskPart UAC during install/update. Mirrors _path_inside_venv in the Python
installers, which already do this.
- amd.py, install_llama_prebuilt.py, install_python_stack.py: normcase the venv
containment check (Windows paths are case-insensitive) and run the
HIP_PATH/ROCM_PATH candidate through it too.
- setup.ps1: fall back to a short TORCHINDUCTOR cache dir when long paths are
unavailable, and create the dir wildcard-safely.
- tests: isolate sys.prefix in the gate helper, add HIP_PATH/ROCM_PATH cases, and
assert the PowerShell venv exclusion.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Windows installer: install ROCm PyTorch directly for a known AMD arch
When the GPU arch is known (name-inferred from the GPU-name table) but ROCm
could not be probe-verified (no HIP SDK, no amd-smi), the bootstrap installed
a CPU PyTorch base that setup.ps1 then force-reinstalled as ROCm. The
repo.amd.com wheels bundle their own runtime (no HIP SDK required), which
setup.ps1 already relies on, so the CPU base was a pure wasted download/install.
- Gate the ROCm index on a known arch, not only on probe-verified ROCm, so a
mapped arch installs ROCm torch directly. Unmapped arches and no-GPU hosts
still get CPU (unchanged).
- Fall back to a CPU base if the ROCm-index install fails, so a transient
repo.amd.com outage does not abort the install (setup.ps1 retries ROCm).
- Correct the stale comment that claimed ROCm wheels need a confirmed HIP SDK.
- Add a regression test for the arch-based gate.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Windows installer: correct the unsloth.exe rename-removal comment
The comment claimed the base upgrade 'routes through pip on Windows' and that
pip 'moves the old unsloth.exe aside, then writes the new one'. That is not what
the code does. install_python_stack tries uv first; on a locked launcher uv
aborts and falls back to pip, but the pip fallback strips --upgrade-package and
base.txt lists only bare unsloth/unsloth-zoo, so pip finds them already
satisfied and no-ops. The running unsloth.exe is left intact at its current
version either way. Reword the comment to describe the real uv-first /
pip-fallback-no-op behavior. No functional change.
* Windows installer: close two gaps in the venv-internal hipinfo exclusion
Review follow-up. The amd-smi/DiskPart gate could still reopen in two cases:
- setup.ps1 ran the HIP probe long before $VenvDir is assigned, so without
VIRTUAL_ENV (the `unsloth studio update` path) $venvRoots was empty and the
venv-internal hipInfo.exe was not recognized. Seed the venv root from
UNSLOTH_SETUP_PYTHON and the default Studio home too (both installers).
- The HIP_PATH/ROCM_PATH candidate was accepted without the venv filter, so an
env var pointing into the venv (AMD wheel) still set $HipSdkInstalled. Run
Test-HipinfoIsVenvInternal on the candidate as well (both installers).
Extend the PS gate test to assert both. Both .ps1 parse clean; install tests
pass (the venv-internal / HIP probe coverage at 359 passed).
* Windows installer: correct the CPU-base message for arches with no ROCm wheels
After gating the ROCm index on a known arch, a mapped arch sets $ROCmIndexUrl
and installs ROCm directly, so it no longer reaches the "temporary CPU base"
branch. That branch is now reached only by a name-inferred arch with no ROCm
wheels (e.g. RDNA2 gfx103X), where setup.ps1 does NOT install ROCm. The old
text ("setup replaces it with GPU ROCm wheels ... the final install IS
GPU-accelerated") was therefore always wrong there. Say plainly that PyTorch
stays on CPU for this GPU.
* Windows installer: seed the venv-internal hipInfo check from a custom Studio home
Test-HipinfoIsVenvInternal seeded the venv root from VIRTUAL_ENV, VenvDir, the
setup python, and the default %USERPROFILE% path only. A standalone
`unsloth studio update` with a custom UNSLOTH_STUDIO_HOME (or STUDIO_HOME alias)
and none of those set would not recognize the venv hipInfo on PATH, reopening the
amd-smi/DiskPart gate. Seed the custom home too, in both installers, and assert
it in the gate test.
* Studio installer: resolve venv aliases and expand ~ in the hipInfo venv filter
Two review points on the amd-smi/DiskPart UAC gate:
1. _path_inside_venv compared os.path.abspath of sys.prefix and the hipInfo
path, which does not resolve symlinks, junctions, or 8.3 short names. A venv
reached through an aliased path then fails the check, so its bundled
hipInfo.exe is mistaken for an external HIP SDK and amd-smi runs (the
DiskPart prompt this fix exists to suppress). Switch to os.path.realpath in
all three copies (amd.py, install_llama_prebuilt.py, install_python_stack.py).
2. setup.ps1's early venv-internal hipInfo probe seeded the venv root from a
custom Studio home (UNSLOTH_STUDIO_HOME / STUDIO_HOME) without expanding a
leading ~, while the canonical resolver does. With a tilde form,
[IO.Path]::GetFullPath kept the literal ~ relative to cwd, so the custom-home
hipInfo escaped the filter and reopened the gate. Expand ~ in the probe the
same way as the resolver.
tests/studio/install/test_pr5940_followups.py: 30 passed (adds a symlink
realpath case and a setup.ps1 tilde-expansion guard).
* Studio installer: mirror the hipInfo venv filter and ROCm wheel pins into install.ps1
Follow-up review on the same install.ps1 paths:
1. install.ps1's venv-internal hipInfo probe (Test-HipinfoIsVenvInternal)
seeded the venv root from a custom Studio home without expanding a leading
~, unlike the canonical resolver and setup.ps1. A tilde form left
[IO.Path]::GetFullPath with the literal ~ (relative to cwd), so the
custom-home hipInfo escaped the filter and reopened the amd-smi/DiskPart
gate. Expand ~ in the probe, matching the setup.ps1 fix.
2. The AMD ROCm path installed torchvision/torchaudio bare while pinning torch
to below 2.12. AMD's per-arch index publishes the companions independently
and may ship torchvision 0.27 (for torch 2.12) before removing 0.26, so a
bare resolve can pick an ABI-incompatible set and fall back to CPU. Add
torchvision/torchaudio floor maps and pass the pinned specs, mirroring
setup.ps1 and install_python_stack.py.
3. The ROCm-to-CPU fallback torch install used Invoke-InstallCommand (no
retry), the only torch step in the file without it. Switch to
Invoke-InstallCommandRetry so the recovery path survives a transient index
failure.
tests/studio/install/test_pr5940_followups.py: 33 passed (parametrized tilde
check over both installers, a torch/companion floor-map parity test, and a
CPU-fallback retry guard).
* Studio installer: scan all PATH hipinfo so the venv copy can't shadow a real HIP SDK
The amd-smi HIP-SDK probe used shutil.which("hipinfo") / Get-Command hipinfo,
which return only the first hit on PATH. The AMD torch wheel ships hipInfo.exe
inside the venv and the bnb fix (plus the Studio backend) prepend the venv
Scripts dir to PATH, so that venv-internal copy lands first. When a real HIP SDK
hipinfo sits later on PATH with HIP_PATH/ROCM_PATH unset, the first-hit probe
stopped at the venv copy, treated it as "not a HIP SDK", and closed the amd-smi
gate -- AMD users in that PATH-only SDK setup lost amd-smi telemetry and could
fall back to CPU. Scan every PATH entry and keep the first hipinfo that is not
venv-internal; only the venv copy is ignored, so the UAC/DiskPart suppression is
unchanged.
Applied to all three Python copies (install_llama_prebuilt.py,
install_python_stack.py, backend/utils/hardware/amd.py) via a new
_external_hipinfo_on_path helper, and both PowerShell callers (install.ps1,
setup.ps1) now use Get-Command hipinfo -All filtered by Test-HipinfoIsVenvInternal.
tests/studio/install/test_pr5940_followups.py: 36 passed (real-PATH scan tests, a
shadow-regression test for the exact venv-first ordering, and a parity check that
every Python copy uses the scanning helper).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio uninstallers: fix leftovers (false "removed", shared icon, llama lock)
Auditing a dual native+WSL uninstall on a real device surfaced three leftovers:
1. uninstall.ps1 removed the data dir (which holds unsloth.ico) before the
shortcuts that reference that icon, so Explorer's icon cache briefly held it
open. Remove-Item -Recurse reported success yet left the locked file, and the
dir was never re-attempted, so it orphaned with a false "removed" log.
_RemovePath now verifies the path is actually gone (retrying transient locks)
and reports honestly, and the data dir is re-swept after the shortcuts go.
2. install.sh writes a shared unsloth.ico to %LOCALAPPDATA%\Unsloth Studio for
the WSL shortcut, but uninstall.sh never removed it, orphaning the icon (and
dir) after a WSL uninstall. uninstall.sh now drops that icon and the dir when
empty, in both the powershell.exe and drvfs-fallback paths.
3. ~/.unsloth/.llama.cpp.install.lock was never removed, so the rmdir of
~/.unsloth failed and the dir lingered. Both uninstallers now remove the lock.
Verified by running both uninstallers on a real dual install: device fully clean
(no install dirs, shortcuts, PATH/registry entries, shared icon, or lock left).
* install.sh: auto-route Strix Halo WSL to an existing Ubuntu 24.04
ROCm-on-WSL is the GPU runtime for Strix Halo and only targets Ubuntu
24.04. When the installer runs in a newer default distro (e.g. 26.04) it
cannot enable the GPU and silently falls back to CPU. If a 24.04 distro
already exists, re-run the install there and stop in the current one so the
GPU path is taken without the user having to know about the distro
requirement.
Runs before venv creation so the wrong distro is left untouched, guards
against re-route loops via UNSLOTH_WSL_REROUTED, leaves a working ROCm
distro alone (librocdxg present), and skips the GGUF-only / opt-out /
non-Strix cases. When no 24.04 distro exists we keep today's behaviour:
continue to CPU and print the `wsl --install Ubuntu-24.04` guidance, never
auto-downloading a distro.
Adds tests/sh/test_strixhalo_wsl_reroute.sh (hermetic: extracts the
function, rewrites its paths to fixtures, mocks wsl.exe) covering the full
decision matrix, wired into tests/run_all.sh.
* uninstall.ps1: keep shared unsloth.ico for a surviving WSL shortcut
A dual native+WSL install shares %LOCALAPPDATA%\Unsloth Studio\unsloth.ico:
install.sh points the WSL shortcut's icon there while the native install owns the
dir. The native uninstaller removed the whole dir unconditionally, so uninstalling
native while keeping WSL left the WSL shortcut with a blank icon. The old code only
avoided this when Explorer happened to hold the icon open, which is unreliable; on a
real dual install the dir was deleted and the WSL shortcut went blank.
_RemoveDataDirKeepingWslIcon now scans the Start Menu + Desktop for a surviving
"Unsloth Studio (WSL ...).lnk" and, if found, removes everything in the data dir
except unsloth.ico (keeping the dir) instead of deleting it; with no WSL shortcut it
removes the dir as before. uninstall.sh still drops the icon and the empty dir when
WSL itself is uninstalled, so every uninstall order ends clean.
Adds tests/studio/test_uninstall_dual_install_icon.ps1 (AST-extracts the helper and
runs it against a temp dir with controlled shortcut dirs) covering the dual,
native-only, empty, and missing-dir cases, wired into the windows-inference smoke
workflow. Verified on a real dual install: native uninstall now keeps unsloth.ico
and the WSL shortcut's icon stays intact.
* installer: condense AMD/ROCm code comments (no behavior change)
Tighten the comments added for the Strix Halo native+WSL installer work so
they are shorter and clearer without losing intent: the venv-internal hipInfo
amd-smi gate, the ROCm torch/companion floor maps, the WSL 24.04 reroute, and
the dual-install uninstall icon handling. Comment-only; code paths unchanged.
107 insertions, 166 deletions across 11 files.
* install.sh: run the Strix Halo WSL reroute before any STUDIO_HOME write
The reroute fired after mkdir -p "$STUDIO_HOME" and the legacy-venv migration,
so rerouting 26.04 -> 24.04 left an empty ~/.unsloth/studio stub in the origin
distro (and ran venv migration in the distro about to be abandoned). Move the
reroute ahead of the venv section so the origin distro is left untouched, matching
the function's own comment. Behavior is identical on every non-reroute path.
* installer: fix ROCm CPU-fallback, hipinfo gate edge cases, uninstall icon, WSL 22.04
- install.ps1: clear $ROCmIndexUrl/$ROCmTorchFloor after the CPU fallback so the
flavor-repair block does not retry the failed ROCm index and abort the install;
pin the ROCm companion specs ($visionSpec/$audioSpec) in the repair path too.
- install.ps1 + setup.ps1: skip a bare drive root in Test-HipinfoIsVenvInternal so a
non-venv UNSLOTH_SETUP_PYTHON does not match the whole drive; iterate
HIP_PATH/HIP_PATH_57/ROCM_PATH and take the first non-venv hipinfo.
- amd.py, install_llama_prebuilt.py, install_python_stack.py: strip surrounding
quotes from PATH entries before probing for hipinfo.
- install.sh: pipefail the WSL reroute curl|sh; do not reroute supported Ubuntu 22.04.
- uninstall.sh: keep the shared unsloth.ico while any Unsloth shortcut (native or
another WSL distro) still references it, in both the powershell and drvfs paths.
- tests: regression coverage for all of the above.
* installer: forward reroute options, guard ROCm bootstrap, harden hipinfo gate
- install.sh: forward the caller's --package/--python/--verbose/--tauri and a custom
UNSLOTH_STUDIO_HOME into the WSL reroute (was a bare default install); bail on
--local; run the reroute BEFORE dependency/uv install so the origin distro is left
untouched; set UNSLOTH_SKIP_ROCM_WSL_SETUP after a failed reroute so the later
ROCm-on-WSL bootstrap does not install into the unsupported origin distro.
- install.ps1 + setup.ps1: Get-Command hipinfo -CommandType Application so only real
executables match (not an alias/function named hipinfo).
- uninstall.ps1: guard $env:APPDATA when building the default shortcut search dirs.
- tests: cover option forwarding, --local bail, the bootstrap guard, and the gate change.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* installer: guard origin ROCm bootstrap on every CPU-only fallback; harden ~ expansion
WSL reroute: the no-wsl.exe, no-24.04-target and --local fallbacks all tell the
user the install continues CPU-only, but only the failed-reroute branch set
UNSLOTH_SKIP_ROCM_WSL_SETUP=1. The later _maybe_bootstrap_rocm_wsl gate keys off
that flag, so the other three branches could still install ROCm into the
unsupported origin distro (e.g. 26.04). Set the skip guard on all of them.
Forward UNSLOTH_ROCM_WSL_AUTO into the reroute so a Tauri/consented GPU bootstrap
carries through to the rerouted 24.04 child instead of dropping to the prompt path.
install.ps1/setup.ps1: guard the venv-probe ~ expansion on a non-empty
$env:USERPROFILE so Join-Path does not throw on a profile-less service account.
Tests: add no-wsl.exe and UNSLOTH_ROCM_WSL_AUTO reroute cases, the USERPROFILE
guard assertion, and route shell-test fixtures through a single trap-cleaned root.
* installer: pin + soften Windows ROCm Python repair, reroute to 22.04, harden gates
install_python_stack.py: the Windows AMD ROCm repair in _ensure_rocm_torch()
installed bare torch/torchvision/torchaudio via the fatal pip_install -- the same
asymmetry already fixed on the PowerShell side. A transient repo.amd.com failure
could abort the whole install even after install.ps1/setup.ps1 fell back to CPU.
Pin companions per-arch (gfx120X/Strix -> the rocm7.2 trio, mirroring the PS floor
maps) and make the retry nonfatal: keep the existing build and let the user re-run
update to retry ROCm, so the chain install.ps1 -> setup.ps1 -> stack stays CPU-safe.
install.sh: reroute now targets an installed Ubuntu 24.04 OR 22.04 (24.04 preferred);
both are AMD-supported for ROCm-on-WSL, matching the leave-alone set, so a box with
only 22.04 reaches the GPU instead of staying CPU-only.
install.ps1/setup.ps1: a bare ~ for UNSLOTH_STUDIO_HOME left an empty Join-Path child
(PS 5.1 throws); fall back to USERPROFILE directly and only join a real remainder.
_path_inside_venv (amd.py + both installers): guard a root-dir sys.prefix so commonpath
can't classify every path on the drive as venv-internal (defensive; venv never at root).
uninstall.sh: guard an empty LOCALAPPDATA in the PS-interop icon cleanup (mirror APPDATA).
Tests: add 22.04-target reroute cases, Windows ROCm pin+nonfatal coverage (text +
behavioral), root-dir guard coverage, and bare-~/LOCALAPPDATA guard assertions.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* install.sh: match WSL reroute target by exact distro name, not substring
The 24.04/22.04 reroute target was chosen with grep -F (substring), so a custom
distro such as 'Ubuntu-24.04-test' (with no exact Ubuntu-24.04) was picked as the
target; the later 'wsl -d Ubuntu-24.04' then fails and the Strix Halo install stays
CPU-only. Match whole lines (grep -ixF) and reuse the matched name so only a real
Ubuntu-24.04/22.04 is targeted. Adds substring-rejection + exact-vs-custom tests.
* install.sh: keep the WSL reroute target to Ubuntu 24.04 (helper-supported only)
The ROCm-on-WSL bootstrap (scripts/install_rocm_wsl_strixhalo.sh) dies on any
VERSION_ID other than 24.04 and pins the noble repo, so treating 22.04 as
GPU-supported let the parent report a successful reroute while the child fell
back to CPU. Drop 22.04 from the supported set and the reroute target list;
24.04 stays the sole target (keeping the exact whole-line distro match). An
already-working ROCm on any other version is still left alone by the librocdxg
check above.
tests: reroute 22.04 cases updated to the 24.04-only behavior; make the
"no wsl.exe" case hermetic so a real host wsl.exe can't leak in on dev boxes;
stop the tauri exit-order check from mis-flagging the reroute helper's
[ "$TAURI_MODE" = true ] && ... --tauri one-liner.
* installer: tighten comment wording across the Strix Halo install/uninstall paths
Condense the verbose multi-line comment blocks (amd-smi hipinfo gate, ROCm
torch install + CPU fallback, WSL reroute, uninstall icon-keep) into fewer,
clearer lines. Comments and a few docstrings only; no code, logic, or
behavior change. Verified with bash -n, the PowerShell parser, and ast.parse,
and the installer test suite still passes.
* add AGPL-3.0 SPDX headers to the .sh/.ps1 scripts missing them
Every shell and PowerShell script under the Studio/installer surface now
carries the standard SPDX-License-Identifier: AGPL-3.0-only + copyright
header (after the shebang where present): the installer (install.sh,
install.ps1), build.sh, the .github and src-tauri scripts, the installer
test suite, and the moe kernel test. Header-only, line endings preserved;
bash -n, the PowerShell parser, and the installer tests all pass.
* installer: drop the duplicate AGPL header from install.sh and install.ps1
Both already carry an SPDX-License-Identifier: AGPL-3.0-only header below
their usage comment block; the prior header pass added a second one at the
top because it only scanned the first few lines. Remove the duplicate so each
file keeps a single original header.
* installer: force-reinstall CPU fallback torch; propagate Tauri NEED_SUDO from reroute
install.ps1/setup.ps1: when the AMD ROCm wheel install fails and we fall back to a
CPU base, force-reinstall the torch/vision/audio triplet. A failed ROCm install can
leave an unpinned ROCm torch (e.g. 2.10.0+rocm on gfx110X/gfx90a) that still
satisfies the CPU torch>=2.4,<2.11.0 range, so without --force-reinstall uv keeps the
ROCm build and only swaps the companions -- a mismatched venv the flavor-repair block
won't fix. setup.ps1 scopes the forced reinstall to the ROCm-fallback path
() so the genuine CPU-only install stays fast.
install.sh: the Strix Halo WSL reroute treated every nonzero child exit as a reroute
failure and fell back to CPU. In --tauri mode the child uses exit 2 ([TAURI:NEED_SUDO])
to ask the desktop app to elevate for the target distro; capture the child's exit code
and propagate exit 2 in Tauri mode (the child already printed the NEED_SUDO line)
instead of masking it. CLI mode still falls back to CPU on a generic failure.
Tests: reroute Tauri exit-2 propagation (and non-Tauri CPU-fallback) cases;
run_func now preserves the child exit code; force-reinstall assertions for both
PowerShell installers.
Note: codex's _rr_q apostrophe finding is a false positive -- the helper already
emits POSIX-correct 'O'\''Brien' and round-trips under both sh and bash.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* setup.ps1: fix $cpuForce array collapse in the ROCm->CPU torch fallback
An if-expression assignment ($cpuForce = if ($ROCmCpuFallback) { @("--force-reinstall") })
collapses the single-element array to a scalar string, so @cpuForce splatting enumerated
it character-by-character into broken single-letter args (- - f o r c e ...), which made
uv/pip reject the install and aborted the whole Studio setup on the AMD ROCm->CPU fallback
path. Build $cpuForce as a real array assigned outside the if-expression so the splat passes
a single --force-reinstall arg. Genuine CPU-only installs stay fast (empty array, no flag).
Test now asserts the array-build form and rejects the if-expression form.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* uninstall: remove the isolated Node.js runtime (~/.unsloth/node)
The isolated Node.js runtime (install_node_prebuilt.py, added with the managed-Node
change) installs to ~/.unsloth/node in default mode -- a sibling of studio, so deleting
<studio> leaves it behind (~200MB orphaned after uninstall). Both uninstallers already
remove the other default-mode siblings (llama.cpp/.cache/.staging); add node alongside
them. uninstall.ps1 also adds it to the handle-lock sweep so a held node.exe can't block
the delete. Env/custom mode nests node under the custom root, removed with that root.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Suppress only torchao's cosmetic cpp-extensions warning
torchao logs a cosmetic WARNING on torch < 2.11 from torchao/__init__.py
(logger.warning("Skipping import of cpp extensions due to incompatible
torch version...")). The bnb-4bit and Unsloth paths do not use torchao's
cpp kernels, so the message is noise.
The quiet block raised the whole torchao logger to ERROR, which also hid
any genuine torchao warning, plus a redundant stderr substring filter.
Replace both with a HideLoggingMessage logging filter scoped to the
torchao logger that drops only this one record, leaving every other
torchao log intact.
* torchao cpp-extensions filter: tighten comment (no code change)
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
* loader: build UNSLOTH_MODEL_NAME fresh per load instead of accumulating
UNSLOTH_MODEL_NAME was built by prepending the previous os.environ value. That
value is inherited across processes (a save->reload subprocess) and accumulated
stale load flags. A leftover '_load_in_4bit_' from an earlier bnb-4bit load made
gpt-oss wrongly take the BnB router patch (router.linear.weight) when later
reloading a merged 16bit checkpoint (router.weight), raising 'some weights are
not initialized'. Build the string fresh from this load's model name + flags.
* loader: keep the raw model name/path out of UNSLOTH_MODEL_NAME
Building the sentinel string from lowered_model_name meant a non-4bit load from a
local path that happens to contain a flag sentinel (e.g. .../model_load_in_4bit_x)
copied that substring into UNSLOTH_MODEL_NAME, so downstream gpt-oss patches that
test "_load_in_4bit_" in UNSLOTH_MODEL_NAME would wrongly take the BnB 4bit route
with load_in_4bit=False. Only the model TYPE tokens and the explicit load flags are
consumed downstream, and the raw name was not part of this string before the
fresh-build change, so build it from model_types_all + flags only.
* loader: encode effective bnb state in UNSLOTH_MODEL_NAME (not requested)
A checkpoint already quantized with a non-bitsandbytes method (gpt-oss MXFP4, gptq,
awq, compressed-tensors) has load_in_4bit/8bit disabled later by
check_and_disable_bitsandbytes_loading. Recording the REQUESTED _load_in_4bit_ here
meant the public default load_in_4bit=True on a native MXFP4 gpt-oss (stock
router.weight) routed onto the BnB router patch (router.linear.weight) and failed
with 'some weights are not initialized'. Mirror that normalization from the model's
own config via get_quant_type so the name reflects the effective load. Falls back to
the requested flags on any error.
* gpt-oss: detect non-bnb base quant for adapter-only PEFT repos
When loading an adapter-only PEFT repo, model_config is still None at the
point UNSLOTH_MODEL_NAME is built (the base config loads later), so the
_load_in_4bit_/_load_in_8bit_ tokens were always appended even when the base
is non-bnb quantized (e.g. a LoRA over gpt-oss MXFP4). Resolve the base
checkpoint's config via peft_config.base_model_name_or_path so the bnb load
flags reflect the effective quant method. Wrapped in try/except with the prior
behavior as fallback.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss: sync UNSLOTH_MODEL_NAME bnb tokens from the effective load state
The per-load UNSLOTH_MODEL_NAME is built in FastModel.from_pretrained before the base
is remapped (get_model_name) and before check_and_disable_bitsandbytes_loading runs, so
its _load_in_4bit_/_load_in_8bit_ tokens can be wrong: an adapter-only PEFT repo has
model_config=None there, and a base may be remapped to an Unsloth -bnb-4bit build or be
native MXFP4/GPTQ/AWQ. The gpt-oss patch keys its BnB vs stock router/expert classes off
these tokens, so a stale token mismatches the loaded checkpoint.
Add sync_unsloth_model_name_bnb_flags(load_in_4bit, load_in_8bit) and call it right after
check_and_disable_bitsandbytes_loading in both load paths (llama.py, vision.py), where the
effective bnb state is finally known. It makes the tokens match the effective flags, is
gated to gpt-oss (the only consumer; the token is inert for every other model), and is a
strict no-op otherwise.
This supersedes the earlier pre-remap PEFT-base probe (reverted): probing the base config
in FastModel could not see the get_model_name remap, so it mis-set the token for an adapter
whose base remaps to bnb-4bit. The post-disable sync handles every case uniformly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss: do not stamp a synthetic bnb quantization_config over a non-bnb model
The post-load 'fix up bitsandbytes config' block runs under the requested load_in_4bit,
which stays True for a native MXFP4/GPTQ/AWQ checkpoint even though
check_and_disable_bitsandbytes_loading disabled bnb for the actual load. Stamping a
synthetic bitsandbytes quantization_config there overwrites the real one and would be
saved as a fake bnb config. Guard it: only stamp when the loaded model is bnb or
unquantized (get_quant_type(model.config) in (None, 'bitsandbytes')).
* gpt-oss loader: tighten comments/docstrings (no code change)
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix save crash for legacy list-form _tied_weights_keys (NemotronH)
transformers >= 5 save_pretrained reads module._tied_weights_keys.keys(),
which raises 'list' object has no attribute 'keys' for modules that still
declare the attribute as a list (e.g. NemotronH backbone.layers.N.mixer.*_proj),
crashing GGUF export and merged saves part-way through.
Coerce any legacy list/tuple _tied_weights_keys into the dict form transformers
5.x expects, mapping each key to itself. Only the keys are read (as dedup
patterns) so behaviour is preserved, and older transformers that iterate the
attribute directly see the same keys. The helper is idempotent and best-effort
so a save never fails over it. Called from unsloth_save_model,
unsloth_save_pretrained_gguf and unsloth_generic_save after tokenizer patching.
Adds version-independent unit tests covering list/tuple coercion, dict and
None/empty pass-through, idempotency and odd-object tolerance.
* Coerce empty/set _tied_weights_keys too
transformers only skips _tied_weights_keys when it is None, so an empty list,
tuple or set still reaches .keys() and raises the same AttributeError. Coerce
every non-dict container (including the empty case and sets) to a dict, and add
tests for empty/set inputs.
* Tighten comments in tied-weights save fix
* Scope tied-weights-keys coercion to the save call
Coercing legacy list-form _tied_weights_keys to {k: k} fixed the transformers
5 save crash, but persisted a self-mapping on the live model. transformers 5
re-ties from the dict's values, so a later resize/re-tie would no-op the tie
instead of pointing the output weights back at the input embeddings.
Replace the in-place mutation with a decorator that coerces before the save and
restores the originals afterwards (including on exception), so the save sees the
dict form transformers needs while the model keeps its original tie metadata.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments to be more succinct
---------
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Load repo-code VLMs that register AutoModel in auto_map
FastModel.from_pretrained already falls back from the VLM auto class to
AutoModelForCausalLM for repo-code VL models that register only that class
in their auto_map (e.g. Nemotron-VL). Models like DeepSeek-OCR and
DeepSeek-OCR-2 instead register their architecture under AutoModel, so they
fell through to AutoModelForImageTextToText and raised "Unrecognized
configuration class ... for AutoModelForImageTextToText".
Generalize the guard: when neither vision auto class is registered, fall
back to whichever generic auto class the repo actually registered
(AutoModelForCausalLM, else AutoModel).
* Do not hard-error on a newly initialized position_ids buffer
RaiseUninitialized turns transformers' "some weights of ... were not
initialized" warning into a hard error. position_ids is a deterministic
arange buffer that transformers itself lists in
_keys_to_ignore_on_load_missing, so re-initializing it is correct rather
than a sign of a corrupt checkpoint. Some VLMs (e.g. DeepSeek-OCR) ship it
non-persistently, which tripped the guard. Allowlist position_ids alongside
the existing classifier/predictions head weights.
* Only ignore missing-weight records that are exclusively position_ids
The previous substring check skipped the whole "Some weights of ..." record
whenever position_ids appeared anywhere in it. Transformers reports every
missing key in one record, so a corrupt or incompatible checkpoint missing a
real parameter could load with randomly initialized weights as long as one
missing key contained position_ids. Parse the "newly initialized: [...]" list
and suppress only when every listed key is a position_ids buffer; otherwise
raise as before.
* Match the concrete VLM auto class name when checking auto_map
Transformers resolves remote code by the exact auto class name being called,
and AutoModelForVision2Seq aliases to AutoModelForImageTextToText on
transformers >= 5. Checking for both spellings treated a config that only
registers the legacy AutoModelForVision2Seq key as having a supported VLM
class, skipping the AutoModelForCausalLM fallback that used to load it and
failing as an unrecognized config under AutoModelForImageTextToText. Match
only the concrete class name we would actually pass, keeping the AutoModel
and AutoModelForCausalLM fallbacks.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Preserve VLM mode on the vLLM path when falling back to AutoModel
A repo-code VLM that registers only AutoModel or AutoModelForCausalLM (DeepSeek-OCR, Nemotron-VL) routes to that generic class, so is_vlm, derived from the resolved auto class, is False. That is correct for processor selection (these repos ship no AutoProcessor) but wrong for the vLLM path, where is_vision_model=is_vlm made vLLM treat a vision_config model as text-only and skip the VLM guard and conversion.
Add is_vlm_config, derived from the config vision_config (and gated on not text_only so a text-only resolve still wins), and use it for the fast_inference VLM guard and the is_vision_model flags passed to load_vllm, get_vllm_state_dict and convert_vllm_to_huggingface. Processor selection still uses is_vlm, so DeepSeek-OCR keeps loading via its tokenizer. DeepSeek-OCR with fast_inference now raises the clear 'Fast inference is only supported for ...' error instead of being mishandled as text-only.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The colocate GRPO setup set args.vllm_enable_sleep_mode from the raw
UNSLOTH_VLLM_STANDBY env var (== '1'). When unsloth-zoo's vision standby gate
disables sleep mode for a multimodal run, the engine is built with
enable_sleep_mode=False but TRL was still told sleep mode is on, so it could
drive sleep/wake against an engine that did not enable it.
Read enable_sleep_mode from the colocated engine
(model.vllm_engine.llm_engine.vllm_config.model_config), the same path
check_sleep_mode uses, and fall back to the standby env var (!= '0', matching
load_vllm/patch_vllm) only when the engine cannot be introspected.
Pairs with unslothai/unsloth-zoo#768.
* Shim removed vllm.transformers_utils.tokenizer for older unsloth-zoo
vLLM >= 0.22 (PR vllm-project/vllm#35024) deleted
`vllm.transformers_utils.tokenizer`. Older unsloth-zoo
patch_vllm_lora_tokenizer() does an unguarded
`import vllm.transformers_utils.tokenizer`, crashing fast_inference with
`No module named 'vllm.transformers_utils.tokenizer'`.
Add fix_vllm_lora_tokenizer_module(): a meta path finder appended after
the real finders that provides a no-op stub module only when vLLM no
longer ships it. Registered in _gpu_init.py before vLLM is imported, so
users who upgrade unsloth but keep an older unsloth-zoo are protected.
Refs unslothai/unsloth#6385
* Shorten comments in fix_vllm_lora_tokenizer_module
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Harden Trainer._load_rng_state against malicious checkpoints (CVE-2026-1839)
Resuming from an untrusted checkpoint loads its rng_state.pth via transformers' Trainer._load_rng_state, which calls torch.load without weights_only=True. On torch < 2.6 (weights_only not yet the default, safe_globals() does not restrict the unpickler) a crafted rng_state.pth runs arbitrary code. transformers fixed this in 5.0.0rc3, but we pin 4.57.x for compatibility, so add an import-time monkey patch instead.
patch_unsafe_trainer_rng_load() wraps the method and forces torch.load with weights_only=True for the call, mirroring the upstream fix while keeping the surrounding distributed/device logic intact and tracking transformers' own safe_globals() allowlist. It is idempotent and a no-op on torch >= 2.6 (already the default) and transformers >= 5.0.0rc3 (already fixed), so it self-disables once the pin is bumped. Wired into _gpu_init.py so it applies on import unsloth.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Gate Trainer._load_rng_state via check_torch_load_is_safe instead of forcing weights_only
Review feedback was correct: forcing weights_only=True is not the right fix. It is not a safe boundary below torch 2.6 (CVE-2025-32434), where transformers' safe_globals() is also a nullcontext, so a legitimate rng_state.pth (which holds numpy state) would fail to load there. Call transformers' own check_torch_load_is_safe() before the load instead: it raises on torch < 2.6 and is a no-op on torch >= 2.6, where torch.load already defaults to weights_only=True. This also drops the temporary global torch.load swap, removing the thread-safety race.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments in the RNG-load guard
* Harden only the rng torch.load via a thread-local gate
Address review feedback on the rng-load guard:
- Import Trainer on its own and fall back to a local torch-version check when
transformers does not export check_torch_load_is_safe, so older supported
4.51.x installs are still protected.
- Force weights_only=True at the rng load so TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD
cannot re-enable pickle execution.
- Gate at the actual rng torch.load through a thread-local flag rather than up
front, so model-only or rng-less resumes still proceed on torch < 2.6 and
other torch.load callers are unaffected.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix undefined variable 'e' in Version parsing exception
* Fix SyntaxError for tuple unpacking in array index for Python < 3.11
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Expose MLX grad value clipping in Studio
* update test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* dataset ordering + wd
* fix mlx smoke step expectations
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* cast norm activation output back to original input dtype
* address mlx studio review feedback
* Fix present-but-None seed override for PR #5656
studio/backend/core/training/worker.py
`config.get("model_random_state", random_seed)` only fills the
default when the key is absent. When a caller passes
`config["model_random_state"] = None` explicitly (which happens
any time a JSON payload sends an explicit `null`), the old code
forwarded `None` to FastMLXModel and disabled deterministic init
silently. Same for `lora_random_state`. Treat absent and explicit
None the same way: fall back to random_seed.
studio/backend/tests/test_training_raw_support.py
Update the source-string assertions to match the new lines.
* Guard optional MLXTrainingConfig fields and normalize random_seed for PR #5656
The MLX worker now passes `cast_norm_output_to_input_dtype` and
`dataset_order` only when the linked unsloth-zoo dataclass actually
declares them. Released zoo trees that predate the paired PR can still
construct `MLXTrainingConfig` without raising
`TypeError: unexpected keyword argument`. Once the dependency floor is
bumped to a release that contains both fields, the feature-detect
guards become no-ops.
`random_seed = config.get("random_seed", 3407)` was unguarded against
explicit `None` from raw / backend callers. The same value seeded the
trainer and was the fallback target for `model_random_state` /
`lora_random_state`. Normalize once at the top of the function and use
the normalized value everywhere so an explicit `None` cannot reach
FastMLXModel / get_peft_model / MLXTrainingConfig.
Existing seed source-pattern test updated to match the new normalize
helper. New test asserts the feature-detection guards exist and that
the unconditional kwargs do not include the gated fields.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Normalize seed / cast / max_grad_value at TrainingBackend for PR #5656
Round-3 review consensus: the per-field guards that landed in the MLX
worker only protect the MLX path. The same `TrainingBackend.start_training`
config still reaches the CUDA/text trainer at `worker.py:2267`, the
embedding LoRA init at `worker.py:2450`, and embedding TrainingArguments
at `worker.py:2624` with raw `None` values, so an explicit
`random_seed=None` from a raw / backend caller still breaks non-MLX
training even after the previous fix.
Move the normalization into `TrainingBackend.start_training` itself,
where it runs once for every training mode:
- `_coerce_seed(value)`: explicit `None`, non-int, or absent all become
3407. Every downstream worker now sees an int.
- `_coerce_optional_bool(value, default)`: explicit `None` falls back
to `default` instead of `bool(None) == False`. Also normalizes the
common raw-config / YAML string aliases ("true" / "false" / "0" /
"1"). Used for `cast_norm_output_to_input_dtype`.
- `_coerce_optional_nonneg_float(name, value)`: rejects negative
numerics from raw / backend callers, matching the Pydantic
`ge=0` constraint the HTTP route already enforces. Used for
`max_grad_value`.
worker.py MLX path: the existing `bool(config.get(key, True))` for
`cast_norm_output_to_input_dtype` was changed to also fall back on
explicit `None`, so direct worker callers (bypassing
`TrainingBackend.start_training`) are equally safe. `max_grad_value`
also raises on negative values inside the worker for the same reason.
TrainingStartRequest.random_seed default bumped from 42 to 3407 so
direct REST callers that omit the field receive the same default as
the Studio frontend and the MLX worker.
New regression test exercises the three new helpers across explicit
None, valid values, string aliases, and negative-value rejection.
* Tighten feature-detect test paren tracking for PR #5656
The block-extraction used , which stops at the
first inner closing paren (e.g. )
and would silently miss a future unconditional
/ added later in the same dict literal. Switched to
proper paren-depth tracking so the unconditional block is checked end-to-end.
* Shorten verbose comments in MLX Studio backend
* Handle MLX Studio EOS appending by mode
* Wire MLX leaf norm clipping through Studio
* Respect VLM layer filters for explicit LoRA targets
Rationale / guardrails for the local Studio/vision push:
When callers provide explicit VLM LoRA target_modules together with layer filters, FastVisionModel still needs to route the explicit targets through get_peft_regex. Otherwise the layer filters are ignored and adapters can be attached outside the requested language/vision scope.
Do not revert this to plain list(target_modules) for explicit module lists. The CUDA/Studio-facing contract is that explicit targets and layer filters compose: target_modules selects module names, while finetune_language_layers / finetune_vision_layers / finetune_attention_modules / finetune_mlp_modules constrain where those targets are allowed.
The regression test covers the language-only explicit q_proj case and source-checks that explicit targets are wrapped through get_peft_regex when filters are active.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Refresh MLX smoke clip-config note for leaf_norm default
Trim the 11-line comment block to 5 lines and correct the stale claim
that MLXTrainingConfig defaults to max_grad_value=1.0. The new default
is max_grad_leaf_norm=1.0 (same memory profile as elementwise but
direction-preserving). The smoke still pins max_grad_value=1.0
explicitly to keep the 13-seed pass-rate fixture stable.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Forward max_grad_leaf_norm through the training route and warn when layer filters constrain explicit target_modules for PR #5656
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han-Chen <info@unsloth.ai>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
The generated install command builds unsloth from git with
--no-build-isolation, so pip uses the environment's existing setuptools
rather than the pinned build-system requirement. On setuptools < 77 the
PEP 639 license string in pyproject.toml fails to validate and the
install aborts. Upgrade setuptools and wheel up front so the source
build always has PEP 639 support.
oobabooga