* Restore dropped FP8 weight_scale_inv tensors on load
Some block-scale FP8 checkpoints (for example Qwen3.6-27B-FP8, issue #6200) load
with transformers leaving an mlp.gate_proj as a plain bf16 Linear instead of an
fp8 module. Its raw quantized values are read into the bf16 weight and the
weight_scale_inv is dropped as an unexpected key, so the weight is used un-scaled
and the base model is garbage (perplexity around 2 million).
After load, for every checkpoint weight_scale_inv whose live weight is not fp8,
dequantize the orphaned weight in place using the block scale from the checkpoint
index. Modules that were converted correctly keep an fp8 weight and are skipped,
so healthy checkpoints and single-file checkpoints are a no-op.
Verified on Qwen3.6-27B-FP8: 64 gate_proj scales restored, perplexity 2028902 to
8.9. No-op on Qwen3-8B-FP8 (all scales already live).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden FP8 weight_scale_inv restore from review
- Skip restore when the model has no fp8 weights, so an intentionally
dequantized load (load_in_16bit) is never re-scaled and corrupted.
- Thread revision, subfolder and cache_dir through the index and shard
downloads so scales come from the same snapshot as the weights.
- Cover unsharded single-file model.safetensors checkpoints (no index).
- Handle transposed block-scale layouts and skip on a true grid mismatch
instead of applying a wrong scale.
- Match text-only VLM loads where the language_model prefix was stripped.
- Restore on the FastLanguageModel text path too, not only vision.
- Handle a scalar weight_block_size; per-tensor error handling so one bad
tensor cannot abort the rest or hide a partial mutation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address second review round on FP8 scale restore
- Bound peak memory: dequantize block views in place with the fp32 scale
broadcast instead of materializing a full expanded scale and fp32 copy,
so a near-VRAM-limit load is not pushed into OOM by the repair.
- Restore on the sequence-classification load path too.
- Cover more VLM key remappings (language_model.model.* to
model.language_model.*) when matching modules.
- Skip the restore for variant loads (variant=...) rather than risk
applying default-checkpoint scales to variant weights.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Align FP8 scale restore revision with the loaded weights and warn on disk-offloaded layers
In llama.py the CausalLM/SequenceClassification weight loads resolve model_name on its
default branch (revision is not forwarded there), so read the dropped weight_scale_inv
tensors from the same default branch instead of the requested revision, avoiding rescaling
default-branch weights with scales from another revision.
In loader_utils.py a disk-offloaded layer keeps its weight on the meta device until the
offload hook materializes it, so the scale cannot be applied in place. Skip such layers
explicitly and print a warning rather than silently leaving them unscaled.
* Tighten comments in the FP8 scale restore path
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
parse_direct_linux_release_bundle and direct_linux_release_plan are no
longer reached by any live code path. Fork Linux installs resolve through
_fork_manifest_release_plans -> _linux_published_attempts, and the upstream
(ggml-org) path uses direct_upstream_release_plan. The dead parser also
called _resolve_linux_bundle_profile, which no longer exists, so its CUDA
branch would raise NameError if ever executed.
Drop both functions and the obsolete TestDirectLinuxNvidiaCpuGate; its live
equivalent TestLinuxPublishedAttemptsNvidiaCpuGate already covers the
NVIDIA no-silent-CPU behaviour.
* Keep native RoPE scaling when extending context; carry rope_theta for linear
When max_seq_length exceeds a model's native window, the loader overwrote the
model's rope_scaling with linear scaling. For models that already ship a scaled
RoPE (llama3/yarn/longrope) that is far worse for long context, and on
transformers v5 the linear dict omitted rope_theta (v5 keeps it under
rope_parameters), so the rotary base fell back to 10000 and broke past ~8K tokens.
Keep the native scaling and just widen the window; only synthesize linear for
plain-RoPE models, and carry rope_theta so v5 keeps the real base.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Only preserve native llama3 when extending context; keep linear fallback otherwise
The patched attention constructor (patch_llama_rope_scaling) rebuilds only linear,
llama3 and longrope and its longrope branch reads a top-level
original_max_position_embeddings, so preserving yarn or a nested-only longrope config
would raise during construction on transformers <= 4.47.1. Keep only llama3 native;
yarn/longrope/other types fall back to the linear override, still carrying rope_theta.
* Correct long-context extension comment to match llama3-only preservation
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix fast_gemv crash on compressed-tensors FP8 models
Loading a compressed-tensors FP8 checkpoint (for example
unsloth/Llama-3.2-1B-Instruct-FP8-Block) with fast_inference=False and
running a forward crashed with 'Parameter object has no attribute absmax'
inside fast_gemv.
A compressed-tensors CompressedLinear exposes an already dequantized bf16
weight at forward time while keeping a weight_scale Parameter. The quant
state resolution in get_lora_parameters/get_lora_parameters_bias fell back
to that weight_scale, so a bf16 weight was routed into the bitsandbytes
fast_gemv/fast_dequantize path, which expects a bitsandbytes QuantState
with an absmax attribute.
Only fall back to weight_scale_inv/weight_scale when the weight is still
fp8. A decompressed bf16 weight then resolves to no quant state and flows
through the normal bf16 path, which already handles bias and the LoRA
backward. Real fp8 and bitsandbytes 4bit weights are unchanged.
* Skip the fast_gemv dispatch test before importing unsloth when bitsandbytes is absent
* Fix Windows installer torch index override
* Clear inherited uv index env vars for pinned installs in studio/setup.ps1 (#6898)
* Harden setup.ps1 index-var clearing to truly remove vars (#6898)
* Apply UV_DEFAULT_INDEX torch index fix to Linux/Mac install.sh (#6898)
* Neutralize all uv index env vars for pinned torch installs (#6898)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix FastSentenceTransformer Qwen embedding preprocessing
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Document Transformer.load embedding modality fix for #6881
* Harden #6881 fix and add forwards/backwards-compatible regression tests
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fall back to Transformer constructor on legacy sentence-transformers without Hub-capable load
* Mirror legacy sentence-transformers fallback in embedding-parity tripwire test
* Tighten #6881 comments and docstrings
* Skip embedding-parity test on CPU-only runners since FastSentenceTransformer requires CUDA
* Honor the transformer module's saved subfolder when loading
modules.json records a path for the Transformer module (root for
decoder embedders like Qwen3-Embedding, 0_Transformer for the classic
layout). Pooling/Normalize already load from their saved path; thread the
same path into Transformer.load as subfolder so config and tokenizer
resolve like stock ST. stays a no-op, so single-module models are
unchanged.
* Make embedding-parity test bf16-aware
fp16 overflows to NaN on bf16-native embedders such as EmbeddingGemma
(Gemma3), producing a false parity failure. Prefer bf16 when the GPU
supports it so the tripwire can guard the full documented embedding
matrix (Qwen3-Embedding, EmbeddingGemma, BGE-M3, all-MiniLM, GTE-ModernBERT),
not just fp16-safe models.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Retry the Studio UI shutdown re-login on transient goto timeout
The Chat UI Playwright smoke intermittently failed at the pre-shutdown
re-login: page.goto('/login') can hit a 60s TimeoutError on a slow runner
even while the server is healthy, and the surrounding except only tolerated
ERR_ABORTED / interrupted-navigation, so a plain timeout hard-failed the job.
Wrap the re-login goto/wait/fill/submit in the same 3-attempt retry the
change-password step already uses (recover_or_replace_page between tries,
per-attempt fail screenshots, wait_for_health pre-gate). The composer wait
stays outside the loop so a retry never re-navigates after login has set
tokens (which would redirect to /chat via the guest guard); it remains the
authoritative confirmation, so a genuinely broken login still fails.
* Catch transient login-request failures and preserve error listeners on recovery
Wait on the /api/auth/login POST inside the retry (via click_and_wait_for_response)
so a transient 4xx/5xx is retried in-loop instead of surfacing only at the
out-of-loop composer wait, matching the change-password step. When
recover_or_replace_page swaps in a fresh page, re-attach the pageerror/console
listeners so error tracking survives the replacement.
* Stabilize floating monitor drag
* Restore floating monitor exit animation
* Harden Windows Studio smoke checks
* Keep API menu badge removed
* Apply no-build-tools env overrides in-script
The runner does not apply step-level env keys containing parentheses,
so ProgramFiles(x86) kept its real value and Find-VsBuildTools still
detected VS through vswhere. Set the overrides inside each pwsh step
instead; child processes inherit them. The resolver step moves to pwsh
because bash cannot export a variable named ProgramFiles(x86).
* Reset chat UI session without a second browser context
macOS runs Chromium with --single-process, where closing the last
context tears down the whole browser, so the shutdown re-login died
with TargetClosedError on new_page. Clear cookies and swap pages
inside the same context instead, opening the replacement page before
closing the old one.
* Keep the no-build-tools Path filtered across session refreshes
install.ps1's Refresh-SessionPath and setup.ps1's Refresh-Environment
rebuild the session Path from the Machine and User registry scopes, so
the process-level filter could be undone mid-install and re-expose
CMake. Filter those scopes in the Prepare step with normalized dir
matching and restore them in cleanup.
* Drop stale localStorage auth tokens before re-login
Auth tokens live in localStorage, not cookies, and the login guest
guard redirects on their mere presence. Remove them during the session
reset so the /login navigation is deterministic instead of relying on
the tolerated redirect bounce.
* fix: Remove moot has_blackwell_gpu() function
Fixes unslothai/unsloth#6961. This function skipped flash-attn on Blackwell GPUs because no prebuilt wheel existed;
Dao-AILab now ships one and url_exists() already gates resolution.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: use torchao 0.17.0 for Blackwell
Fixes#6961. Torchao 0.16.0's cpp extensions are built against CUDA 12, so on a CUDA-13
torch (cu130 / Blackwell) they fail to load with "libcudart.so.12: cannot
open shared object file". Select 0.17.0 there instead: its cpp targets torch
2.11, so it is skipped cleanly rather than crashing. CUDA-12 / ROCm / CPU
torch 2.10 keeps 0.16.0 and its working kernels.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Condense torchao version-selection comments (no behavior change)
* Support torch 2.11 in the Studio installer via the torch2.10 prebuilt wheels
Map torch 2.11 to the torch2.10 prebuilt wheels for flash-attn, causal-conv1d,
and mamba through wheel_utils.prebuilt_wheel_torch_mm, applied in direct_wheel_url
(filename) and flash_attn_wheel_url (version). Those torch2.10 CUDA wheels load and
pass each project's own test suite on torch 2.11 (verified on B200), so a torch 2.11
environment gets the prebuilt accelerators instead of skipping or building from source.
Raise _CUDA_TORCH_PKG_SPEC to <2.12.0 (torchvision <0.27.0, torchaudio <2.12.0) so
the CUDA torch repair path can install torch 2.11, where torchao 0.17's cpp kernels
load cleanly. Add tests for the mapping.
* Keep has_blackwell_gpu as a False stub for future arch gating
* Restore has_blackwell_gpu as a return-False probe kept for future arch gating
Keep the nvidia-smi compute_cap detection and its two call sites, but short-circuit
with return False at the top so flash-attn is no longer skipped on Blackwell (sm_100+
now has prebuilt wheels and url_exists gates resolution). Drop the early return to
re-enable arch-based detection later.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: source CPU llama.cpp prebuilts from the unslothai fork
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: reject unknown Linux CPU arches and keep ROCm-tooling hosts off the CPU prebuilt
* Studio: extend the resolve-prebuilt ROCm-tooling guard to Windows
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: let ROCm-SDK-only CPU hosts take the fork CPU prebuilt
* Studio: accept windows-arm64 prebuilt kind and refresh stale fork-routing comments
* Studio: correct stale fork-routing comments and --resolve-prebuilt help
* Refresh stale ggml-org routing comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* feat: detect installed coding agent CLIs in Studio settings
The API-keys panel only ever showed the "claude" flavor of the
`unsloth start` command, so anyone using Codex, OpenCode, OpenClaw,
Hermes, or Pi had to manually rewrite the copied command by hand.
Add a backend check that looks for each agent's CLI binary on PATH
(shutil.which, mirroring the pattern already used elsewhere in
studio/backend/utils) and expose it as GET /api/settings/coding-agents.
The API-keys panel now renders a picker for all six supported agents,
marks the ones it finds installed, and defaults to one of those instead
of always falling back to claude.
Includes unit tests for the detection helper.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* address review feedback on coding-agent detection
Three fixes from PR review:
- detect_installed_coding_agents now treats a PATH lookup failure as
"not installed" instead of letting it bubble up and break the
settings endpoint; added a regression test for it.
- CodingAgentsResponse.agents is now typed as an immutable tuple
instead of a list built from one, matching CODING_AGENTS itself.
- Fixed a race in the API-keys panel: picking an agent while the
installed-CLI check is still in flight could get silently overwritten
once that check resolved. A ref now tracks whether the user has made
a manual choice, so the auto-detected default only applies before
that happens.
* Address Codex feedback: GGUF gating and remote-detection scope
- codex refuses to launch against a non-GGUF (transformers-backed) model
(unsloth_cli's _require_gguf_for_codex), so auto-defaulting to it produced
a copy-pasteable command that fails immediately whenever the loaded model
isn't GGUF. Add useActiveModelIsGguf() (looks up the active checkpoint in
the chat runtime store) and a correction effect that steers the auto-pick
away from codex unless the loaded model qualifies, without ever touching a
choice the user made by hand.
- Detection runs via shutil.which on the Studio backend host, which isn't
the same machine as the browser in a tunnel/remote session. Reword the
'installed'/'detected' copy to say so explicitly when the tunnel URL is
in use, instead of implying the check ran on the viewer's own device.
* Rework auto-default per review: loopback gating + inline GGUF check
Replaces the previous approach with the exact shape discussed on the PR:
- Export isLoopbackHost/normalizeHost from agent-command.ts. The detection
endpoint runs shutil.which on the Studio backend, which only describes the
browser's own machine when the base this panel targets resolves to
loopback. For a LAN or tunnel/remote base, gate the whole thing off --
don't mark anything as "detected" and don't let it drive the default --
instead of just relabeling the copy.
- Drop the separate GGUF-correction effect and useActiveModelIsGguf hook.
Read useChatRuntimeStore.getState().activeGgufVariant inline inside the
existing detection effect's .then() (so it doesn't need to sit in the
effect's deps), and pick the first detected agent that isn't codex unless
the loaded model is GGUF, leaving the existing default untouched when no
compatible agent is detected.
Verified both branches (loopback vs LAN/tunnel base, gguf vs non-gguf,
manual pick preserved, no-compatible-agent fallback) with a standalone
port of the .then() logic.
* Address latest Codex findings: stale detection, model swap, cache
- Clear detectedAgents (and skip the network call entirely) when the panel
leaves a loopback base, instead of leaving a previous loopback detection
result marked 'installed' for a command that now targets a LAN/tunnel/
remote host.
- Add a separate, network-free correction effect keyed on the live
activeGgufVariant: if codex was auto-picked while a GGUF model was loaded
and the user then switches to a transformers-backed model while this panel
stays mounted, steer away from codex instead of leaving a command that
unsloth_cli's _require_gguf_for_codex will now reject. Never touches a
manual pick.
- Drop coding-agents.ts's module-lifetime cache. Installed-CLI detection is
environment state, not a persisted setting, so a stale positive/negative
from before the user installed something (or reopened the tab) is worse
than one extra cheap local API call per mount; keep only the in-flight
de-dupe for concurrent callers.
Verified the correction-effect logic (gguf->non-gguf swap with/without a
fallback, still-gguf no-op, manual pick never overridden) with a standalone
port of the effect.
* Make the codex/GGUF auto-pick symmetric in both directions
The correction effect only steered away from codex when the model stopped
being GGUF; it never steered back toward codex if the model became GGUF
*after* a non-GGUF-gated fallback had already picked something else (e.g.
codex is the only detected CLI, a transformers model is loaded so the
selection correctly falls back to the claude default, then the user loads a
GGUF model while the panel stays mounted -- codex never gets reconsidered).
Consolidate into one effect that re-derives the preferred detected agent
from scratch whenever detectedAgents or activeGgufVariant changes, in either
direction, instead of only reacting to the codex-specific downgrade case.
The fetch effect now only populates detectedAgents/availableAgents; this
effect is the single source of truth for what gets auto-picked from that
list. Never overrides a manual choice.
Verified both transition directions plus the manual-pick-survives and
initial-detection cases with a standalone port of the derivation logic.
* Reset the auto-pick to the default when it stops being trustworthy
Two more real gaps from the latest Codex pass on d988f52:
- The unified derivation effect only handled the case where a *different*
detected agent could take over. If codex was the only detected agent and
auto-picked while a GGUF model was loaded, then the model stopped being
GGUF, 'preferred' came back undefined and the effect silently left the
selection on codex -- exactly the command unsloth_cli's
_require_gguf_for_codex now rejects. Fall back to DEFAULT_AGENT in that
case instead of leaving it untouched.
- Leaving a loopback base cleared detectedAgents (so the 'installed' badges
correctly disappear) but left whatever agent had been auto-picked from
that now-stale, server-side-only detection still selected. Reset to
DEFAULT_AGENT there too, unless the user picked by hand.
Introduces a shared DEFAULT_AGENT constant instead of repeating the "claude"
literal at each reset site. Verified all five cases (both new resets, both
manual-pick-survives variants, and the existing multi-detected-agent
fallback still preferring another compatible agent over resetting) with a
standalone port of the effects.
* Derive GGUF-ness from the actual loaded state, not just the variant string
activeGgufVariant only covers an HF-repo GGUF pick (a specific quant
variant string). A direct local .gguf file -- custom folder, LM
Studio, or drag-drop -- is just as much a GGUF the codex preflight
(unsloth_cli's _require_gguf_for_codex) would accept, but it never has
a "variant" to report, so it read as non-GGUF here even though
/api/inference/status correctly reports is_gguf: true for it. That
mismatch could leave a Codex-only install not auto-selected, or reset
an auto-picked Codex, for a model that actually supports it.
Combined activeGgufVariant with activeNativePathToken (covers the
drag-drop/picked-file case) and ggufContextLength (only ever populated
when the backend last reported is_gguf: true for the active model, see
applyActiveModelStatusToStore) so all three paths a model can be GGUF
through are covered, matching the same is_gguf-or-equivalent check
hasGgufSource already applies to a staged pick elsewhere in this
codebase.
* Clear stale native-path token on a non-GGUF status refresh
When a native (drag-dropped or picked) GGUF was loaded and the backend later
switches to a transformers model outside the UI load path, refresh() adopts the
new /api/inference/status via setCheckpoint and applyActiveModelStatusToStore.
Those reset activeGgufVariant and ggufContextLength but never clear
activeNativePathToken, so the isGguf OR stays true after the switch and a
Codex-only detection auto-selects unsloth start codex for a non-GGUF model its
preflight rejects.
Drop activeNativePathToken in applyActiveModelStatusToStore whenever the status
is non-GGUF. A real GGUF load reports is_gguf: true, so its token is preserved
(the load path owns it); only a non-GGUF status clears it.
* Add the AGPL-3.0 header to the new studio contract test
* Fix/adjust agent detection for PR #6909
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
Co-authored-by: wasimysaid <112766706+wasimysaid@users.noreply.github.com>
* version-compat CI: fake CPU training runs for SFT/GRPO/DPO
Adds a runtime layer on top of the patch-run canary: actually runs
trainer.train() for a couple of steps on a CPU-only runner under the CUDA
spoof, wrapping a plain tiny HF model in the Unsloth-patched trainer. Exercises
the real train() loop (collation, generation, the injected
_get_per_token_logps_and_entropies, loss, backward, optimizer) so a TRL or
transformers change that breaks the loop at runtime -- not just the source
structure -- surfaces here. No GPU, no meaningful numerics.
Needs a chain of small CPU shims (eager torch.compile, dynamo suppress, cuda
tensor-alloc redirect to CPU, model.for_training/for_inference equivalents)
documented inline. Does not exercise Unsloth's Triton/GPU kernels (CPU can't).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* cpu fake-train: force adamw_torch + disable dynamo for CPU runner
On a real CPU-build torch runner (GitHub CI) two things bit that a CUDA-build
torch with GPUs hidden masked locally:
- The default optimizer is adamw_8bit (bitsandbytes), whose is_on_gpu() check
dies on CPU tensors. Force optim=adamw_torch in all three configs.
- import unsloth reinstalls the real torch.compile over the eager passthrough,
so the GRPO hot path (chunked_selective_log_softmax) actually compiles and
inductor picks the spoofed CUDA device, crashing on device props
(gcnArchName). Re-apply the eager passthrough after import and flip
torch._dynamo.config.disable so every @torch.compile runs eager at call time.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* cpu fake-train: write checkpoints under pytest tmp_path
Use pytest's tmp_path for each trainer's output_dir instead of a hardcoded
relative temp/ci_* path, so a local pytest run does not leave untracked dirs in
the repo tree and the tests are CWD-independent.
* version-compat CI: disable dynamo at process level for the fake-run job
Set TORCHDYNAMO_DISABLE / TORCH_COMPILE_DISABLE in the fake-run step env so
dynamo/inductor is off before conftest.py's early import unsloth, not only via
the per-test runtime shim. Defense in depth on the GPU-less runner: the GRPO
hot path never compiles regardless of when its functions were decorated.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix PEFT replacement for TRL >= 1.7.0, add missing compute_aux_loss for TRL 1.7.0
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix GRPO for TRL >= 1.7.0: PEFT ref-adapter removal and return arity
rl.py: for trl >= 1.7.0, scope the PEFT removal regex to the ref-adapter
block only by anchoring the end on ref_param.data.copy_(param.data), so it
no longer also deletes the following gradient-checkpointing
enable_input_require_grads() block. Neutralize TRL 1.7.0's
`if _is_quantized_model:` bf16 cast the same way the existing
is_loaded_in_4bit cast is handled.
rl_replacements.py: initialize _extra_moe_kwargs before use (it was
referenced before assignment whenever compute_aux_loss was passed) and only
request output_router_logits when the aux loss is actually wanted.
rl_replacements.py: _get_per_token_logps_and_entropies now returns a 3-tuple
(logps, entropies, aux_loss) for trl >= 1.7.0 and a 2-tuple for older TRL,
matching how every TRL call site unpacks the result. Without this, TRL 1.7.x
_generate_and_score_completions unpacks 3 values from a 2-tuple and raises
"not enough values to unpack (expected 3, got 2)".
* Return zero aux_loss placeholder and drop inference-mode aux collection
* GRPO TRL >= 1.7.0: reject router aux-loss opt-in at init; drop zero aux placeholder
Unsloth's optimized GRPO forward cannot compute the MoE router auxiliary loss.
Previously an explicit opt-in (router_aux_loss_coef > 0) returned a fabricated
zero, silently training without the requested load-balancing penalty. Now reject
it at trainer init with a clear NotImplementedError, and return None (not zero)
for the aux slot of TRL's 3-tuple. Default stays off (coef 0), so the common
path is unaffected.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO hidden-states fallback: free ModelOutput before chunked log-softmax
The old/ref logprob fallback binds the full ModelOutput (which holds every
layer's hidden_states when output_hidden_states=True) and kept it alive across
chunked_hidden_states_selective_log_softmax, an avoidable OOM on large models.
Extract logits then del outputs in both the text and VLM branches.
* Version-compat CI: proactively catch TRL GRPO breakage
The existing TRL canary is a static symbol/source grep: it verifies symbols
exist but is blind to structural changes (TRL 1.7.0's 2->3-tuple per-token-logps
return arity and restructured PEFT ref-adapter block, which the fix in this PR
addresses, both slipped past it because the methods still existed).
Two additions:
- test_trl_grpo_pinned_symbols.py: extend TRL_TAGS to 1.5/1.6/1.7 and pin the
exact source-string contracts the rl.py / rl_replacements.py transforms depend
on for TRL >= 1.7.0 (PEFT elif ref-adapter block + enable_input_require_grads
survival, if _is_quantized_model, aux_loss_enabled anchor, compute_aux_loss
arity). A future TRL change fails on main a few days before the PyPI release.
- test_trl_grpo_fake_run.py + a version-compat-ci job: fake-CUDA run that drives
the real GRPO/SFT/DPO source-transform patchers against latest + main TRL on a
CPU-only runner (no training) and asserts the generated Unsloth trainer still
satisfies the transform contracts. Catches behavioral regressions the grep
cannot see.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fake-run test: use a normal Version import for the aux gate
* version-compat CI: fix fake-run job gate + torch-absent collection
- Drop the invalid job-level matrix if (matrix is not available in
jobs.<id>.if -> 'Unrecognized named-value: matrix' fails the whole
workflow). Use a single job that runs vs TRL latest always and re-runs
vs TRL main only on schedule/dispatch via a step-level github.event_name
guard. Validated with actionlint.
- Module-level skip the fake-run test when torch is absent so
daily-fresh-fetch (pytest-only, collects tests/version_compat/) does not
crash on the top-level spoof import.
* fake-run test: do not skip on import failure
unsloth/trl are installed in the grpo-fake-run job, so a failing import is the
import-time drift this canary must catch. Keep only the not-installed find_spec
skips; let a real import error fail the test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO arity gate: regex downgrade + fail loud + CI coverage
The TRL < 1.7.0 per-token-logps return downgrade was an exact-string replace
anchored on the full return line incl. its comment, so a reformat (e.g.
pre-commit) could silently no-op it and ship a 3-tuple to older TRL. Switch to
a regex tolerant of comment/whitespace drift, and raise if the anchor stops
matching (re.subn count != 1) instead of failing silently. Add a monkeypatched
trl_version unit test asserting both arities, since CI only installs TRL >= 1.7.0
and never exercised the downgrade otherwise.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fake-run: give SFT/DPO a real contract, not just ast-parse
The SFT/DPO fake patch runs only checked the generated trainer parses. Also
assert the shared QLoRA _is_quantized_model bf16 cast is neutralized (TRL 1.7's
spelling, present in both sft_trainer and dpo_trainer), so a structural TRL
change to that block is caught for SFT/DPO too, not just GRPO.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO PEFT ref-adapter removal: lower gate to the TRL 1.4.0 floor
The elif is_peft_model(model) and args.beta != 0.0: ref-adapter block was
introduced in TRL 1.4.0 and is unchanged through 1.7.x, but the removal was
gated at >= 1.7.0, so for 1.4 <= TRL < 1.7 the transform fell through to the
0.27 branch (which matches the older if is_peft_available()... form) and
silently no-oped: a PEFT + beta != 0 GRPO run then computed the KL reference
from the copied ref adapter instead of the base model. Lower the gate to
1.4.0 and keep the 1.7.0-only router aux-loss fail-fast nested. Widen the
pinned-symbol contract test to run from 1.4.0 so the covered versions are
actually exercised.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Move New badge to System settings tab
Show the "New" badge on the System tab and drop it from Connections.
* Stabilize refresh revocation UI test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Resolve the app-sidebar.tsx conflict: main refactored the chat-export dropdown to a
format-based CHAT_EXPORT_OPTIONS + dynamic-import exportConversationByFormat dispatcher,
which the merged body already uses. Keep main's dispatcher and drop the branch's static
export imports; keep TestTubeOutlineIcon imported from the shared @/lib/hugeicons-derived
module (also used by images-page) rather than main's duplicate inline definition. The
branch's Images and Video nav items are preserved.
* Studio: add assistant response details panel
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Hide model badge by default, show on hover/focus
Wrap MessageResponseModelBadge in a span with hidden/group-hover visibility classes to reduce visual clutter. The badge now only displays when hovering or focusing on the assistant message, improving the UI presentation. Updated corresponding tests to verify the new CSS classes.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add RDNA 2/3/4 ROCm routing tests via a CPU-only torch spoof
Introduces tests/_zoo_rocm_spoof.py, the ROCm sibling of _zoo_aggressive_cuda_spoof.py: it reuses the CUDA spoof's torch.cuda no-op machinery and overlays an AMD Radeon identity (torch.version.hip, gcnArchName, capability) for any RDNA 2/3/4 gfx target, so hip code paths run on CPU-only CI with no AMD hardware.
tests/studio/install/test_rocm_rdna_routing.py then asserts unsloth_zoo routes every RDNA arch (gfx1030/1031/1032/1034, gfx1100/1101/1102, gfx1150/1151, gfx1200/1201) correctly: device_type resolves to hip, llama.cpp target resolves to (rocm, gfx), and the per-family ROCm bundle suffix (gfx103X/gfx110X/gfx120X, or self for gfx1150/1151) is picked. The torch-facing checks run in a subprocess so the spoof never leaks into sibling tests and DEVICE_TYPE (cached at import) resolves from a clean process; the pure gfx-family mapping runs in-process. Guarded by importorskip so it runs where torch and unsloth_zoo are installed (the Repo tests CPU job) and skips elsewhere.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Guard RoPE scaling against the transformers v5 buffer blank; honor extended factor
Add a family-agnostic guard that builds each rotary from a scaled config,
blanks its non-persistent buffers (what transformers v5 does on load), runs
loader._fix_rope_inv_freq, and asserts every buffer is restored to its scaled
value (llama3 and longrope). This catches the whole bug class, not just the
one call site, and is validated to fail on the pre-fix repair.
Also make LlamaExtendedRotaryEmbedding read the llama3 factor from the config
instead of hardcoding 8 (wrong for Llama-3.2, factor 32), falling back to the
Llama-3.1 defaults when built without a config.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pass config into extended rotary codegen; skip v5 round-trip on transformers 4.x
- patch_llama_rope_scaling now builds the llama3 extended rotary with
config=self.config so it reads the real factor (32 for Llama-3.2) instead
of falling back to 8; the template already references self.config.
- test_v5_blank_repair_roundtrip now skips when loader._NEEDS_ROPE_FIX is
False, since _fix_rope_inv_freq is a no-op on transformers 4.x and cannot
restore the blanked buffers there.
* Raise stream deadlock-guard timeouts from 0.2s to 5.0s in passthrough tests
These asyncio.wait_for guards bound test setup and cross-task event
signaling that complete near-instantly on success; the 0.2s budget is a
latency assertion in disguise and times out under CI scheduling load
(seen on the 3.11 matrix leg while 3.10/3.12/3.13 pass the same commit).
5.0s matches the timeout used elsewhere in the suite and still fails fast
on a real hang. No test relies on the guard expiring.
* Extended rotary reads rope_parameters as well as rope_scaling
transformers v5 stores llama3 scaling under config.rope_parameters and
exposes rope_scaling only as a back-compat property. Reading that property
works on 5.0-5.13 (verified: factor resolves to 32 for Llama-3.2), but a
future release may drop the shim, after which the subclass path would fall
back to factor 8. Read either field so the factor survives the rename.
Adds test_extended_rotary_reads_rope_parameters_v5 (fails on the old
single-field read: rope_parameters-only config resolves to 8, not 32).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The custom-root sd.cpp removal requires the .unsloth-studio-owned marker (written by
install_sd_cpp_prebuilt) before deleting, so a user's own stable-diffusion.cpp checkout
beside a custom root is kept. The default-mode removal of ~/.unsloth/stable-diffusion.cpp
was unconditional, so a user who keeps their own checkout at that path (or points
UNSLOTH_SD_CPP_PATH there), or a pre-marker Studio build, would have it deleted on
uninstall. Guard the default-mode removal on the same owner marker in both uninstall.sh
and uninstall.ps1. Extends the sd.cpp uninstall shell test with owned (removed) and
unowned (kept) default-mode cases.
* WSL ROCm: generalize ROCm-on-WSL bootstrap from Strix-only to any RDNA arch
install_rocm_wsl_strixhalo.sh hardcoded gfx1151, so its verify step died on
discrete Radeon cards even though the ROCm + librocdxg setup is arch-agnostic.
Auto-detect the GPU arch from rocminfo (override via UNSLOTH_WSL_GFX), verify any
GPU agent enumerates over DXG, and map the arch to AMD's per-arch wheel family for
the optional smoke test (injecting librocdxg into torch/lib so torch's bundled
ROCr finds the DXG bridge). Verified on gfx1200 (Radeon RX 9060 XT) in WSL2 +
Ubuntu 24.04 -- torch.cuda now enumerates the GPU.
* WSL ROCm: trigger the ROCm-on-WSL bootstrap for discrete Radeon GPUs too
_maybe_bootstrap_rocm_wsl only fired for Strix APUs (matched via /proc/cpuinfo,
which discrete cards don't appear in). Add _wsl_amd_gpu_name() -- queries the
Windows host via WMI -- and broaden the trigger gate plus the 'already-usable
ROCm' rocminfo check from gfx1151-only to any real GPU agent (gfxNNNN, excluding
the gfx11-generic fallback ISA). The generalized bootstrap then auto-detects the
arch. Enables 'curl install.sh | sh' to set up ROCm-on-WSL on discrete Radeon RX
7000/9000 in WSL2 + Ubuntu 24.04, not just Strix Halo/Point.
* WSL ROCm: address review -- filter generic ISA in bootstrap, bound the host GPU query
- install_rocm_wsl_strixhalo.sh: exclude the gfx11-generic fallback ISA in arch
detection (grep -v generic), matching install.sh's rocminfo check, so a generic
agent listed before the real one can't be picked as the arch.
- install.sh: wrap the powershell.exe Win32_VideoController query in _run_bounded
(10s timeout) so an unstable WSL-interop / busy host can't hang the installer.
* WSL ROCm: harden arch-detect + librocdxg copy under set -eo pipefail (review)
- _detected_gfx: append '|| true' so a no-GPU rocminfo (empty pipeline, non-zero
under pipefail) doesn't abort the assignment before the '[ -z ]' branch prints
the diagnostic + die message.
- smoke-test librocdxg copy: gate on '[ -d "$_tlib" ]' instead of '[ -n ]' so a
non-directory value can't make cp rename librocdxg to 'lib'.
* WSL ROCm: address Codex review (gfx000, 24.04 reroute for discrete, test locator)
- Exclude gfx000 (the CPU agent) from the WSL 'usable ROCm' check and the bootstrap
arch-detect: match gfx[1-9] (nonzero arch), so a partial ROCm install that only
reports the CPU ISA no longer short-circuits the librocdxg setup. (P2)
- Reuse the Ubuntu-24.04 reroute for discrete Radeon: broaden
_maybe_reroute_strixhalo_to_2404's gate with the same _wsl_amd_gpu_name (WMI)
fallback, so a discrete card on 26.04 reroutes to a 24.04 distro like Strix does
instead of falling to CPU. Moved _wsl_amd_gpu_name above the reroute and made it
self-contained + 10s-bounded (it runs before _run_bounded is defined). (P2)
- Update TestInstallShDropinPersistence to locate the gate by its unique
'!/generic/' clause now that the gfx1151 literal is gone. (P1)
* Condense ROCm-on-WSL comments in install.sh and bootstrap helper
* Guard WSL reroute from NVIDIA hybrid hosts and fix GFX-override pipefail check
* Honor CUDA_VISIBLE_DEVICES-hidden NVIDIA in the WSL reroute guard
* Reuse _has_usable_nvidia_gpu in the WSL reroute guard
---------
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Fix repeated base model downloads across checkpoint exports (#6890)
Pre-warm the HF hub cache with the 16bit base weights before
merge_and_overwrite_lora runs. The merge fetches shards with
hf_hub_download(local_dir=...), which never populates the hub cache, so
temporary merge directories (GGUF checkpoint exports) forced a full
re-download of the base model for every checkpoint. The first export now
downloads once into the cache and later exports copy from it.
Skips itself when already cached, offline, on Kaggle/Colab, for local or
nf4/fp4 bases, non-downloading save methods, or low disk. Opt out with
UNSLOTH_PREWARM_HUB_CACHE=0.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Show MB for small base models in the pre-warm download message
* Harden pre-warm: getattr for model config, abspath for relative HF_HUB_CACHE
- Read config._name_or_path via getattr so a model without a config skips
cleanly instead of taking the outer error path.
- abspath the cache probe so a relative HF_HUB_CACHE walks up to a real root
rather than "", which would zero the free-space check and skip pre-warm.
Both from PR review; each covered by a test that fails without the fix.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pre-warm the live-env HF cache so runtime redirects still hit (#6890)
Resolve the hub cache the same way the merge does (unsloth_zoo _active_caches,
live env) instead of huggingface_hub's import-time-frozen constants.HF_HUB_CACHE,
and pass it as cache_dir to the cached probe, disk check and snapshot_download.
Without this, a runtime HF_HOME/HF_HUB_CACHE redirect (unsloth_zoo
redirect_hf_cache_if_readonly on a read-only default cache, or Studio) makes the
pre-warm populate a different directory than the one the merge reads, so the
cache-copy fast path misses and the base re-downloads on every export anyway.
Adds 3 regression tests covering the cache_dir threading and the redirect case.
* Apply ruff-format kwarg spacing to the pre-warm cache-dir changes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pre-warm the 16bit sibling for FP8 bases so their merged_16bit exports reuse the cache too
For a merged_16bit export of an FP8 base with an existing 16bit sibling, the merge
swaps to the sibling and downloads that (unsloth_zoo _resolve_fp8_16bit_sibling), so
pre-warming the FP8 repo missed the cache and re-downloaded the sibling every export.
Mirror the swap and pre-warm the sibling. No sibling still caches the FP8 repo for the
in-place dequant path. Adds 2 regression tests.
* Filter pre-warm shards through the safetensors index like the merge does
Repos that ship a leftover shard set the index does not reference (e.g. granite-3.2)
made the disk gate over-count and snapshot_download fetch shards the merge never reads.
Mirror the merge: on the download path, keep only index-referenced shards. Runs after
the already-cached check so the cached fast path stays network-free. Adds 2 tests.
* Tighten pre-warm comments
---------
Co-authored-by: Unsloth <michaelhan@Michaels-MacBook-Pro.local>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
- _local_model_task now tags a local diffusers pipeline that resolves to a video
family (LTX / Wan / Hunyuan) as text-to-video, mirroring the cached-repo
_cached_repo_task, so supported local video pipelines surface in the Video
On-Device picker instead of being routed to the Images picker where the image
loader rejects them. Gated on _local_is_diffusers so only a real loadable
pipeline dir reaches the video check.
- Video validate_load_request now rejects a local pipeline pick whose directory has
no model_index.json before the GPU handoff, mirroring the image loader, so a bad
local pipeline can no longer evict the resident model and only then fail deep in
from_pretrained.
- The custom/env-mode uninstall now removes a sibling stable-diffusion.cpp only when
it carries the Studio owner marker. install_sd_cpp_prebuilt writes the canonical
.unsloth-studio-owned marker on install; uninstall.sh and uninstall.ps1 keep any
unowned checkout (a user's own git clone of stable-diffusion.cpp beside a custom
Studio root is no longer deleted). A pre-marker Studio build is left behind rather
than a user file removed.
Adds regression tests: local video pipeline tagged text-to-video (and a video-named
non-pipeline dir stays untagged so it can never trigger a doomed pipeline load), the
video local-pipeline preflight rejection, the install ownership marker, and the
uninstall keeping an unowned sibling while removing an owned one.
- Diffusion dataset upload now streams each file into a sibling temp file and
atomically os.replace()s it into place only after the whole file is written and
within the size cap. A mid-batch 413 (or any abort) removes the temp, never an
example already stored under the same name, so re-uploading a too-large batch can
no longer truncate or delete a previously uploaded image.
- _diffusion_dataset_summary counts an image as captioned only when it resolves to a
non-empty caption via the same sidecar-over-metadata precedence the trainer uses. An
empty (tombstone) sidecar shadows a metadata row and makes the trainer skip the
image, so counting it over-reported caption_count and mislabeled an effectively
uncaptioned dataset as captioned.
- uninstall.sh/.ps1 now remove a custom/env-mode Studio's native diffusion build that
installs beside the root as a stable-diffusion.cpp sibling (find_sd_cpp_binary
resolves it from the Studio home's parent), guarded by the same unsafe-path check,
and stop processes locking the default-mode stable-diffusion.cpp before removing it.
Adds regression tests for the upload data-loss and caption-count paths and a hermetic
shell test for the custom-root stable-diffusion.cpp removal.
* Fix llama3 RoPE scaling dropped on transformers v5
transformers v5 loads on meta then blanks non-persistent buffers, so
_fix_rope_inv_freq rebuilds inv_freq after load. It recomputed a vanilla
inv_freq and applied _apply_inv_freq_scaling, a no-op on the base
LlamaRotaryEmbedding used by the config/llama3 path, so inv_freq ended up
divided by 1 instead of the config factor (8 for Llama 3.1, 32 for Llama
3.2). This corrupts long-range positions and inflates long-context loss
about 3-5x. transformers 4.x was unaffected.
Route __init__ and the v5 repair through one _unsloth_recompute_inv_freq
so they cannot diverge, and stash the config on the rotary module so the
repair can rebuild the same scaled value.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add test for llama3 RoPE scaling under the transformers v5 repair
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update RoPE drift guard for the recompute refactor and guard the v5 repair
The drift guard's AST tripwire asserted the config-scaling call lived in the
if config is not None branch of LlamaRotaryEmbedding.__init__. The fix moved
that into _unsloth_recompute_inv_freq, so follow it there (with a fallback to
the old inline branch) and add a guard that loader._fix_rope_inv_freq rebuilds
inv_freq through the same helper. Also add a CPU functional check of the helper
and drop the redundant standalone test.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Honor an explicit sdpa or flex_attention request when flash is disabled
When flash attention is disabled for a model, the fallback selection could
downgrade a caller who explicitly passed attn_implementation='sdpa' or
'flex_attention' to a different backend, because the disable reason is
flash-specific. Keep an explicit non-flash request as-is; flash requests
still fall back as before.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments
* Gate honor-explicit attention on provenance and flex support
Only honor an explicit non-flash attention request when it comes from the
caller argument, not from a config value the loaders synthesize (the language
path seeds attn_implementation=sdpa). Honor explicit flex_attention only when
supports_flex_attention is True so excluded/broken configs (e.g. gpt_oss) fall
back instead of selecting a known-broken backend. Explicit sdpa stays honored.
* Honor explicit sdpa through the resolver guard
* Keep SDPA exclusions when honoring an explicit sdpa request
An explicit attn_implementation="sdpa" was re-enabling sdpa for models in
_SDPA_EXCLUDED_MODELS (e.g. gpt_oss) where sdpa is known-broken: the helper
honored the request and the resolver's final not-supports_sdpa guard skipped
the eager downgrade for any explicit request. Honor an explicit sdpa only when
the model is not sdpa-excluded, mirroring the flex guard that already falls
back for _FLEX_EXCLUDED_MODELS via supports_flex_attention. Conservative
supports_sdpa=False (large head dim / attention-sink models) still honors an
explicit sdpa; a synthesized/default sdpa still downgrades to eager.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor DISABLE_SDPA_MODEL_NAMES when honoring explicit sdpa
The honor-explicit-sdpa guard only skipped the sdpa->eager downgrade for models
in _SDPA_EXCLUDED_MODELS (gpt_oss). Gemma3/Gemma3Text disable SDPA through the
loader's DISABLE_SDPA_MODEL_NAMES (their bundled SDPA modules are wrong), so an
explicit sdpa request bypassed the downgrade and re-enabled a known-wrong path.
Extend _is_sdpa_excluded to also treat DISABLE_SDPA_MODEL_NAMES membership as
excluded, replicating the loader's trailing-comma substring match so gemma3 and
gemma3_text match but gemma3n does not. Move the constant into _utils.py (single
source of truth, re-exported from loader.py) to avoid a loader -> _utils cycle.
Conservative supports_sdpa=False models not in either list still honor explicit
sdpa.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Scope MoE expert LoRA detection to actual MLP projection targets
_moe_target_set_from_string treated any regex containing the substring mlp
or ffn as targeting the expert MLP projections. Unsloth's auto-generated
attention-only regex lists mlp, ffn and feed_forward as allowed intermediate
path segments while its final group matches only q_proj/k_proj/v_proj/o_proj,
so attention-only finetuning on MoE models silently enabled expert LoRA as
well: the experts were trained and every MoE layer paid the extra expert LoRA
grouped matmuls. Detect expert intent from the projection names themselves
(gate_proj/up_proj/down_proj/gate_up_proj) instead of the mlp substring.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments
* Detect MoE expert LoRA via mlp path segment, not proj names
The auto-generated target regex always lists every projection leaf
(q/k/v/o and gate/up/down), so keying detection on a proj name mis-fired:
it enabled expert LoRA for attention-only regexes and dropped the
mlp/ffn path regexes. Key on the mlp/ffn/feed_forward/experts path
segment instead, which is present only when the MLP/experts are actually
targeted. Add a regression test for the attention-only case.
* Scope expert LoRA targets to the leaves a regex names
An mlp path alternative with attention-only leaves, for example
(mlp|self_attn).(q_proj|o_proj), no longer enables expert LoRA, and a
regex naming a single expert leaf such as .*experts.*down_proj now
targets only that projection instead of the whole broad set. Generic
mlp projections (.*mlp.*proj) and the auto regex mlp tag block keep the
broad set for fused-expert models whose leaves are plain Parameters.
* Route explicit leaf list into MoE expert detection
An attention-only explicit target_modules list routed through get_peft_regex
for family scoping (e.g. FastVisionModel with vision layers off) yields a
regex carrying the full mlp|feed_forward|ffn|dense component block even though
its leaf group only names q/k/v/o_proj. Keying expert detection on that regex
trained the experts for a language-only/attention-only request. Use the
caller's original leaf list for detection; only the auto path uses the regex,
where the mlp block is the sole MLP-intent signal on fused-expert models.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Respect finetune_mlp_modules and finetune_language_layers scope for MoE expert detection
When an explicit leaf list that names MLP projections (gate_proj/up_proj/down_proj)
is routed through get_peft_regex under finetune_mlp_modules=False, the scoped regex
correctly drops the MLP leaves, but MoE expert detection was still keyed on the
original list and re-added mlp.experts.* via target_parameters, training the experts
the caller had frozen. Same gap for finetune_language_layers=False on vision-only runs.
Prefer the original list only when MLP and language families are both in scope
(preserving the attention-only fix); otherwise honor the scoped result so the frozen
family is respected. Factored the choice into _select_moe_detection_targets with unit
tests over the full selection matrix.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Handle odd shapes and non-float scales in FP8BlockQuantLinear
Small fp8 checkpoints (e.g. tiny test models) break the block-quantized
linear in three ways: weight scales stored in a float8 dtype such as
float8_e8m0fnu have no triton dtype mapping; activations whose hidden dim is
not a multiple of the activation quant block fail act_quant's divisibility
assert; and weights whose dims are not multiples of the weight block cannot
be tiled by the triton dequant kernel.
Cast non-float scales to float32 on entry, and when the hidden dim does not
divide into the activation block, dequantize the weight and run a plain
matmul instead of the fp8 block matmul. The dequant goes through a new
shape-safe helper that falls back to a torch-native scale expansion when the
weight does not tile evenly; backward uses the same helper so the gradient
path works for every shape the forward accepts. Full-size checkpoints are
unaffected.
* Add tiny / e8m0 fp8 block-quant regression test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix FP8 block-quant fallback: real block size in dequant and scalar-scale fast path
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Route rectangular fp8 blocks through torch dequant and keep block_size across e8m0 upcast
The triton weight_dequant kernel uses one BLOCK_SIZE for both axes, so
rectangular blocks (block_size[0] != block_size[1]) mis-index the column
scale and corrupt grad_X. Route those through the torch scale expansion,
which handles each dimension independently, and keep the triton path for
square blocks only.
Also preserve a block_size attribute carried on the scale tensor across the
e8m0 -> float32 upcast so the later lookup no longer falls back to [128, 128].
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix export-time trust_remote_code bypass in FP8/INT8/GGUF-LoRA export
The torchao, compressed-tensors, and LoRA GGUF export paths re-read the merged
checkpoint and used to set trust_remote_code from the checkpoint config's static
auto_map (the torchao path also scanned the staged tokenizer/processor configs).
A model that loads with built-in Transformers classes can carry an auto_map entry,
which skips the load-time remote-code consent scan (that only runs when the load
already requested trust_remote_code) yet flips trust_remote_code on at export,
running unvetted custom code.
Derive the reload trust_remote_code from the approved load decision instead: a new
_loaded_via_remote_code() checks whether the in-memory model / tokenizer was itself
loaded from custom code (its class lives in the transformers_modules package),
walking PEFT / wrapper layers. Built-in-loaded models no longer gain trust from
config metadata; genuine custom-code models (loaded with consent) still reload
correctly. Add CPU-only regression tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden _loaded_via_remote_code against a None/missing __module__
Read type(node).__module__ via getattr and require a string before startswith,
so a dynamically created or C-extension class with a None module does not raise
during export. Add a regression test.
* Split model and tokenizer trust for the compressed subprocess, walk processor components
The compressed-tensors export collapsed model and tokenizer trust into
one --trust-remote-code flag, so an approved custom tokenizer would have
let an unapproved model's custom code run inside the quantization
subprocess. The subprocess now takes --trust-remote-code-tokenizer for
the processor load and keeps --trust-remote-code for the model loads,
matching the torchao path's separate model_trust / tok_trust.
_loaded_via_remote_code now also walks processor components (tokenizer,
image_processor, feature_extractor, video_processor), so an approved
custom tokenizer held inside a built-in ProcessorMixin keeps its trust
on the export reload instead of failing with trust_remote_code=False.
The walk is a bounded BFS with a seen set so wrapper cycles terminate.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix: skip fp16/bf16 validation for full finetuning in RL trainers
When doing full finetuning (FFT) of a bfloat16 model, the fp16/bf16
mismatch validation fires before the corrective logic runs, causing a
misleading error even though the code would properly handle it downstream.
Skip the validation when full_finetuning is active.
Fixes#6731
* Fix: auto-correct fp16/bf16 mismatches for full finetuning before validation
Instead of entirely skipping validation (which could let mismatches
through when mixed_precision_dtype is float32), auto-correct explicit
fp16/bf16 settings that conflict with the model's dtype for FFT. This
way the existing validation still catches real mismatches for non-FFT
cases, and the corrective logic below handles the normalized settings.
Fixes the issue raised in Codex review of PR #6813.
* Guard Windows ROCm torchao override skip
Detect installed ROCm torch directly before applying the torchao override so Windows ROCm environments never install the crashing torchao package even if the earlier ROCm-installed flag is missing.
* Update unsloth/models/rl.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/install_python_stack.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Harden ROCm probe and sync RL precision flags
Tolerate stray stdout noise when probing Windows ROCm torch installs by checking the last non-empty output line, matching the existing torch version probe behavior. Also keep args.fp16 and args.bf16 synchronized with the full-finetuning precision auto-corrections in the RL trainer patch so downstream eval settings see a consistent TrainingArguments state.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add MLX trainer compatibility shims
Patch imported MLXTrainer and MLXTrainingConfig objects to preserve the expected dataclass field ordering and to provide a _train_dataset_for_batches fallback when older trainers or test doubles only expose train_dataset. Also add focused worker tests covering both compatibility paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope PR to Windows ROCm torchao guard
* Restore PR scope to Windows ROCm guard
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test: cover Windows ROCm torchao skip behavior
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Ayushman Paul <ayushman@HP>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Fix TrainingArguments silently disabling unsloth gradient checkpointing
* Cover loaded adapters and preserve explicit None in GC restore
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: expose full compressed-tensors scheme set in an export formats dropdown
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: multi-select export formats, portable torchao FP8/INT8, GGUF LoRA, source parity
Export page overhaul on top of the formats dropdown:
- Unify merged precision into one sorted multi-select list (16-bit first, then
8-bit, then 4-bit). Drop "vLLM" from labels, add INT8 (W8A8), INT8 (W8A16),
INT4 (W4A16), MXFP4, MXFP8. Quick formats render as toggle pills; the rest live
in a multi-select "More formats" dropdown, so several formats export in one run.
- Add a portable torchao FP8/INT8 save path (Float8WeightOnlyConfig /
Int8WeightOnlyConfig) that needs no NVIDIA GPU to produce and loads in vLLM.
FP8 serializes to safetensors, INT8 to .bin. Wired into save_pretrained_merged
and push_to_hub_merged via a TORCHAO_EXPORT_SCHEMES registry and
_unsloth_save_torchao, parallel to the compressed-tensors path.
- Hide NVIDIA-only compressed-tensors formats when no NVIDIA GPU is present; keep
16-bit and portable FP8/INT8. The backend also rejects a compressed request on
non-NVIDIA hardware so it stays authoritative.
- Relax merged export to non-PEFT models so Local Model and Hugging Face sources
get the same 16-bit / compressed / portable options.
- GGUF: send the whole quant list in one call (merge once, quantize many).
- LoRA: add a GGUF adapter option (convert_lora_to_gguf.py) with an outtype
select (f16/bf16/f32/q8_0/auto), alongside the safetensors adapter.
- Thread the new fields through models, routes, orchestrator, and worker; extend
the export tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate export by accelerator with a torch-aware reason; fix export save dir naming
Export runs through Unsloth, which requires a compute accelerator (NVIDIA/AMD/Intel
GPU or Apple MLX) and has no CPU code path, so a bare-CPU host cannot export even
with PyTorch installed. Add export_capability() in utils/hardware that reports
export_supported plus a precise reason so the UI stops showing a generic "no GPU":
- pytorch_not_installed: a --no-torch install (even a physical GPU is unusable)
- no_accelerator: PyTorch present but no supported accelerator (bare CPU)
- mlx_unavailable: Apple Silicon where the MLX stack is missing or too old
Expose the fields on /api/system/hardware and /api/system, and guard the mutating
export routes (load-checkpoint, export/merged|base|gguf|lora) with HTTP 400 and the
reason, leaving read-only endpoints usable so the Export page still renders.
Make core/export/export.py import without PyTorch and without a usable accelerator
(the Unsloth import is caught) so the export worker degrades to a clear message
instead of crashing at import.
Frontend: keep /export reachable on chat-only hosts and gray out the method and
format options with the backend reason (Alert plus disabled MethodPicker) instead
of silently redirecting to /chat, so users see why export is unavailable.
Also fix the export save directory producing "model/null" for Local Model and
Hugging Face sources that have no run/checkpoint, naming the folder from the model id.
* CI: validate Studio export capability gating on Linux, Windows and macOS
Add a small pytest matrix that runs studio/backend/tests/test_export_capability.py
on ubuntu-latest, windows-latest and macos-latest. It confirms, on each real OS,
that hardware.export_capability() reports the right decision and reason
(pytorch_not_installed, no_accelerator, or mlx_unavailable) and that the export
backend imports without PyTorch and degrades to a clear message instead of crashing.
Hosted runners have no GPU/MLX, so this covers the "export unavailable, here is why"
path a Mac/Windows user without an accelerator sees; a real accelerator export is
validated separately. The job installs only a CPU PyTorch plus the backend import
deps (no unsloth, triton, or llama.cpp), so it runs in seconds with no GPU.
* Studio export: address Codex review (source-aware gating, GGUF LoRA token/MLX/guard)
Frontend (export-page):
- Gate LoRA and quantized-model restrictions on the active source. isAdapter /
isQuantized come from the selected checkpoint; in Local Model / Hugging Face
("model") source mode they were stale, so LoRA stayed wrongly enabled for a
direct base model (backend then rejects "No adapter to export") and a stale
"quantized" flag disabled every method for an unrelated, exportable model. Add
effectiveIsAdapter / effectiveIsQuantized (false outside checkpoint mode) and use
them in the method-reset effect and the MethodPicker disabled state.
- Hide the GGUF LoRA option on a macOS/MLX host (the backend rejects GGUF LoRA on
MLX), so users no longer pick it, wait through the load, and always fail. Disable
the "GGUF adapter" button on a Mac host and never send loraGguf there.
Backend (core/export/export.py):
- Pass the HF token into the GGUF LoRA conversion (save_pretrained_gguf), so a
gated/private base model's config fetch in convert_lora_to_gguf.py is
authenticated; without it the load can succeed but the conversion fails.
- Guard the save_pretrained_gguf capability check with getattr so an older Unsloth
model that lacks the method returns the clean "not supported" message instead of
an AttributeError that surfaces as a generic 500.
* Studio export: address 2nd Codex review (CI index, empty merged, test import)
- studio-export-capability-ci.yml: add --extra-index-url https://pypi.org/simple to
the torch install so torch's transitive deps still resolve; --index-url alone
replaces PyPI with only the CPU wheel index, which does not serve all of them.
- export-page handleStart: reject an empty merged selection (mirrors canExport), so
clicking the panel's Start button with every precision pill deselected no longer
submits mergedSelections: [] and launches an unintended default 16-bit export.
- test_export_imatrix_compressed: the torchao-registry test now reads unsloth/save.py
as text (like the other ast/string checks) instead of `import unsloth.save`, which
raised ModuleNotFoundError in the CPU studio-backend suite that has no unsloth
installed.
* Studio export: make comments succinct across the export changes
* Studio export: use load token for local GGUF LoRA export of gated bases
* Studio export: harden portable torchao path and gate multi-format Hub push
torchao (_unsloth_save_torchao):
- merge to an isolated temp staging dir so a co-selected 16-bit output at save_directory is not deleted
- narrow VLM detection to vision_config / ForVisionText2Text so T5/BART/Whisper are not misrouted
- forward trust_remote_code (from auto_map) to the reload so custom-code models export
Export UI:
- hide portable torchao formats on macOS/MLX (backend rejects quantized export there)
- restrict a Hub merged export to a single format (each writes to the repo root)
* Studio export: torchao tokenizer remote-code + XPU offload, scale GGUF timeout
torchao (_unsloth_save_torchao):
- honor auto_map in the staged tokenizer/processor configs (not just model.config) when
deriving trust_remote_code, so custom-code tokenizers reload after the merge
- offload single-device XPU models to CPU (and empty the XPU cache) before the reload, matching
the CUDA path, so an Intel GPU that fits the model once does not OOM on the second copy
Export orchestrator:
- scale the GGUF wait timeout by the number of requested quants so a multi-quant list export of a
large model does not time out at a flat 3600s
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio export: show portable torchao formats only on non-NVIDIA (CPU) hosts
Portable torchao FP8/INT8 is the fallback for hosts without the NVIDIA compressed-tensors path.
On an NVIDIA GPU the compressed-tensors FP8/FP4/INT formats are the intended path (llm-compressor
auto-installs), so hide the portable duplicates there; keep them on CPU / non-NVIDIA hosts and
continue hiding them on macOS/MLX.
* Studio export: report all output folders and the exported formats
- Multi-format merged export now collects every sibling output directory (one per selected
precision) instead of only the last; the success banner lists them all.
- Show the selected precision formats in the run summary (a Formats row, like GGUF Quantizations),
so the panel says what is being exported rather than just 'Merged Model'.
- Persist the selected formats in the run summary and seed them on mount, so navigating away and
back (or toggling the export method) restores the selection instead of resetting to 16-bit.
* Studio export: list all output formats, add GGUF LoRA target, default Q8_0, auto-select newest checkpoint
- Progress/summary panel now shows a Formats row with the selected merged
formats, and the success banner lists every output folder a multi-format
merged run creates (one line per format) instead of only the last one.
- Merged format selection is seeded from the active run, so navigating away
and back (or switching method cards) no longer resets it to 16-bit.
- GGUF / Llama.cpp now offers an Export target toggle (Full model or LoRA
adapter) for adapter checkpoints, reusing the LoRA GGUF export path.
- Removed the Auto GGUF LoRA output type and defaulted to Q8_0 in the UI,
the request model, and the backend defaults; the outtype list is now
Q8_0/F16/BF16/F32. Core save.py still accepts auto for external callers.
- When a finetune has no checkpoint selected, auto-select the newest one.
* Studio torchao export: robust reload class + optional VLM import
Two fixes to the portable torchao FP8/INT8 export reload, from review of the
narrowed VLM detection:
- Encoder-decoder seq2seq checkpoints (T5/BART/Whisper) are not causal LMs.
With the narrowed is_vlm test they now correctly skip the image-text class,
but fell through to AutoModelForCausalLM and failed to reload after the merge.
Reload them with their own architecture class from the config instead.
- AutoModelForImageTextToText was imported unconditionally at the top of the
torchao path, so on Transformers builds without that class the import aborted
every torchao export (even text-only). Import it lazily only for a VLM, with
the AutoModelForVision2Seq fallback used elsewhere in Unsloth.
* Studio: enable FP8/FP4 compressed export for newer-transformers models
The shipped llm-compressor 0.10.x pins transformers<=4.57.6, so FP8/FP4 export failed
for models needing a transformers 5.x sidecar (Qwen3.5, Gemma-4, Qwen3-Next): the
quantization subprocess crashed importing the removed TORCH_INIT_FUNCTIONS.
Run the quantization against a dedicated llm-compressor-main "shadow": a --target
package dir (transformers 5.10.2 + llm-compressor main + compressed-tensors) layered
over the existing torch. It installs --no-deps so torch is never touched (works on any
Studio torch build), is provisioned lazily and fingerprint-cached, and can be turned
off with UNSLOTH_DISABLE_LLMCOMPRESSOR_MAIN.
- transformers_version.py: provision + validate .venv_llmcompressor.
- export.py: route all compressed exports through the shadow when available; else keep
the workspace 0.10.x path and fail fast past its transformers ceiling.
- save.py: launch _compressed_quantize.py with a clean PYTHONPATH = shadow.
- _compressed_quantize.py: skip linear_attn / vision tower / MTP modules (matches the
RedHatAI and NVIDIA reference quants, and is required by the grouped schemes).
Verified all four schemes (fp8, w8a8, w4a16, mxfp4) on Qwen3.5-9B and Llama-3.2-1B, and
fp8 on Gemma-4, end to end through Studio.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix GGUF LoRA export tests
* Fix export CI expectations
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <112766706+wasimysaid@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* fast_generate: clear error for vLLM-style inputs when fast_inference=False
When fast_inference=False, fast_generate falls back to HuggingFace
generate, and the wrapper already rejects vLLM-only usage (a
sampling_params or lora_request kwarg, or a string prompt). A vLLM prompt
dict ({'prompt':..., 'multi_modal_data':...}) or a SamplingParams passed
positionally slipped through and hit transformers.generate, raising a
cryptic 'SamplingParams object has no attribute update'. Detect both and
raise the same clear 'only supported with fast_inference=True' error.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate: also reject positional list of SamplingParams and list of vLLM prompt dicts
Address review feedback: the slow-mode guard missed SamplingParams passed inside a
positional list and a list of {"prompt": ...} dicts, both valid vLLM batched shapes
that leaked into transformers.generate. Fold the checks into small predicates and
extend the GPU-free test (now 7 reject + 3 pass).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test_fast_generate_slow_guard: expose assertions via a test_ function so pytest collects them
The assertions lived in run(), only called from __main__, so pytest reported no tests
collected and CI skipped the coverage. Rename to test_fast_generate_slow_guard; the
standalone script entrypoint still works.
* fast_generate: reject vLLM tokenized/embeds prompt dicts in the slow-mode guard
vLLM also accepts prompt dicts keyed by prompt_token_ids or prompt_embeds, not just
prompt/multi_modal_data. Those slipped past the slow-mode guard and fell through to
HuggingFace generate with a cryptic error. Recognize all vLLM prompt-dict keys and
add a TokensPrompt test case (now 8 reject + 3 pass).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate slow-mode guard: catch vLLM prompts= keyword form
vLLM's generate names its first argument `prompts`, so a slow-mode call
like fast_generate(prompts="hi") or prompts=[{"prompt": ...}] bypassed the
guard and leaked into HuggingFace generate as an unexpected kwarg. Check
kwargs["prompts"] with the same _is_vllm_prompt predicate and add two test
cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate slow-mode guard: reject vLLM tokenized prompt kwargs
vLLM's legacy call shape passes tokens as prompt_token_ids= (and prompt_embeds=),
which are not HuggingFace generate arguments. In slow mode these bypassed the
guard and leaked into HF generate as unexpected kwargs. Reject their presence
with the same tokenize-first message and add a test case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate slow-mode guard: treat prompts= as vLLM-only
prompts is a vLLM keyword, not a HuggingFace generate argument, so any value
passed as prompts= (including a bare token-id list, which _is_vllm_prompt
deliberately ignores for positional HF token ids) is a vLLM-style call. Reject
prompts= / prompt_token_ids= / prompt_embeds= on presence, and keep the
conservative _is_vllm_prompt check only for the positional arg.
* fast_generate slow-mode guard: reject vLLM prompt kwargs on presence
prompts / prompt_token_ids / prompt_embeds are vLLM-only keyword names that
HuggingFace generate does not accept, so a defaulted call like prompts=None
should raise the actionable slow-mode error instead of leaking a None kwarg
into HF generate. Check membership in kwargs rather than a non-None value.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* feat: add mlx public trainer api
* test: cover mlx public trainer api
* fix: preserve mlx epoch trainer configs
* fix: pass mlx warmup ratio through config
* fix: align mlx trainer dataset order
* fix: keep mlx chat templates import-light
* fix: infer mlx trainer context length
* fix: mirror cuda mlx context defaults
* fix: align mlx notebook trainer defaults
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: keep mlx public helpers import-light
* refactor: reuse mlx optimizer normalization
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: address mlx review feedback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: tighten mlx training argument parity
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: align mlx trainer eos default
* Fix MLX trainer to accept DataCollatorForSeq2Seq and handle TokenizerWrapper in get_chat_template
* Trim redundant docstrings on internal MLX helpers
* MLX review fixes: Studio optimizer import-safe on non-MLX hosts, preserve explicit max_length, skip MLX tests before import
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* MLX review round 2: defer max_length to model context, optimizer alias fallback for older zoo, skip non-MLX test on missing GPU deps
* MLX review round 3: keep chat_templates importable without torch on MLX
* fix: preserve MLX trainer notebook shims
* fix: ignore CUDA tokenizer moves on MLX
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: harden MLX trainer shims
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: unwrap MLX scheduler enum args
* fix: coerce integral MLX epoch counts
* fix: spoof CUDA compatibility APIs on MLX
* fix: harden MLX notebook compatibility shims
* MLX: add torch.cuda.mem_get_info to the compatibility shim
Notebook memory cells call torch.cuda.mem_get_info()[0] directly (not gated by
is_available), so on MLX it raises without a shim. Return (free, total) bytes
from the MLX device stats, consistent with the other torch.cuda compat helpers,
and add a matching assertion to the compat-API test.
* MLX: use active memory for mem_get_info; fix BatchEncoding.to keyword device
Address review on the MLX compatibility shim:
- torch.cuda.mem_get_info() now derives free bytes from current active MLX
memory instead of the peak high-water mark, so a capacity check stays
accurate after a transient spike or a prior run.
- BatchEncoding.to(device=...) passed by keyword no longer forwards a positional
None alongside the keyword (which raised "multiple values for 'device'"), so
non-CUDA keyword moves like .to(device="cpu") delegate correctly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* MLX: accept preserve_dataset_order; stub RL trainers with a clear error
Two fixes so unmigrated notebooks behave predictably on MLX (torch present):
- preserve_dataset_order is a real MLXTrainingConfig field but was missing from
the extra-argument allowlist, so passing it (as a config or trainer kwarg)
could be rejected as unknown on a zoo without the field. Add it to
_MLX_IMPLEMENTED_EXTRA_ARGUMENTS so the documented no-shuffle path is reachable.
- GRPO/DPO/ORPO (and KTO/PPO/Reward) have no MLX trainer yet. Retarget the ones
the installed trl exposes to a stub that raises a clear 'not supported on MLX'
error instead of importing the real torch/CUDA trainer and crashing deep
inside it. Only existing trainers are retargeted (no invented attributes),
idempotent across re-imports.
* MLX: make RL-trainer stubbing import-safe; back current-memory APIs with active memory
Address review on the MLX shims:
- The RL-trainer stub loop probed trl with getattr(_trl, name), which triggers
trl's lazy trainer import and pulls torch -- that can crash import unsloth on a
torch-free MLX install just to check existence. Decide what to stub from
trl.__all__ + already-materialized attrs (vars) instead; never resolve the real
trainer. All trl trainer names are in __all__, so they are still stubbed (even
torch-free), and the probe no longer imports torch.
- torch.cuda.memory_reserved / memory_allocated (the current, non-max APIs) were
aliased to peak max_memory_reserved. Back them with current active MLX memory so
cleanup / capacity checks see live usage; max_* keep the peak high-water mark.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* MLX: keep TRL's SFTConfig epoch default under the trl.SFTConfig alias
Unmigrated notebooks import SFTConfig from trl, which the MLX build aliases to
the public training-args class. TRL/HF SFTConfig defaults to num_train_epochs=3
(max_steps=-1); the native MLX config defaults to max_steps=60. So an SFTConfig
built without an explicit length silently ran 60 MLX steps instead of TRL's 3
epochs under the alias. Alias trl.SFTConfig to a thin subclass that seeds the
TRL epoch default only when neither max_steps nor num_train_epochs is given;
explicit lengths pass through untouched, and the native public args class keeps
its MLX default. Epoch mode is supported by the MLX trainer.
* MLX CI: keep the GGUF reload smoke under the job timeout
The RELOAD-GGUF-via-llama-cli step timed out at 300s. BF16 GGUF decode is
CPU-bound on the macOS runner (~10s+/token), so generating 24 tokens landed
right on the 300s cliff and killed the process. This step is a save/reload
integrity smoke (it only needs a few chars of output), so the token count is
incidental: generate 8 tokens with explicit threads and a small headroom on the
subprocess timeout, all env-tunable (UNSLOTH_GGUF_RELOAD_N / _THREADS /
_TIMEOUT). Cuts the reload well under the 25 minute job budget.
* MLX: broaden trainer stubs, real peak-memory reset, fix shim tests
Address review on the MLX public API:
- The SFTConfig identity tests asserted trl.SFTConfig is UnslothTrainingArguments,
but the alias now points at the _MLXSFTConfig subclass that preserves TRL's
epoch default, so the MLX suite failed before testing the shim. Assert
issubclass instead.
- torch.cuda.reset_peak_memory_stats was a no-op, so max_memory_reserved kept
earlier model-load peaks across a scoped run. Wire it to mx.reset_peak_memory
with the same core/metal fallback used for the reads.
- The unsupported-trainer stubs were a fixed list, so trainers outside it (a
newer RLOOTrainer) still routed to the real torch trainer. Derive the set from
trl.__all__ (every non-SFT *Trainer) so all non-SFT surfaces fail with a clear
MLX message; names come from __all__ so trl is never resolved.
- The non-MLX export smoke skipped only on missing bitsandbytes/triton; other
absent GPU deps (numpy/torch/unsloth-zoo, or _gpu_init re-raising ImportError)
made it fail on CPU hosts. Skip on any ImportError.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: keep MLX notebook compatibility minimal
* MLX CI: force CPU + small context for the GGUF reload smoke
The RELOAD-GGUF-via-llama-cli step timed out even at 8 tokens (>420s), so it is a
fixed hang, not per-token cost: on the paravirtual macOS runner GPU llama.cpp's
Metal backend stalls, and the gemma3 GGUF advertises a 32768 context that llama-cli
would otherwise fully allocate. Run llama-cli CPU-only (-ngl 0) with a small context
(-c 256); keep generation short. All env-tunable (UNSLOTH_GGUF_RELOAD_NGL / _CTX /
_N / _THREADS / _TIMEOUT). Also print llama.cpp's partial stdout/stderr on timeout so
a future hang is diagnosable instead of an opaque TimeoutExpired.
* MLX CI: export the reload-smoke GGUF as q8_0, not bf16
The GGUF reload via llama-cli timed out on the runner even CPU-only with a tiny
context and 8 tokens. Root cause is the format, not the flags: the smoke exported
quantization_method='not_quantized', which maps to a bf16 GGUF, and llama.cpp's
bf16 CPU decode is unusably slow on the paravirtual macOS runner. Export q8_0
(fast_quantized, the exporter default and what users deploy) instead -- llama.cpp
has optimized q8_0 CPU kernels, so the fresh-process reload loads and generates in
seconds. The reload stays CPU-only (-ngl 0) with a small context.
* test: clear TRL shim before availability check
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Add --with-llama-cpp-dir flag to install.ps1 and install.sh
Users can now pass --with-llama-cpp-dir /path/to/llama.cpp to the
installer to skip downloading or building llama.cpp and use a local
directory instead. A junction (Windows) or symlink (Linux/macOS) is
created at the canonical install location, bypassing both the prebuilt
download (Phase 3) and source build (Phase 4) steps in setup.ps1/setup.sh.
The path is passed via UNSLOTH_LOCAL_LLAMA_CPP_DIR env var which
setup.ps1 and setup.sh read directly.
Ported from the idea in unslothai/unsloth#4384, reimplemented against
current Studio architecture.
* test: add static wiring test for --with-llama-cpp-dir flag
Cross-checks install.sh, install.ps1, studio/setup.sh and studio/setup.ps1
so the flag's contract (parse -> UNSLOTH_LOCAL_LLAMA_CPP_DIR env var -> link
local dir, skip prebuilt download and source build) can't silently regress.
Wired into studio-backend-ci.yml alongside the other tests/sh installer tests.
* Address review feedback on --with-llama-cpp-dir flag
- setup.ps1: delete an existing junction/symlink via DirectoryInfo.Delete()
instead of a recursive remove, which can traverse the link and wipe the
user's real llama.cpp directory on PowerShell 5.1.
- setup.ps1: short-circuit the build chain when a local dir is linked so CMake
never runs inside the user's checkout when it lacks a Windows-layout binary.
- install.sh / setup.sh: resolve paths with CDPATH= cd -P so a set CDPATH
cannot corrupt the resolved path.
- install.sh: seed _WITH_LLAMA_CPP_DIR from UNSLOTH_LOCAL_LLAMA_CPP_DIR so an
exported env var (piped-install style) is honored instead of being clobbered.
- setup.sh: create the root llama-quantize shim when linking a local source
build so GGUF export's check_llama_cpp() still finds it.
- setup.sh / setup.ps1: drop a stale link before the custom-home ownership
assert so re-runs with the flag stay idempotent.
- test: pin the new linked-dir build short-circuit.
* Harden --with-llama-cpp-dir against Codex/Gemini review findings
- install.sh: error when --with-llama-cpp-dir is the final arg with no path,
matching the existing --package/--python post-loop guards (was a silent
fallback to the normal prebuilt/source install).
- studio/setup.sh: canonicalize LLAMA_CPP_DIR before the self-link no-op
compare. _RESOLVED_LOCAL is fully resolved while LLAMA_CPP_DIR was textual,
so a symlinked $HOME made the guard miss and the rm -rf could wipe the
user's real llama.cpp tree.
- studio/setup.sh: make the llama-quantize shim non-fatal; it writes through
the link into the user's tree, which may be read-only (shared/CI cache),
and under set -e a failed ln aborted an otherwise-good reuse.
- studio/setup.ps1: detect a broken junction via Get-Item -Force instead of
Test-Path so a dangling link from a prior run is removed and mklink can
relink to a new valid directory.
- studio/setup.ps1: use Copy-Item -LiteralPath so a source path containing
[ ] isn't treated as a wildcard in the junction copy fallback.
- tests: update the wiring assertions for the LiteralPath copy and the
canonicalized compare.
* Validate/reuse local llama.cpp tree and guard the in-use case
Addresses the second Codex pass on the --with-llama-cpp-dir flag:
- Validate the linked tree before disabling installs (setup.sh + setup.ps1):
reusing a local dir skips BOTH the prebuilt download and the source build,
so the dir must already contain a runnable llama-server (build/bin on
Linux/macOS, build\bin\Release\llama-server.exe on Windows). Bail out with a
clear message instead of linking an unbuilt/wrong-platform checkout and
leaving Studio with no usable binary.
- Treat a canonical-path target as already linked when it holds a build
(setup.sh + setup.ps1): point the flag at ~/.unsloth/llama.cpp itself and an
existing build is reused (skip prebuilt + source) rather than clobbered by
the staged prebuilt installer (which uses os.replace()/replace). An empty
canonical dir still falls through to the normal in-place install.
- Abort when an in-use llama.cpp can't be removed on Windows (setup.ps1):
Remove-Item -ErrorAction SilentlyContinue can silently leave a locked tree
in place; detect that and stop with the same active-process message + exit 3
the prebuilt path uses, instead of junctioning over a half-present dir.
Left as follow-up (already tracked by the PR author as a non-blocker): the
in-app "Update llama.cpp" updater does not yet recognize a local-link install
as externally managed; that fix belongs in studio/backend/utils/llama_cpp_update.py.
* Accept all backend llama-server layouts in --with-llama-cpp-dir validation
The linked-tree validation only accepted build/bin[/Release]/llama-server, but
LlamaCppBackend._layout_candidates() resolves a root-level llama-server first,
then build/bin, then build/bin/Release on Windows. A `make` build or a flat
release extract (binary at the dir root) was therefore rejected with a hard
installer failure even though Studio would have run it.
Validate the same candidate set the backend uses in both setup scripts, and add
wiring-test assertions so the check can't silently narrow again.
* Treat --with-llama-cpp-dir local links as externally managed
A --with-llama-cpp-dir install junctions/symlinks the canonical llama.cpp dir to
the user's own checkout, but two backend paths still treated it as a Studio-owned
tree:
- The in-app updater (llama_cpp_update) offered and could apply an official
prebuilt over the link, writing through it into the user's checkout (or
failing) and silently dropping the link the flag created.
- Orphan cleanup (LlamaCppBackend._kill_orphaned_servers) resolved the linked
root into its kill allowlist, so a llama-server the user launched from the same
checkout was classified as ours and killed on startup.
Detect the canonical dir being a symlink/junction (reparse point) and treat the
install as unmanaged: get_update_status reports unsupported, start_update refuses
with reason "local_link", and the linked root is left out of the orphan
allowlist. Adds behavioral tests (link vs plain dir, updater refusal, and the
spared-vs-killed orphan control).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add behavioral shell test for --with-llama-cpp-dir linking
The existing tests/sh/test_with_llama_cpp_dir_flag.sh is a static grep of the
scripts. This adds a behavioral test that extracts the real link block from
studio/setup.sh (by content anchors, with a self-validating extraction) and runs
it against hermetic fake dirs, asserting the outcomes that matter:
- an external CMake build links and arms neither the prebuilt download nor the
source build
- a flat / make tree (root-level llama-server, no build/bin) is accepted too
- an unbuilt tree is rejected with a non-zero exit and no link left behind
- relinking over a stale link preserves the target's contents (no data loss)
- pointing at the canonical path is a no-op reuse, not a self-referential link
Symlink-identity checks run only where real symlinks exist (skipped on Windows
git-bash copy-mode); the link/skip/no-data-loss checks run everywhere. Wired into
studio-backend-ci.yml next to the static test.
* Install psutil in backend CI so orphan-cleanup tests run
The new orphan-cleanup tests import psutil for the process scan, but the Backend
CI deps step installed studio.txt plus a fixed extras list that omits it, so the
two tests failed with ModuleNotFoundError. Add psutil to both backend pytest dep
steps (kept in shared shape), and guard the import with pytest.importorskip so a
minimal env without psutil skips these tests instead of erroring.
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: Implementation of the Portuguese (Brazil) language and VRAM/RAM monitor.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/frontend/src/hooks/use-gpu-utilization.ts
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/backend/main.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/backend/main.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/backend/utils/hardware/hardware.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/backend/utils/hardware/hardware.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/frontend/src/features/settings/components/usage-examples.tsx
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/backend/main.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/backend/utils/hardware/hardware.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/frontend/src/features/studio/sections/progress-section.tsx
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fix: resolve automated review feedback on API shape
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix review issues for PR #6509: Cpu icon, VRAM percent, system polling
- model-inspector: use the exported CpuIcon (Cpu is not a Hugeicons export)
- app-sidebar: guard the VRAM percent on totalVram to avoid Infinity, and
reset the system poll cache only after each request settles so a slow probe
is reused instead of stacking overlapping requests
- use-gpu-info: populate CPU/RAM on hosts without a GPU
- progress-section: label GPUs by visible_ordinal instead of array index
- hub-page: base the RAM label on systemRamTotalGb
- usage-examples: emit JS sampling and tool options at the top level instead
of nesting them under extra_body (the JS SDK does not unwrap extra_body)
- main: read torch and transformers versions from package metadata instead of
importing the libraries on every system poll, and guard the VRAM math
against null values
- hardware: translate a leftover comment to English
* Harden /api/system: guard psutil.boot_time for PR #6509
Simulating restricted containers and some VMs (where psutil.boot_time can raise)
showed the /api/system endpoint would 500 on the unguarded boot_time call, the
same failure class already handled for cpu_freq, disk_usage, and Process. Wrap
boot_time and return uptime_seconds as null when it is unavailable so the sidebar
monitor degrades gracefully instead of breaking. Widen the uptime_seconds type to
number | null to match.
* Studio: make the sidebar hardware monitor a toggle (default on) for PR #6509
Adds a "Show hardware monitor" switch under Settings > Appearance > Layout,
backed by a localStorage preference (default on), mirroring the existing
useSidebarPin pattern. When turned off, the sidebar hides the VRAM/RAM meters
and useSystemInfo stops the 3s /api/system poll entirely, so no nvidia-smi /
SMI probes run while the monitor is disabled. Adds the en and pt-BR strings.
* Studio: default the sidebar hardware monitor to off (opt-in) for PR #6509
* Studio pt-BR: fix three small translation defects for PR #6509
- learningRateDescription: "5e-5 for CPT" -> "5e-5 para CPT" (leftover English)
- exportScopeRecents: "Recents" -> "Recentes" (untranslated)
- relativeMonthsAgo/relativeYearsAgo: add the missing space ("há {count} meses"/
"há {count} anos") so they no longer render as "há 3meses"
* Studio pt-BR: translate the last 10 fallback keys for PR #6509
Adds the settings.general.storage block (Armazenamento) and the
settings.chat.modelDisclaimer pair, so pt-BR now covers all en keys
(679/679) with no English fallbacks.
* Studio: hide sidebar VRAM row on CPU-only hosts for PR #6509
* Studio: tighten and trim code comments for PR #6509
* fix: UI issue in the stop button dialog box (fine-tuning)
* Studio pt-BR: translate 18 new keys from main merge (password dialog, GGUF export, dataset streaming) for PR #6509
* Rounding to GB
* Fix/adjust System resources tab for PR #6509
* Fix/adjust GPU monitor review items for PR #6509
* Fix/adjust remaining GPU monitor review items for PR #6509
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust MLX resource fallback for PR #6509
* floating window implementation
* resize for floating window
* Fix resource monitor review items
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Restore frontend optional dependency lock entries
* Make GPU selection tests hermetic
* Fix GPU monitor CI test failures
* Bound MLX GGUF reload smoke
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix MLX GGUF reload smoke exit
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Fix gpt-oss offload_embedding and generate() logits_to_keep on fused models
offload_embedding=True moved embed_tokens to CPU but left the input/output device-shuffling forward hooks commented out ('[TODO] Doesn't seem to work!'), so an eager forward/generate with CUDA input_ids hit the CPU embedding and raised a device-mismatch RuntimeError. Re-implement them in a testable helper _install_offload_embedding_hooks that saves the origin device on the module (the pre-hook returns a new tensor, so a device stashed on the original input is lost) and runs the lookup on the embedding weight's CURRENT device. Reading the weight device at call time (not a hard-coded cpu) also handles a non-quantized (bf16) embedding that a later model.to(...) pulls back onto the GPU, which the hard-coded version broke in the opposite direction.
unsloth_base_fast_generate injected logits_to_keep/num_logits_to_keep whenever an inner submodule forward accepted it, but transformers validates generate kwargs against the top-level prepare_inputs_for_generation (plus forward when it takes kwargs). On fused/PEFT-wrapped gpt-oss this raised 'model_kwargs are not used by the model: [logits_to_keep]'. Only inject when the top level would accept it, mirroring transformers _validate_model_kwargs. Behavior is unchanged for every model that works today.
Adds tests/test_offload_embedding_hooks.py and tests/test_generate_kwarg_gate.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload hooks: store origin device on the tensor, not the shared module
The pre-hook stashed the input device on embed_tokens itself, which races when
concurrent forwards share the module (serving). Ride it on the moved tensor and
read it from the post-hook args instead: stateless and thread-safe.
* Also strip mm_token_type_ids that generate() rejects (Qwen3-VL vision GRPO)
The vision processor (Transformers 5.x path) emits mm_token_type_ids, which
Qwen3-VL's generate() then rejects in _validate_model_kwargs on transformers
4.x, so vision GRPO fails at the first rollout:
ValueError: The following `model_kwargs` are not used by the model:
['mm_token_type_ids']
Unlike logits_to_keep this is an incoming kwarg rather than one we inject, so
drop it in unsloth_base_fast_generate when the top level generate does not
accept it, reusing the same _unsloth_generate_accepts_kwarg gate. Extends the
GPU-free gate test with the accept/reject mm_token_type_ids cases (7/7 pass).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim mm_token_type_ids comment
* Trim comments in gpt-oss offload/logits fix (comment-only)
* gpt-oss offload: return embedding output to the decoder device, not the input's
When offload_embedding moves the embedding to CPU, model.device can become CPU and
inputs then arrive on CPU, so returning the output to the input device left it on CPU
and the CUDA decoder hit a device mismatch. Capture the decoder device before offload
and always return there. This also drops the per-request tensor state (stateless, so
concurrent forwards stay correct).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: refuse offload_embedding for tied word embeddings
Tied models share embed_tokens.weight with lm_head, so offloading the weight
to CPU strands the output projection there (device mismatch at generate) and
saves no VRAM since lm_head still needs it on GPU. Detect the shared weight via
get_output_embeddings and raise NotImplementedError instead of loading into a
crash. Untied models (gpt-oss, Llama-3.1-8B) offload unchanged.
Adds tests/test_offload_tied_guard.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: skip embedding offload on fast_inference (vLLM)
vLLM manages its own weights, so offload_embedding cannot apply on the
fast_inference path (previously it was silently ignored). Disable it with a
notice, mirroring the WSL and Windows skips.
* Trim offload embedding comments (comment-only)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: track decoder device live so it survives model.to()
The post-hook returned the embedding output to a device captured at load time.
If a model is loaded on CPU then moved with model.to(cuda), that device is
stale and the output lands on the wrong device. Read the decoder device live
from the (untied) output embeddings, keeping the captured device as a fallback.
Adds a stale-fallback regression test.
* Make generate-kwarg-gate cases pytest-collectable
Cases lived in run(), which pytest does not collect, so CI never exercised the
gate. Expose them as test_generate_kwarg_gate; still runnable via __main__.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* gpt-oss offload: skip a meta (disk-offloaded) lm_head as the return device
A device_map that disk-offloads an untied lm_head leaves its weight on the meta
device until that module's own hook runs, so reading it as the decoder device
would move real hidden states to meta. Skip meta (and a missing weight) and fall
back to the captured device. Adds a regression test.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Daniel Han