unsloth_pip_shim.py: close three more ways a protected package slipped past
_KEEP. An editable line (-e/--editable <target>) inside a -r requirements file
is a real install target, so a protected editable there is now classified and
dropped like the command-line case (new _parse_editable). pip/uv accept the
attached short forms -rreqs.txt / -cconstraints.txt / -epath / -Pname as one
token; these were falling through as opaque options (so an attached -r-only cell
no-op'd and an attached -c/-e/-P value bypassed _KEEP), so the 2-char flag is now
split from its value and routed through the separated-form handling. And a nested
-c constraint inside a -r file no longer records its transformers pin as an
install request (a constraint is not a request; mirrors the top-level -c path).
entrypoint.sh / Dockerfile: gate the CUDA 13 ptxas + NVRTC to sm_103 / sm_121 at
runtime instead of a global build-time default. A cu13 cubin needs a >= 580
driver to LOAD even when it targets an older arch (CUDA has forward, not
backward, cross-major driver compatibility), but the image supports Turing..
sm_120 on a 570+ driver, so the previous global TRITON_PTXAS_PATH ENV + cu13
NVRTC symlink would break ordinary Triton/NVRTC JIT on 570-579 driver hosts. The
build still bakes cu13 (saving the cu12.8 NVRTC as .cu128.orig); a new
select_cuda_jit_tools() in the entrypoint reads the device compute_cap and only
activates cu13 for sm_103/sm_121 (which ship >= 580 drivers), otherwise leaving
Triton on its bundled cu12.8 ptxas and restoring the cu12.8 NVRTC in both the
base and Studio venvs. The base ENTRYPOINT runs for the Studio image too.
Adds 9 pip-shim regression tests and tests/sh/test_select_cuda_jit_tools.sh
(7 device-gating cases); registers the latter in CI and tests/run_all.sh.
docker-publish.yml: freeze the requested unsloth ref to one sha in the prepare
job before the matrix fans out. UNSLOTH_REF / UNSLOTH_STUDIO_REF were raw
expressions re-evaluated per base arch leg and in the Studio build, so a mutable
branch (the workflow_dispatch default unsloth_ref=main) advancing during the run
could bake different unsloth commits under one manifest. Resolve once (same
precedence: dispatch input, else pushed tag, else triggering sha, else main;
ls-remote a branch/tag to a sha, mirroring the zoo/notebooks steps) and read
needs.prepare.outputs.unsloth_ref everywhere.
Dockerfile.studio: run the Studio venv NVRTC cu13 swap on both arches, not arm64
only. amd64 sm_103 (B300/GB300) needs cu13 NVRTC just as arm64 sm_121 does, and
the CUDA dedup never touches cuda_nvrtc, so an amd64 Studio venv would otherwise
keep its bundled cu12.8 libnvrtc and fail NVRTC/jiterator JIT on compute_103. The
base cu13 layer installs cuda-nvrtc-13-0 on both arches, so the target .so.13
exists here regardless of TARGETARCH.
unsloth_pip_shim.py: close three ways a protected package slipped past _KEEP.
Treat -e/--editable as a value-taking flag paired with its target and drop both
when the target is protected (was leaving a dangling -e that failed the cell);
filter -P/--upgrade-package values through _KEEP (a named baked package could be
refreshed while installing another target); and parse the PEP 427 distribution
name out of a wheel URL/path so a bare `pip install https://.../torch-...whl`
drops instead of reinstalling the baked torch. Non-protected editables, upgrade
selectors, and wheels are unchanged. Adds tests/python/test_unsloth_pip_shim.py
(18 regression tests, exec captured via a patched os.execv).
The Studio build symlinks the Studio venv's CUDA libs onto the base venv's
copies to reclaim ~3.7GB. That is only safe when both venvs run the same torch,
but the pre-dedup guard only checked the CUDA family (endswith('+cu128')). A
Studio venv that installed torch 2.10.0+cu128 (an installer capped below the
base's 2.11.0, or a build-time nvidia-smi fallback) would pass that check yet
mismatch the base's 2.11.0+cu128, and the dedup would link incompatible libs.
Capture the base venv's torch from its metadata and assert the Studio venv torch
equals it exactly (version and family) before the dedup runs, so a mismatch
fails the build loudly instead of silently linking skewed CUDA libs. Comparing
to the base venv also avoids hardcoding the version here. The Studio venv reaches
torch 2.11.0+cu128 via the installer's UNSLOTH_TORCH_INDEX_FAMILY=cu128 handling
and its CUDA torch spec allowing 2.11.x.
Base image (torch 2.11.0):
- amd64 unsloth extra: cu128-ampere-torch2100 -> cu128-ampere-torch2110.
The old extra pulls xformers 0.0.34, which hard-pins torch==2.10.0 and
conflicts with the torch==2.11.0 held throughout the build; the torch2110
family pulls xformers 0.0.35 (no torch pin) and resolves cleanly. This
needs an unsloth carrying the torch2110 CUDA extras on main, so merge the
torch2110 extras PR first (default UNSLOTH_REF=main).
- notebook-deps assertion: startswith('2.10.0') -> '2.11.0' so the layer
actually verifies the torch it now installs.
- refresh the torch2100/xformers 0.0.34 references in the surrounding
comments to the torch2110/0.0.35 line.
sm_103 (B300/GB300) JIT override (Codex item):
The cu13 NVRTC/ptxas override was arm64-only (sm_121), and its comment
claimed triton 3.6.0 bundles cu13 ptxas and set TRITON_PTXAS_PATH -- neither
was true: triton 3.6.0's bundled ptxas is CUDA 12.8 (V12.8.93, tops out at
sm_120) and TRITON_PTXAS_PATH was never set. So sm_103 (amd64) and even
sm_121 (arm64) Triton JIT were unfixed.
Run the cu13 install on both arches and actually wire the ptxas override:
- NVRTC swap (cu13 libnvrtc.so.13 over torch's bundled cu12.8 .so.12) now
runs on amd64 too.
- ENV TRITON_PTXAS_PATH=/usr/local/cuda-13.0/bin/ptxas routes every Triton
JIT through the cu13 ptxas. Global rather than per-arch is safe: cu13.0
ptxas spans sm_70..sm_121 (verified: Volta/Turing/Ampere/Hopper through
Blackwell), so no regression for the older GPUs in the arch list.
Verified on amd64 in the built base image: cuda-nvrtc-13-0/cuda-nvcc-13-0
install cleanly from the base's CUDA repo, ptxas lands at
/usr/local/cuda-13.0/bin/ptxas (V13.0.88) and libnvrtc.so.13 at
/usr/local/cuda-13.0/lib64/. The sm_103/sm_121 runtime path itself is not
hardware-tested (no such GPU on hand); precompiled SASS still covers both
via sm_100/sm_120 forward-compat, so only JIT-heavy paths rely on this.
* docker: Colab-grade JupyterLab and Studio UX for the Blackwell image
Stacks a Colab-like JupyterLab and Studio experience on top of the
existing Blackwell image. Additive only: the training stack, CUDA/torch
pinning, and the Studio/JupyterLab/sshd service trio are unchanged.
JupyterLab labextension (prebuilt in a throwaway builder stage, so the
runtime image stays Node-free):
- Unsloth Dark (Monokai) theme, adaptive light/dark by system preference
- Colab-style ArrowDown/Up cell navigation
- top-bar Unsloth logo (stock Jupyter logo disabled and locked)
- #@title lines render as collapsible Heading-2 form bars
- Ctrl+A in a cell output selects only that output, not the whole
notebook (the old behaviour ran notebook:select-all and was laggy)
- right activity bar hidden by default
- overrides.json: per-cell run button without auto-advance, labeled
Restart and Run All, windowing off so collapsing an output does not
snap to the cell top, news/update prompts suppressed
Studio and login branding: Unsloth favicon, page logo, and a dark
Unsloth login page that rotates through the curated Studio sloth
stickers (fail-soft to the logo).
Notebook organization and Colab compatibility (base image):
- categorized folder view built from relative symlinks mirroring the
README sections, rebuilt each boot; real .ipynb files never moved,
and the symlink tree is invisible to the sync state machine
- AMD-* notebooks shown only on an AMD/HIP host (autodetected)
- Docker-only strip of the Colab "Run all on Colab" intro sentence
from unedited notebooks (upstream notebooks unchanged)
- hoist %%capture above a leading #@title form so the cell runs
- the per-cell transformers-sidecar log is silent unless
UNSLOTH_ENABLE_LOGGING=1
Dependency pinning and naming: the curated notebook extras are pinned to
their resolved versions for reproducible rebuilds; decord is split into
its own fail-soft install (no aarch64 wheel). The lean base image is
renamed from :base to :core.
Adds tests/validate_studio_features.py, a static self-test for the
labextension plugins, overrides keys, and branding wiring.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* docker: address review feedback on the JupyterLab/Studio UX
- unsloth_nb_view.py: rebuilding the categorized view no longer deletes
user files. The view is also JupyterLab's landing dir, so a user may
save real notebooks there; _clear_view now unlinks only the symlinks we
own and removes only folders that end up empty, leaving regular files
in place. It also tests islink before isdir, so a view that is itself a
symlink to a directory is unlinked instead of being walked into (which
would have wiped the symlink target).
- studio_launch.sh: derive the landing URL and preferred_dir from
UNSLOTH_NOTEBOOKS_VIEW_DIR / UNSLOTH_SKIP_NOTEBOOK_VIEW, the same env
the sync script uses, instead of hard-coding /workspace/Unsloth
Notebooks. A relocated or disabled view no longer opens JupyterLab on a
missing folder; it falls back to the default /lab over /workspace.
- Dockerfile.studio: the labext-builder stage now installs Node 20 from
NodeSource. Ubuntu 24.04's distro nodejs is 18, below JupyterLab 4.6's
declared Node >=20 engine. Node stays confined to the throwaway builder
stage, so the runtime image is unchanged.
- .dockerignore: explicitly allowlist jupyter/install_sloth_stickers.py
alongside its sibling jupyter assets, rather than relying on the
directory re-inclusion.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* docker: publish lean image as :core and full image as :studio
Complete the base->core (and "studio as studio") tag rename so the publish
workflow matches the user-facing helpers and the Dockerfile.studio header.
- The lean training image now publishes as :core (core-<tag>, core-nightly,
core-sha-*); run.sh / docker_confirm.* already told users to pull :core, but
docker-publish.yml still tagged it :base, so that pull would have 404'd. The
per-arch digest artifacts are renamed to match.
- The full Studio image keeps :latest and gains a stable :studio alias, matching
the Dockerfile.studio header.
Both the merge and post-publish smoke-test metadata blocks are updated together.
Internal "base image" wording (the layer Studio builds FROM) is left as-is.
* docker: address second-round review feedback on the JupyterLab/Studio UX
- studio_launch.sh: also gate the categorized-view landing URL on
UNSLOTH_SKIP_NOTEBOOK_SYNC (the entrypoint skips building the view entirely in
that mode), not just UNSLOTH_SKIP_NOTEBOOK_VIEW, so a no-sync container does not
land on a missing folder.
- Dockerfile.studio: scope the sticker-install "|| echo" fallback to only the
sticker step via a { ...; } group. It was attached to the whole branding &&
chain, so a failure in a REQUIRED step (JS resolve, favicon/logo/login copy)
was swallowed and the build continued with broken branding.
- unsloth_nb_view.py: when creating the categorized symlinks, only replace our
own stale symlinks; if a real user file already occupies that name, keep it and
skip the link instead of os.remove-ing it.
- overrides.json: drop doNotDisturbMode (it silenced ALL JupyterLab toasts,
including kernel-restart / connection-drop feedback). The news/update prompts
are already off via fetchNews / checkForUpdates.
- Dockerfile: keep decord mandatory on amd64 (fail the build on a missing or
incompatible wheel) and only fail-soft on arm64/other arches that have no wheel.
- cellNav.ts: do not hijack ArrowUp/Down when focus is in an interactive output
widget / form control, or while a completion popup is open, so ipywidgets
controls and autocomplete at cell boundaries keep working.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* docker: keep Studio branding RUN free of comments inside the line continuation
Move the sloth-sticker fail-soft explanation above the RUN so no comment line
sits between backslash-continued commands. BuildKit strips such comments, but
keeping the RUN body a plain && chain removes the ambiguity for non-BuildKit
builders and static linters. The { ...; } fail-soft scoping is unchanged.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* docker: AGPLv3 attribution + integrity guard for the Studio/JupyterLab image
Make it obvious the image is built by Unsloth and hard to white-label out with a
shallow find-and-replace, and surface the AGPLv3 license + copyright in the UI.
Visible attribution (labextension):
- Help > "About Unsloth Docker Studio" dialog (about.ts): Unsloth logo, the
AGPLv3 notice, "Copyright 2026-Present the Unsloth team", and source/website/
license links. Added to the Help menu and the command palette.
- The JupyterLab loading splash is replaced with a spinning Unsloth logo
(splash.ts, provides ISplashScreen; honors prefers-reduced-motion). The stock
@jupyterlab/apputils-extension:splash is disabled+locked at build time, like
the stock logo.
- AGPLv3 footer (license + copyright + links) on the branded login page.
- Labextension relicensed AGPL-3.0-only; SPDX headers on every source file.
Anti-tamper (no encoded/obfuscated strings -- plain readable text only; the one
data URI is the logo image):
- A canonical, plain-text attribution set lives in unsloth_branding.py with a
TypeScript mirror (branding.ts) bundled verbatim into the labextension, so the
phrase, copyright, links and plugin ids are spread across independent layers.
- unsloth_branding.py verifies all of these across the installed files (AGPLv3
text, login footer, theme, labextension package + built bundle strings, logo,
favicon) and fails loudly if any are missing. It runs at three layers:
build time (fails the image build), the whole-container launcher
(studio_launch.sh refuses to start), and as a jupyter_server extension
(refuses to serve JupyterLab).
- tests/studio/test_branding_guard.py: positive + per-marker negative coverage,
plus a check that no base64/decoder obfuscation crept into the attribution.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* docker: address #6681 review round 2 (colab magics, output select, branding guard)
- unsloth_colab_compat.py: only hoist a leading `%%` cell magic above the Colab
`#@title` form for magics whose body runs as code (capture/time/bash/python/
...). Content magics (%%writefile, %%html, %%latex, ...) are left untouched so
the form comment is never injected into the written file / rendered output.
- outputSelect.ts: stop trusting the text selection anchor to decide ownership
of Ctrl/Cmd+A. A stale selection inside an output survives a click onto a
command-mode cell or the file browser, which made select-all keep re-selecting
the old output. Gate on the keystroke target or the last pointer-down (reset to
null on any click outside an output) instead.
- unsloth_branding.py: also reject page_config.json that disables the Unsloth
labextension or any of its plugin ids via disabledExtensions (dict or list
form); that leaves the bundle on disk so the prior checks passed while the
logo/About/splash attribution was stripped at load. Lock unsloth-jupyterlab in
Dockerfile.studio as well (defense in depth), and add guard tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* labext: pin JupyterLab extension deps; confirm.ps1 /login probe
Pin the unsloth-jupyterlab npm deps to exact versions matching the baked
jupyterlab==4.6.0 (builder stays 4.5.9, its newest release) instead of floating
^/~ ranges, so the same commit always builds the same labextension bundle.
Also probe JupyterLab /login (not /api, which 403s behind a password hash) in
the Windows confirmation script.
* docker: categorize AMD/domain notebooks and wire the feature validation into CI
unsloth_nb_view.parse_readme only reset the folder section on level-3
(###) headings. The notebooks README carries level-1 domain headers
(# AMD Notebooks, # Kaggle Notebooks) with their own nb/*.ipynb link
tables and no intervening ###, so those notebooks were mis-filed under
the previous stale section (all 148 AMD notebooks landed in Other
Notebooks on an --amd build). Reset on any heading level and strip a
leading emoji/symbol run so the domain notebooks get their own clean
folder.
Also run tests/validate_studio_features.py explicitly in the repo CPU
job. It is named validate_* (not test_*) so pytest never collected it,
which meant a regression in the notebook view, Colab compat, strip,
JupyterLab defaults or login branding failed CI only when run by hand.
* labext: use caret ranges so jlpm dedups JupyterLab/Lumino singletons
The exact pins introduced earlier (@jupyterlab/* 4.6.0, @lumino/widgets
2.8.0, @jupyterlab/builder 4.5.9) break the Dockerfile.studio
labext-builder stage. Exact-pinning the framework packages defeats
jlpm's (yarn classic) hoisting: transitive @jupyterlab deps request
caret ranges that resolve to newer patch releases (e.g. @jupyterlab/
notebook pulls @jupyterlab/cells ^4.6.0 -> a newer patch), so jlpm
installs a second nested copy alongside the exact top-level one. Two
copies of @jupyterlab/cells and @lumino/widgets in the tree produce
TS2345 "not assignable" errors (protected-member/identity mismatch)
and the build fails.
Caret ranges let jlpm collapse every @jupyterlab and @lumino package
to a single hoisted copy, which is required for a JupyterLab prebuilt
(federated) extension: at runtime those packages are shared singletons
provided by the host JupyterLab, so the build-time versions only need
to type-check against one consistent tree, not match an exact runtime
patch. This is the version set the published image was built and
validated with end to end.
Verified by building the labext in isolation against the base image
(Node 20 + bundled jlpm): caret ranges build clean (webpack compiled
successfully); the exact pins fail with the duplicate-package TS
errors.
* ci(studio-backend): trigger on docker/** so the JupyterLab feature validation guards docker-only changes
The 'Docker JupyterLab/notebook feature validation' step runs
tests/validate_studio_features.py, which checks docker/jupyter (the
labextension, overrides.json, login branding) and the docker notebook
helpers. The pull_request paths filter listed studio/unsloth/tests but
not docker/**, so a PR that only touches docker/ would skip that step
and a regression in those files could pass CI. Add docker/** so the
validation runs whenever the files it checks change.
* jupyter: center the login card and place the attribution below it
#site was a flex container using the default row direction with two children
(the login card and the AGPLv3 attribution), so they rendered side by side:
the card sat left of centre and the attribution floated up to the top-right.
Stack them in a column so the card is horizontally centred and the attribution
sits below it as a footer, matching the intended single-column layout.
* jupyter: refresh Studio attribution, About dialog and loading splash
- Attribution now reads 'Built by the Unsloth team' with a single Apache 2.0 /
AGPLv3 license link (to the repo license section) on the login page and in the
About dialog, replacing the plain 'Built by Unsloth. Licensed under the GNU
AGPLv3.' line. The integrity guard, its canonical PHRASE and the branding tests
are updated to match.
- About dialog: left-align the link rows so the labels line up instead of each
row centering independently; add an 'Unsloth Reference' link to the docs, and a
Licenses section listing Unsloth Studio (AGPLv3) and Unsloth Core (Apache 2.0)
alongside the full license link.
- Loading splash now reads 'Loading Unsloth Docker' instead of the attribution
label, via a dedicated SPLASH_LABEL constant.
* docker: document the branding attribution as an AGPLv3 Section 7 notice
Add docker/NOTICE and docker/jupyter/BRANDING.md so the Unsloth attribution that
unsloth_branding.py enforces is also a written license condition, not only a
build check. docker/NOTICE designates the attribution (the "Built by the Unsloth
team" label, the copyright line, the license notice, the logo and theme, and the
Help > About links) as required Appropriate Legal Notices under AGPLv3 Section
7(b), referencing /studio/LICENSE.AGPL-3.0 and /LICENSE. BRANDING.md is a
human-readable note next to the guard describing what must stay, where it lives
and how it is enforced.
* ci(studio-backend): restore docker/** trigger path
The docker/** pull_request path added in b558bc7d was dropped by a later
rebase, so the "Docker JupyterLab/notebook feature validation" step (which runs
tests/validate_studio_features.py against docker/jupyter branding and notebook
helpers) no longer ran on PRs that only touch docker/. Re-add docker/** so a
docker-only change is validated on the PR rather than only after merge to main.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sync the docker image branch with main (15 commits) so the branch's Python
files carry main's current formatting and pre-commit.ci runs cleanly on a
non-stale checkout.
Bump the base image torch triplet to torch==2.11.0 / torchvision==0.26.0 /
torchaudio==2.11.0 and the paired torchcodec to 0.11.0, and hold torch at
2.11.0 during the vLLM resolve so uv lands on the vLLM 0.20+ line that pins
torch 2.11.0 (the split-install rationale already anticipated the bump). Update
the build-time self-test assertion, its status line, and the test_locally.sh
log grep to match, plus the FA2 wheel note.
Also clarify the advertised architecture support: forward-compatible SASS
covers precompiled kernels on sm_103 (B300/GB300), but runtime Triton/NVRTC JIT
targets the actual device cap and the bundled cu12.8 ptxas/NVRTC cannot emit
compute_103. arm64 sm_121 is handled by the cu13 NVRTC/ptxas override; amd64
sm_103 has no cu13 override yet, so JIT-heavy paths there can fail until it
lands. Precompiled SASS still runs on sm_103 via sm_100 forward-compat.
The bundled launcher only forwarded HF/W&B/license/CPU vars, so the documented
Studio service config read by studio_launch.sh was silently dropped when running
the full image through this wrapper: JUPYTER_PASSWORD fell back to a random
password, PUBLIC_KEY/SSH_KEY never enabled sshd, and UNSLOTH_JUPYTER_CLOUDFLARE
never started the tunnel. Forward them with the same dash-only -e VAR form as the
secrets above, so the value is read from the parent env and never lands in argv.
The zoo_ref prepare step emitted the bare branch name (main) on the normal
push/schedule path, and both arch matrix legs plus the Studio build pass that
to pip install unsloth-zoo @ git+...@REF. If unsloth-zoo advanced mid-build a
single multi-arch tag could bake different zoo code across architectures or
between the base and Studio venvs. Resolve a branch/tag to its current sha via
ls-remote here (mirroring the notebooks step), so the whole matrix pins one
immutable commit. A 40-char sha input stays frozen; a lookup miss falls back to
the bare ref so the build can still fetch by name.
* Note the bundled flash-linear-attention kernels for gated-deltanet models
Unsloth Zoo now bundles the flash-linear-attention (fla) gated-delta Triton
kernels and injects them automatically, so gated-deltanet models (Qwen3-Next,
Qwen3.5, Kimi-Linear) get the fast path with no pip install. Replace the old
install advisory with a one-time note that fires only when the bundled kernels
could not be enabled on the current setup (no CUDA, or torch < 2.7 / triton < 3.3),
i.e. exactly when transformers falls back to the slow pure PyTorch path.
* Tighten comments
* Normalize model_types in fla install advisory for None and single string
* Cover olmo_hybrid in the gated-deltanet fla advisory
* Add gemma4, glm4_moe and qwen3_moe to the FORCE_FLOAT32 fallback list
Keeps the fallback list (used only if the unsloth_zoo import fails) in sync with
unsloth_zoo/model_lists.py, which now force-float32s these MoE archs so a float16
request loads bf16 and trains finite instead of NaNing the grad_norm.
* Union FORCE_FLOAT32 fallback so new archs force float32 with older unsloth_zoo
A model path ending in -bf16 unconditionally forced 16-bit loading, so a
LOCAL checkpoint directory whose name happens to end in -bf16 could never be
loaded in 4-bit, 8-bit or fp8: the suffix rule silently overrode the caller's
quantization flags. Hub repo ids keep the existing behavior (the suffix is a
publishing convention there), but for a local directory (expanduser-aware, so
tilde paths are detected too) the requested quantization is preserved unless
the caller explicitly passes load_in_16bit=True.
* Auto-enable grouped MoE on loaded / PEFT'd models via loader hook
Wraps the FastLlamaModel and FastBaseModel from_pretrained / get_peft_model leaves with wrap_loader_for_grouped_moe so the grouped-GEMM MoE forward is installed on the live instance after the model and its compiled module are built. Gated by UNSLOTH_MOE_GROUPED and wrapped in try/except, so it is a no-op when the unsloth_zoo module is absent or no eligible MoE block exists.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Install grouped-MoE loader wrappers before PatchFastRL
* Re-evaluate grouped MoE after loading a PEFT adapter
When loading an existing adapter through FastLanguageModel.from_pretrained,
the base model is evaluated for grouped MoE when the wrapped from_pretrained
leaf returns, but the adapter is attached afterwards via PeftModel and
patch_peft_model. Re-run auto_enable_grouped_moe on the final model so
blocks whose experts gained LoRA are restored to the original loop,
attention-only adapters keep the grouped path on their frozen experts, and
recompute is re-derived from the final gradient-checkpointing state. Guarded
so it never blocks adapter loading.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments in the grouped MoE loader hooks
Shorten the loader re-eval and llama.py wrapper comments; code is unchanged
(verified comment-only).
* Re-evaluate grouped MoE after loading a PEFT adapter on the vision path
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Honor an explicit sdpa or flex_attention request when flash is disabled
When flash attention is disabled for a model, the fallback selection could
downgrade a caller who explicitly passed attn_implementation='sdpa' or
'flex_attention' to a different backend, because the disable reason is
flash-specific. Keep an explicit non-flash request as-is; flash requests
still fall back as before.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments
* Gate honor-explicit attention on provenance and flex support
Only honor an explicit non-flash attention request when it comes from the
caller argument, not from a config value the loaders synthesize (the language
path seeds attn_implementation=sdpa). Honor explicit flex_attention only when
supports_flex_attention is True so excluded/broken configs (e.g. gpt_oss) fall
back instead of selecting a known-broken backend. Explicit sdpa stays honored.
* Honor explicit sdpa through the resolver guard
* Keep SDPA exclusions when honoring an explicit sdpa request
An explicit attn_implementation="sdpa" was re-enabling sdpa for models in
_SDPA_EXCLUDED_MODELS (e.g. gpt_oss) where sdpa is known-broken: the helper
honored the request and the resolver's final not-supports_sdpa guard skipped
the eager downgrade for any explicit request. Honor an explicit sdpa only when
the model is not sdpa-excluded, mirroring the flex guard that already falls
back for _FLEX_EXCLUDED_MODELS via supports_flex_attention. Conservative
supports_sdpa=False (large head dim / attention-sink models) still honors an
explicit sdpa; a synthesized/default sdpa still downgrades to eager.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor DISABLE_SDPA_MODEL_NAMES when honoring explicit sdpa
The honor-explicit-sdpa guard only skipped the sdpa->eager downgrade for models
in _SDPA_EXCLUDED_MODELS (gpt_oss). Gemma3/Gemma3Text disable SDPA through the
loader's DISABLE_SDPA_MODEL_NAMES (their bundled SDPA modules are wrong), so an
explicit sdpa request bypassed the downgrade and re-enabled a known-wrong path.
Extend _is_sdpa_excluded to also treat DISABLE_SDPA_MODEL_NAMES membership as
excluded, replicating the loader's trailing-comma substring match so gemma3 and
gemma3_text match but gemma3n does not. Move the constant into _utils.py (single
source of truth, re-exported from loader.py) to avoid a loader -> _utils cycle.
Conservative supports_sdpa=False models not in either list still honor explicit
sdpa.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Scope MoE expert LoRA detection to actual MLP projection targets
_moe_target_set_from_string treated any regex containing the substring mlp
or ffn as targeting the expert MLP projections. Unsloth's auto-generated
attention-only regex lists mlp, ffn and feed_forward as allowed intermediate
path segments while its final group matches only q_proj/k_proj/v_proj/o_proj,
so attention-only finetuning on MoE models silently enabled expert LoRA as
well: the experts were trained and every MoE layer paid the extra expert LoRA
grouped matmuls. Detect expert intent from the projection names themselves
(gate_proj/up_proj/down_proj/gate_up_proj) instead of the mlp substring.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten comments
* Detect MoE expert LoRA via mlp path segment, not proj names
The auto-generated target regex always lists every projection leaf
(q/k/v/o and gate/up/down), so keying detection on a proj name mis-fired:
it enabled expert LoRA for attention-only regexes and dropped the
mlp/ffn path regexes. Key on the mlp/ffn/feed_forward/experts path
segment instead, which is present only when the MLP/experts are actually
targeted. Add a regression test for the attention-only case.
* Scope expert LoRA targets to the leaves a regex names
An mlp path alternative with attention-only leaves, for example
(mlp|self_attn).(q_proj|o_proj), no longer enables expert LoRA, and a
regex naming a single expert leaf such as .*experts.*down_proj now
targets only that projection instead of the whole broad set. Generic
mlp projections (.*mlp.*proj) and the auto regex mlp tag block keep the
broad set for fused-expert models whose leaves are plain Parameters.
* Route explicit leaf list into MoE expert detection
An attention-only explicit target_modules list routed through get_peft_regex
for family scoping (e.g. FastVisionModel with vision layers off) yields a
regex carrying the full mlp|feed_forward|ffn|dense component block even though
its leaf group only names q/k/v/o_proj. Keying expert detection on that regex
trained the experts for a language-only/attention-only request. Use the
caller's original leaf list for detection; only the auto path uses the regex,
where the mlp block is the sole MLP-intent signal on fused-expert models.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Respect finetune_mlp_modules and finetune_language_layers scope for MoE expert detection
When an explicit leaf list that names MLP projections (gate_proj/up_proj/down_proj)
is routed through get_peft_regex under finetune_mlp_modules=False, the scoped regex
correctly drops the MLP leaves, but MoE expert detection was still keyed on the
original list and re-added mlp.experts.* via target_parameters, training the experts
the caller had frozen. Same gap for finetune_language_layers=False on vision-only runs.
Prefer the original list only when MLP and language families are both in scope
(preserving the attention-only fix); otherwise honor the scoped result so the frozen
family is respected. Factored the choice into _select_moe_detection_targets with unit
tests over the full selection matrix.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Handle odd shapes and non-float scales in FP8BlockQuantLinear
Small fp8 checkpoints (e.g. tiny test models) break the block-quantized
linear in three ways: weight scales stored in a float8 dtype such as
float8_e8m0fnu have no triton dtype mapping; activations whose hidden dim is
not a multiple of the activation quant block fail act_quant's divisibility
assert; and weights whose dims are not multiples of the weight block cannot
be tiled by the triton dequant kernel.
Cast non-float scales to float32 on entry, and when the hidden dim does not
divide into the activation block, dequantize the weight and run a plain
matmul instead of the fp8 block matmul. The dequant goes through a new
shape-safe helper that falls back to a torch-native scale expansion when the
weight does not tile evenly; backward uses the same helper so the gradient
path works for every shape the forward accepts. Full-size checkpoints are
unaffected.
* Add tiny / e8m0 fp8 block-quant regression test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix FP8 block-quant fallback: real block size in dequant and scalar-scale fast path
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Route rectangular fp8 blocks through torch dequant and keep block_size across e8m0 upcast
The triton weight_dequant kernel uses one BLOCK_SIZE for both axes, so
rectangular blocks (block_size[0] != block_size[1]) mis-index the column
scale and corrupt grad_X. Route those through the torch scale expansion,
which handles each dimension independently, and keep the triton path for
square blocks only.
Also preserve a block_size attribute carried on the scale tensor across the
e8m0 -> float32 upcast so the later lookup no longer falls back to [128, 128].
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* GRPO: optional sequence packing for the no-grad old/ref logp path
Add an opt-in sequence-packing fast path to _get_per_token_logps_and_entropies, enabled with
UNSLOTH_GRPO_SEQ_PACKING=1. When the batch is text-only, the padded [B, Lmax] per-chunk forward is
replaced by a single varlen [1, sum L] forward (BlockDiagonalCausalMask via packed_seq_lengths with
reset position_ids). Per-token logps use the same float32 chunked_hidden_states_selective_log_softmax
as the padded path, so the old and reference logps are bit-for-bit identical.
Safety: the packed path is self-verified once against the padded ground truth on a batch that has at
least two rows with real completion tokens (self._unsloth_seq_packing_nograd_ok), so cross-sample
contamination would actually manifest; a degenerate all-pad / fully tool-masked batch leaves the
verdict unset and re-verifies later. If a backend silently ignores packed_seq_lengths (flat batch run
under a normal causal mask, samples leaking across boundaries), the packed logps will not match and
packing is disabled instead of corrupting logps. It also forces use_cache=False (a populated
past_key_value disables varlen packing), skips packing when a sliding window is shorter than the
packed stream, runs the same GPT-OSS offload device_synchronize the padded loop uses, and falls back
on any exception (UNSLOTH_GRPO_SEQ_PACKING_DEBUG=1 prints the reason).
Default off, so existing behavior is unchanged. Pairs with the matching gradient-path change in
unsloth_zoo so the full GRPO logp + loss + backward can run packed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO no-grad packing: address review feedback
- Cache the packed-vs-padded verdict per unwrapped model instead of on the
trainer, so a separately forwarded reference model is verified on its own
forward path rather than inheriting the policy model's verdict.
- Force the padded path when token_type_ids or mm_token_type_ids are present,
matching the extra vision kwargs the padded loop forwards.
- Require the xformers varlen backend before packing. Without it the packed
mask falls back to a dense O(T^2) SDPA mask that can OOM on the flattened
batch, so we keep the padded loop in that case.
- On any packed-forward failure (missing backend, OOM, unsupported forward)
empty the cache on OOM, disable packing for that model, and fall back to the
chunked padded loop instead of retrying every step.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO no-grad packing: default-on, verify against per-row reference
Redesign of the optional sequence-packing fast path for the no-grad
old/ref logprob recompute, after establishing that the packed forward is
the exact per-row computation and the padded batch forward is the side
that mis-positions left-padded rows on long completions.
- Default the packing on (UNSLOTH_GRPO_SEQ_PACKING, disable with 0).
- Verify the packed logprobs against the per-row clean forward (each
row's real tokens alone, reset 0-based positions, no padding), not the
padded batch which is itself wrong for left-padding. Cross-sample
contamination (a backend ignoring packed_seq_lengths) shows up as a
large mismatch and falls back to the padded loop.
- Make the trust decision shape and RoPE aware: re-verify whenever the
packed total length or the longest segment grows past what was
verified, so a later batch crossing a LongRoPE short/long cache
boundary is re-checked instead of trusted blindly.
- Run lm_head only on completion-prediction positions instead of every
packed prompt token, so long-prompt/short-completion batches do not
pay for projecting the whole packed prompt.
- Drop the hard xformers import so the path also runs in
FlashAttention-only environments; the per-row verification guards
correctness regardless of backend.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO no-grad packing: disable entirely on cross-sample mismatch
When the per-row verification fails, distinguish the two failure modes by
magnitude instead of by sequence length:
- A large mismatch (>= 1.5) is the cross-sample contamination signature:
the model's attention does not honor the block-diagonal packed mask
(seen on some MoE / custom-attention models, e.g. qwen2_moe). Disable
packing entirely for the model so later batches do not pay the
verification cost again.
- A moderate mismatch is more likely a length-boundary effect (a LongRoPE
short/long cache switch): keep marking just that length region unsafe so
packing still runs for smaller shapes.
Validated: Qwen1.5-MoE falls back after a single verification (grad and
no-grad ok flags go False, no re-verify on later steps); dense Llama-3.2
and Qwen3 still verify and engage packing.
* GRPO no-grad packing: trim comments to be concise
* GRPO no-grad packing: fix per-row completion boundary for left-padded rows
The completion-target selection used a single global boundary
(col >= L - logits_to_keep). After left-packing, each row's completion
starts at (L - logits_to_keep) - left_pad[row], so for left-padded rows
the first left_pad completion tokens fall below the global boundary and
were dropped, leaving 0 logprobs at real completion positions that the
loss mask keeps. Use the per-row boundary so packed coverage matches
create_completion_attention_mask exactly, and widen the self-verify mask
to the full per-row completion region so it can catch coverage gaps.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO no-grad packing: gate verification on real completion rows
Count active rows via create_completion_attention_mask (the same mask the
loss uses) instead of any non-pad token in the packed window. Prompt-only
rows carry prompt-overflow tokens in the window and could otherwise satisfy
the >= 2 verification guard, letting a batch with a single real completion
row cache a trust decision. This matches the gradient path, which already
gates on the completion mask. The same mask is reused for the self-verify
comparison.
* GRPO no-grad packing: gate debug logging on UNSLOTH_ENABLE_LOGGING
Use the shared UNSLOTH_ENABLE_LOGGING global (import_fixes, re-exported by
_utils) instead of a bespoke UNSLOTH_GRPO_SEQ_PACKING_DEBUG env var for the
packing debug prints, matching the rest of the codebase.
* GRPO packing: import UNSLOTH_ENABLE_LOGGING inside the injected logp function
_get_per_token_logps_and_entropies is copied verbatim into the generated GRPO trainer
via inspect.getsource, and that module never imported UNSLOTH_ENABLE_LOGGING, so the
default-on packing verify path raised NameError (and the except handler re-raised it).
Import the flag locally, before the try, so the name is defined in the generated module
too. Drop it from the now-unused module-level import.
* GRPO no-grad packing: harden unsafe-length skip, verify guard, fallback cleanup
Three fixes to the no-grad logp packing path, mirroring the grad path:
- skip the packed forward for known-unsafe lengths by reading unsafe_T and
gating on it before the forward, instead of running the full packed pass and
the result build only to discard them (wastes a pass, can OOM at large T)
- only widen the verified T/seg envelope when >= 2 completion rows actually
exercised cross-sample packing; a < 2 row batch cannot expose leakage, so it
must not extend the trusted shape that later multi-row batches skip verify for
- drop the packed intermediates (hidden/sel/result/ref) before the padded
fallback loop so it does not run with the flattened hidden state still resident
* GRPO no-grad packing: cap the flattened forward at one mini-batch budget
The packed path built a single [1, sum L] forward over every row before any
size check, so a large batch could exceed the memory the padded path bounds
per mini-batch. Gate packing on _pk_T <= _pk_cap (B * seq_len, one padded
mini-batch's token budget); larger batches fall back to the chunked padded
loop.
* GRPO no-grad packing: disable unless unsloth_zoo has the masked-column guard
The packed path leaves masked prompt/pad logprob columns at 0, which only stays
finite if unsloth_zoo grpo_compute_loss zeroes them before exp() (zoo#840). An
older unsloth_zoo without that guard would NaN. Detect the guard once (cached on
the model) via inspect.getsource and gate packing on it, so #6738 is safe with
any unsloth_zoo version and re-enables packing automatically once a guarded zoo
is installed, independent of the pinned lower bound.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO packing: hoist env gates and zoo-guard detection to one-time module checks
Read UNSLOTH_GRPO_SEQ_PACKING and detect the unsloth_zoo masked-column guard once at
import time (module constants plus RL_PRE_ITEMS for the generated trainer cache) instead
of per call, and drop the in-function UNSLOTH_ENABLE_LOGGING import for a module-top one.
The UNSLOTH_GRPO_SEQ_PACKING_VERIFY force-verify debug knob is commented out, kept in
place for hand re-enable; the first-use and envelope-growth self-verify stays active.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO packing: cap the flattened forward by the padded chunk rows
B counts chunks at this point, so B * seq_len understated (small runs) or overstated
(large runs) the padded mini-batch token budget; use batch_size * seq_len, the rows the
padded loop actually forwards per chunk.
* Add PrefixGrouper for GRPO: dedup the shared prompt across a group's completions
In GRPO every prompt spawns G=num_generations completions that share the prompt
prefix, so the trunk logprob forward re-encodes that prefix G times. PrefixGrouper
stores the prefix once and concatenates only the G suffixes behind a FlexAttention
shared-prefix mask, cutting the forward from G*(P+R) to P+G*R tokens across both the
no-grad old/ref forwards and the grad logp forward. Default off behind the
UNSLOTH_GRPO_PREFIX_GROUPER env gate, so the gate-unset path is byte-identical to
today. A tok_r auto-gate and a first-use self-verify (fall back and mark the shape
unsafe on mismatch) keep it from ever shipping wrong logprobs silently.
Wired for llama, mistral, qwen3, gemma2, cohere, granite and falcon_h1, plus qwen2
and gemma through the shared LlamaAttention_fast_forward. Stacked on the GRPO
sequence-packing PR (#6738); the grad path lands in a companion unsloth-zoo PR.
Also fixes a latent UNSLOTH_ENABLE_LOGGING NameError in the seq-packing no-grad
verify path by defining the name as a generated-cache pre-item.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* PrefixGrouper: enforce the sliding-window cap, gate softcap models, bound the mask cache
Add a max_segment_cap kwarg to build_group_layout so it falls back when a group's
span (prefix + longest suffix) exceeds the model's local window, and pass the config
sliding_window into the no-grad engage gate the same way the packed _pk guard derives it.
Skip PrefixGrouper entirely for attn_logit_softcapping models, since the FlexAttention
kernel never applies logit softcapping. Bound _BLOCK_MASK_CACHE to a FIFO of 8 so
per-step lengths cannot pin BlockMasks forever, release the PG hidden before the verify
forward, and align the UNSLOTH_ENABLE_LOGGING pre-item truthiness with the canonical form.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* PrefixGrouper: vectorize the real-column scan in build_group_layout
Replace the per-row O(B*L) Python scan of the keep mask with a GPU-derived
contiguous-run fast path (first real column + count per row), keeping the
general scan only as a fallback for non-contiguous rows. Works for both call
sites: the no-grad layout (left-padded prompt + right-padded completion, run
does not start at column 0) and the grad layout (left-packed).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* PrefixGrouper: hoist the gate and kernel imports to one-time module checks, AGPLv3 headers
Read UNSLOTH_GRPO_PREFIX_GROUPER and resolve the prefix_grouper imports once at module
level (source constants plus an RL_PRE_ITEMS entry for the generated trainer cache)
instead of per call, matching the sequence-packing gates. The prefix_grouper env helpers
become one-time module reads with unchanged signatures, and attention_dispatch resolves
the FlexAttention kernel once behind the same gate (lazy fallback kept). The two new
prefix_grouper files move to AGPLv3 headers.
* PrefixGrouper: length-envelope trust and hybrid SSM exclusion
Verified signatures now record (max T, max segment) and re-verify when either grows,
matching the packed path's envelope. Hybrid SSM models (FalconH1 etc.) are excluded at
the gate since only attention gets the shared-prefix isolation, and the FalconH1 wiring
is removed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* PrefixGrouper: defer the unverified no-grad forward until the packed reference exists
Unverified shapes no longer run the whole-batch shared-prefix forward up front; it now
runs at the verify site, only when the packed path produced a reference. A declined
packed path (budget, window) therefore costs no wasted PG forward per step. Trusted
shapes still run it first to skip the full-row forward, with the same fallback.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* PrefixGrouper: disable under vLLM (fast_inference=True)
With colocated vLLM generation the rollout dominates the GRPO step, so the shared-prefix
training forward saves little end-to-end and its first-use self-verify (which also runs
the full-row path) is net overhead. Gate PG on not use_vllm so it only engages on the raw
transformers path, where the training forward is on the critical path. Packing is unaffected.
* PrefixGrouper: compile the FlexAttention kernel with dynamic shapes
GRPO changes the packed length T almost every batch. With dynamic=False the flex
forward+backward kernel recompiled on every new T (~14s each on a 4B trunk), which
dominated the step and made PG a net loss. dynamic=True compiles once, then reuses the
kernel across all lengths recompile-free (a new shape drops from ~14s to ~1.4ms after a
two-graph warmup). T is still padded to a multiple of 128 for the backward block assertion.
* PrefixGrouper: default on
Enable PrefixGrouper by default (UNSLOTH_GRPO_PREFIX_GROUPER defaults to 1; set 0 to
disable). Still auto-disabled under vLLM (fast_inference=True) and by the arch/softcap/
SSM/tok_r gates, and the first-use self-verify falls back on any mismatch, so this is a
memory-first default on the raw-transformers path with no correctness risk.
* GRPO PrefixGrouper: gate on zoo masked-column guard and exclude MoE
- Require the zoo masked-column guard (zoo#840) before PrefixGrouper can engage.
PG rides the sequence-packing path, so when the first-step self-verify is off the
fast path trusts PG output directly; without the guard those masked columns feed
NaN into the packed loss. Gate PG on the same UNSLOTH_ZOO_HAS_MASKED_COL_GUARD
the packing path already checks.
- Exclude MoE configs (num_experts, num_local_experts, n_routed_experts,
moe_intermediate_size) alongside the hybrid-SSM markers. Only the threaded
attention forwards carry the shared-prefix isolation, so a MoE decoder that does
not forward prefix_seg_info would let suffixes leak across completions.
- Refresh the stale default-off comments now that UNSLOTH_GRPO_PREFIX_GROUPER is
on by default.
* GRPO PrefixGrouper: import chunked_hidden_states_selective_log_softmax
The shared-prefix forward passes chunked_hidden_states_selective_log_softmax
into extract_logps, but the name was only ever provided by the generated
trainer cache (rl.py injects grpo_selective_log_softmax_code), never bound in
this module. Import it from unsloth_zoo.rl_replacements next to its sibling
chunked_selective_log_softmax so the source resolves the name in every scope
(the new _pg_run_forward closure included). No runtime change: the cache still
defines the function via template injection.
* GRPO PrefixGrouper: dropout gate, device-safe layout, Mistral mask skip
Addresses three review findings on the shared-prefix path:
- Skip PrefixGrouper when the model sets a nonzero attention_dropout. The normal
backends apply config.attention_dropout while training (e.g. Granite dense
flash/sdpa/xformers), but the FlexAttention shared-prefix path is deterministic,
so gate PG off for those configs rather than train on mismatched activations.
- Move the shared-prefix mask labels to the consumer (Q) device in get_block_mask
and the target index maps to hidden.device in extract_logps, mirroring the packed
path moving its metadata to the consumer device. Prevents cross-device indexing
when the model is sharded across GPUs.
- Do not synthesize a causal attention_mask in the Mistral forward when
prefix_seg_info is present. On the no-xFormers path that synthetic mask tripped
resolve_prefix_seg_info and forced PG to always fall back to the packed forward.
* GRPO sequence packing: tighten comments
* GRPO PrefixGrouper: tighten comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO PrefixGrouper: persistent disable on runtime failure; build block-mask labels with inference mode disabled
- rl_replacements: on a PG forward exception (FlexAttention/Triton compile failure or OOM), set a model-level _unsloth_prefix_grouper_nograd_disabled flag and consult it in the engage gate, mirroring the seq-packing handler, so a GPU-wide failure is not retried and re-paid every step.
- prefix_grouper_kernel: move the .to(device) label copies inside the inference_mode(False) block so a cross-device (model-parallel shard) first build does not capture inference tensors, which otherwise cannot be saved for backward when the grad training forward reuses the cached BlockMask.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* GRPO: optional sequence packing for the no-grad old/ref logp path
Add an opt-in sequence-packing fast path to _get_per_token_logps_and_entropies, enabled with
UNSLOTH_GRPO_SEQ_PACKING=1. When the batch is text-only, the padded [B, Lmax] per-chunk forward is
replaced by a single varlen [1, sum L] forward (BlockDiagonalCausalMask via packed_seq_lengths with
reset position_ids). Per-token logps use the same float32 chunked_hidden_states_selective_log_softmax
as the padded path, so the old and reference logps are bit-for-bit identical.
Safety: the packed path is self-verified once against the padded ground truth on a batch that has at
least two rows with real completion tokens (self._unsloth_seq_packing_nograd_ok), so cross-sample
contamination would actually manifest; a degenerate all-pad / fully tool-masked batch leaves the
verdict unset and re-verifies later. If a backend silently ignores packed_seq_lengths (flat batch run
under a normal causal mask, samples leaking across boundaries), the packed logps will not match and
packing is disabled instead of corrupting logps. It also forces use_cache=False (a populated
past_key_value disables varlen packing), skips packing when a sliding window is shorter than the
packed stream, runs the same GPT-OSS offload device_synchronize the padded loop uses, and falls back
on any exception (UNSLOTH_GRPO_SEQ_PACKING_DEBUG=1 prints the reason).
Default off, so existing behavior is unchanged. Pairs with the matching gradient-path change in
unsloth_zoo so the full GRPO logp + loss + backward can run packed.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO no-grad packing: address review feedback
- Cache the packed-vs-padded verdict per unwrapped model instead of on the
trainer, so a separately forwarded reference model is verified on its own
forward path rather than inheriting the policy model's verdict.
- Force the padded path when token_type_ids or mm_token_type_ids are present,
matching the extra vision kwargs the padded loop forwards.
- Require the xformers varlen backend before packing. Without it the packed
mask falls back to a dense O(T^2) SDPA mask that can OOM on the flattened
batch, so we keep the padded loop in that case.
- On any packed-forward failure (missing backend, OOM, unsupported forward)
empty the cache on OOM, disable packing for that model, and fall back to the
chunked padded loop instead of retrying every step.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO no-grad packing: default-on, verify against per-row reference
Redesign of the optional sequence-packing fast path for the no-grad
old/ref logprob recompute, after establishing that the packed forward is
the exact per-row computation and the padded batch forward is the side
that mis-positions left-padded rows on long completions.
- Default the packing on (UNSLOTH_GRPO_SEQ_PACKING, disable with 0).
- Verify the packed logprobs against the per-row clean forward (each
row's real tokens alone, reset 0-based positions, no padding), not the
padded batch which is itself wrong for left-padding. Cross-sample
contamination (a backend ignoring packed_seq_lengths) shows up as a
large mismatch and falls back to the padded loop.
- Make the trust decision shape and RoPE aware: re-verify whenever the
packed total length or the longest segment grows past what was
verified, so a later batch crossing a LongRoPE short/long cache
boundary is re-checked instead of trusted blindly.
- Run lm_head only on completion-prediction positions instead of every
packed prompt token, so long-prompt/short-completion batches do not
pay for projecting the whole packed prompt.
- Drop the hard xformers import so the path also runs in
FlashAttention-only environments; the per-row verification guards
correctness regardless of backend.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO no-grad packing: disable entirely on cross-sample mismatch
When the per-row verification fails, distinguish the two failure modes by
magnitude instead of by sequence length:
- A large mismatch (>= 1.5) is the cross-sample contamination signature:
the model's attention does not honor the block-diagonal packed mask
(seen on some MoE / custom-attention models, e.g. qwen2_moe). Disable
packing entirely for the model so later batches do not pay the
verification cost again.
- A moderate mismatch is more likely a length-boundary effect (a LongRoPE
short/long cache switch): keep marking just that length region unsafe so
packing still runs for smaller shapes.
Validated: Qwen1.5-MoE falls back after a single verification (grad and
no-grad ok flags go False, no re-verify on later steps); dense Llama-3.2
and Qwen3 still verify and engage packing.
* GRPO no-grad packing: trim comments to be concise
* GRPO no-grad packing: fix per-row completion boundary for left-padded rows
The completion-target selection used a single global boundary
(col >= L - logits_to_keep). After left-packing, each row's completion
starts at (L - logits_to_keep) - left_pad[row], so for left-padded rows
the first left_pad completion tokens fall below the global boundary and
were dropped, leaving 0 logprobs at real completion positions that the
loss mask keeps. Use the per-row boundary so packed coverage matches
create_completion_attention_mask exactly, and widen the self-verify mask
to the full per-row completion region so it can catch coverage gaps.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO no-grad packing: gate verification on real completion rows
Count active rows via create_completion_attention_mask (the same mask the
loss uses) instead of any non-pad token in the packed window. Prompt-only
rows carry prompt-overflow tokens in the window and could otherwise satisfy
the >= 2 verification guard, letting a batch with a single real completion
row cache a trust decision. This matches the gradient path, which already
gates on the completion mask. The same mask is reused for the self-verify
comparison.
* GRPO no-grad packing: gate debug logging on UNSLOTH_ENABLE_LOGGING
Use the shared UNSLOTH_ENABLE_LOGGING global (import_fixes, re-exported by
_utils) instead of a bespoke UNSLOTH_GRPO_SEQ_PACKING_DEBUG env var for the
packing debug prints, matching the rest of the codebase.
* GRPO packing: import UNSLOTH_ENABLE_LOGGING inside the injected logp function
_get_per_token_logps_and_entropies is copied verbatim into the generated GRPO trainer
via inspect.getsource, and that module never imported UNSLOTH_ENABLE_LOGGING, so the
default-on packing verify path raised NameError (and the except handler re-raised it).
Import the flag locally, before the try, so the name is defined in the generated module
too. Drop it from the now-unused module-level import.
* GRPO no-grad packing: harden unsafe-length skip, verify guard, fallback cleanup
Three fixes to the no-grad logp packing path, mirroring the grad path:
- skip the packed forward for known-unsafe lengths by reading unsafe_T and
gating on it before the forward, instead of running the full packed pass and
the result build only to discard them (wastes a pass, can OOM at large T)
- only widen the verified T/seg envelope when >= 2 completion rows actually
exercised cross-sample packing; a < 2 row batch cannot expose leakage, so it
must not extend the trusted shape that later multi-row batches skip verify for
- drop the packed intermediates (hidden/sel/result/ref) before the padded
fallback loop so it does not run with the flattened hidden state still resident
* GRPO no-grad packing: cap the flattened forward at one mini-batch budget
The packed path built a single [1, sum L] forward over every row before any
size check, so a large batch could exceed the memory the padded path bounds
per mini-batch. Gate packing on _pk_T <= _pk_cap (B * seq_len, one padded
mini-batch's token budget); larger batches fall back to the chunked padded
loop.
* GRPO no-grad packing: disable unless unsloth_zoo has the masked-column guard
The packed path leaves masked prompt/pad logprob columns at 0, which only stays
finite if unsloth_zoo grpo_compute_loss zeroes them before exp() (zoo#840). An
older unsloth_zoo without that guard would NaN. Detect the guard once (cached on
the model) via inspect.getsource and gate packing on it, so #6738 is safe with
any unsloth_zoo version and re-enables packing automatically once a guarded zoo
is installed, independent of the pinned lower bound.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO packing: hoist env gates and zoo-guard detection to one-time module checks
Read UNSLOTH_GRPO_SEQ_PACKING and detect the unsloth_zoo masked-column guard once at
import time (module constants plus RL_PRE_ITEMS for the generated trainer cache) instead
of per call, and drop the in-function UNSLOTH_ENABLE_LOGGING import for a module-top one.
The UNSLOTH_GRPO_SEQ_PACKING_VERIFY force-verify debug knob is commented out, kept in
place for hand re-enable; the first-use and envelope-growth self-verify stays active.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* GRPO packing: cap the flattened forward by the padded chunk rows
B counts chunks at this point, so B * seq_len understated (small runs) or overstated
(large runs) the padded mini-batch token budget; use batch_size * seq_len, the rows the
padded loop actually forwards per chunk.
* GRPO sequence packing: tighten comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
The setup.ps1 unit-tests job intermittently fails on the windows-latest
runner with 'No repository with the name PSGallery was found.' when the
default PowerShell Gallery is not registered, so Set-PSRepository throws
before Pester can be installed. Register the default gallery first when it
is missing, then set its policy and install Pester as before.
pip shim (docker/unsloth_pip_shim.py):
- Drop protected packages named via a VCS/URL #egg=NAME fragment so
git+... #egg=torch no longer reinstalls into the baked venv.
- Filter constraint files (-c/--constraint) through the same protected
package filter as requirement files, so a pinned torch/transformers in
a constraint cannot downgrade the baked stack during resolution.
- Recursively filter nested -r/-c includes and absolutise their paths so
the filtered /tmp copy still resolves them and no protected spec deep in
the include tree slips past the keep list.
- Remove an unused subprocess import.
Notebook environment:
- Scope the transformers-request marker per kernel (UNSLOTH_NB_TF_MARKER
keyed on the kernel connection-file id) so concurrent notebooks no
longer read each other's pin.
- Install the IPython startup hook under IPYTHONDIR (set via ENV) so it
loads for any uid, including docker run --user, not just root.
- unsloth_nb_content_sig.py: only treat a %%capture / %%bash cell as
install boilerplate when it carries an install command, so substantive
captured/bash cells are hashed and upstream changes are not skipped.
- unsloth_run.py: clean up the temp dir used to materialise a downloaded
notebook.
- unsloth_sync_notebooks.sh: honor UNSLOTH_KEEP_DELETED_NOTEBOOKS across
GitHub refreshes so a deleted notebook is not restored when upstream
advances.
install_python_stack.py: the --local unsloth-zoo overlay now honors
UNSLOTH_ZOO_REF (default main), matching the install.sh overlay.
synthetic.py: preserve the timeout=None unbounded vLLM startup wait
instead of coercing it to 1200s.
* Fix export-time trust_remote_code bypass in FP8/INT8/GGUF-LoRA export
The torchao, compressed-tensors, and LoRA GGUF export paths re-read the merged
checkpoint and used to set trust_remote_code from the checkpoint config's static
auto_map (the torchao path also scanned the staged tokenizer/processor configs).
A model that loads with built-in Transformers classes can carry an auto_map entry,
which skips the load-time remote-code consent scan (that only runs when the load
already requested trust_remote_code) yet flips trust_remote_code on at export,
running unvetted custom code.
Derive the reload trust_remote_code from the approved load decision instead: a new
_loaded_via_remote_code() checks whether the in-memory model / tokenizer was itself
loaded from custom code (its class lives in the transformers_modules package),
walking PEFT / wrapper layers. Built-in-loaded models no longer gain trust from
config metadata; genuine custom-code models (loaded with consent) still reload
correctly. Add CPU-only regression tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden _loaded_via_remote_code against a None/missing __module__
Read type(node).__module__ via getattr and require a string before startswith,
so a dynamically created or C-extension class with a None module does not raise
during export. Add a regression test.
* Split model and tokenizer trust for the compressed subprocess, walk processor components
The compressed-tensors export collapsed model and tokenizer trust into
one --trust-remote-code flag, so an approved custom tokenizer would have
let an unapproved model's custom code run inside the quantization
subprocess. The subprocess now takes --trust-remote-code-tokenizer for
the processor load and keeps --trust-remote-code for the model loads,
matching the torchao path's separate model_trust / tok_trust.
_loaded_via_remote_code now also walks processor components (tokenizer,
image_processor, feature_extractor, video_processor), so an approved
custom tokenizer held inside a built-in ProcessorMixin keeps its trust
on the export reload instead of failing with trust_remote_code=False.
The walk is a bounded BFS with a seen set so wrapper cycles terminate.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix: skip fp16/bf16 validation for full finetuning in RL trainers
When doing full finetuning (FFT) of a bfloat16 model, the fp16/bf16
mismatch validation fires before the corrective logic runs, causing a
misleading error even though the code would properly handle it downstream.
Skip the validation when full_finetuning is active.
Fixes#6731
* Fix: auto-correct fp16/bf16 mismatches for full finetuning before validation
Instead of entirely skipping validation (which could let mismatches
through when mixed_precision_dtype is float32), auto-correct explicit
fp16/bf16 settings that conflict with the model's dtype for FFT. This
way the existing validation still catches real mismatches for non-FFT
cases, and the corrective logic below handles the normalized settings.
Fixes the issue raised in Codex review of PR #6813.
* Guard Windows ROCm torchao override skip
Detect installed ROCm torch directly before applying the torchao override so Windows ROCm environments never install the crashing torchao package even if the earlier ROCm-installed flag is missing.
* Update unsloth/models/rl.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/install_python_stack.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Harden ROCm probe and sync RL precision flags
Tolerate stray stdout noise when probing Windows ROCm torch installs by checking the last non-empty output line, matching the existing torch version probe behavior. Also keep args.fp16 and args.bf16 synchronized with the full-finetuning precision auto-corrections in the RL trainer patch so downstream eval settings see a consistent TrainingArguments state.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add MLX trainer compatibility shims
Patch imported MLXTrainer and MLXTrainingConfig objects to preserve the expected dataclass field ordering and to provide a _train_dataset_for_batches fallback when older trainers or test doubles only expose train_dataset. Also add focused worker tests covering both compatibility paths.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Scope PR to Windows ROCm torchao guard
* Restore PR scope to Windows ROCm guard
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test: cover Windows ROCm torchao skip behavior
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Ayushman Paul <ayushman@HP>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Fix external drive custom folder selection
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/backend/tests/test_linux_external_media_paths.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep legacy media scan validation strict
* Apply sensitive-dir denylist to legacy folder browser for PR #6799
The legacy /api/models browse endpoint gained the new /run/media mount
roots in its allowlist but not the credential/config guard that scan-folder
registration and the Hub browser already enforce. Filter sensitive names
during enumeration and reject them in _resolve_browse_target so .ssh, .aws,
.config, etc. under allowlisted roots stay unbrowseable, matching the Hub
browser. Add a public contains_sensitive_path_component helper and cover the
legacy resolver with a regression test.
* Trim redundant comments in PR #6799 changes
* Skip sensitive Linux media roots
* Reject sensitive dirs at exact browse roots for PR #6799
Both _resolve_browse_target functions only checked contains_sensitive_path_component
while walking descendant parts, so requesting an allowlisted root itself (empty
relative path) returned it unchecked. A pre-existing scan-folder row under ~/.ssh,
~/.aws, ~/.config, etc. (registerable before the denylist was added) is re-added to
the allowlist on upgrade and could then be browsed. Check the resolved target once
before returning in both the legacy and Hub browsers, and cover the root case in
both test suites.
* fix: avoid unused path helper reexports
* fix: import sensitive path helpers directly
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: oobabooga <112222186+oobabooga@users.noreply.github.com>
* Fix TrainingArguments silently disabling unsloth gradient checkpointing
* Cover loaded adapters and preserve explicit None in GC restore
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: flush passthrough stream headers before upstream prefill stalls
* Studio: clean up delayed passthrough send failures
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: close passthrough preheader cleanup gaps
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: retry delayed passthrough overflow truncation
* Studio: close completed passthrough send responses
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: expose full compressed-tensors scheme set in an export formats dropdown
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: multi-select export formats, portable torchao FP8/INT8, GGUF LoRA, source parity
Export page overhaul on top of the formats dropdown:
- Unify merged precision into one sorted multi-select list (16-bit first, then
8-bit, then 4-bit). Drop "vLLM" from labels, add INT8 (W8A8), INT8 (W8A16),
INT4 (W4A16), MXFP4, MXFP8. Quick formats render as toggle pills; the rest live
in a multi-select "More formats" dropdown, so several formats export in one run.
- Add a portable torchao FP8/INT8 save path (Float8WeightOnlyConfig /
Int8WeightOnlyConfig) that needs no NVIDIA GPU to produce and loads in vLLM.
FP8 serializes to safetensors, INT8 to .bin. Wired into save_pretrained_merged
and push_to_hub_merged via a TORCHAO_EXPORT_SCHEMES registry and
_unsloth_save_torchao, parallel to the compressed-tensors path.
- Hide NVIDIA-only compressed-tensors formats when no NVIDIA GPU is present; keep
16-bit and portable FP8/INT8. The backend also rejects a compressed request on
non-NVIDIA hardware so it stays authoritative.
- Relax merged export to non-PEFT models so Local Model and Hugging Face sources
get the same 16-bit / compressed / portable options.
- GGUF: send the whole quant list in one call (merge once, quantize many).
- LoRA: add a GGUF adapter option (convert_lora_to_gguf.py) with an outtype
select (f16/bf16/f32/q8_0/auto), alongside the safetensors adapter.
- Thread the new fields through models, routes, orchestrator, and worker; extend
the export tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: gate export by accelerator with a torch-aware reason; fix export save dir naming
Export runs through Unsloth, which requires a compute accelerator (NVIDIA/AMD/Intel
GPU or Apple MLX) and has no CPU code path, so a bare-CPU host cannot export even
with PyTorch installed. Add export_capability() in utils/hardware that reports
export_supported plus a precise reason so the UI stops showing a generic "no GPU":
- pytorch_not_installed: a --no-torch install (even a physical GPU is unusable)
- no_accelerator: PyTorch present but no supported accelerator (bare CPU)
- mlx_unavailable: Apple Silicon where the MLX stack is missing or too old
Expose the fields on /api/system/hardware and /api/system, and guard the mutating
export routes (load-checkpoint, export/merged|base|gguf|lora) with HTTP 400 and the
reason, leaving read-only endpoints usable so the Export page still renders.
Make core/export/export.py import without PyTorch and without a usable accelerator
(the Unsloth import is caught) so the export worker degrades to a clear message
instead of crashing at import.
Frontend: keep /export reachable on chat-only hosts and gray out the method and
format options with the backend reason (Alert plus disabled MethodPicker) instead
of silently redirecting to /chat, so users see why export is unavailable.
Also fix the export save directory producing "model/null" for Local Model and
Hugging Face sources that have no run/checkpoint, naming the folder from the model id.
* CI: validate Studio export capability gating on Linux, Windows and macOS
Add a small pytest matrix that runs studio/backend/tests/test_export_capability.py
on ubuntu-latest, windows-latest and macos-latest. It confirms, on each real OS,
that hardware.export_capability() reports the right decision and reason
(pytorch_not_installed, no_accelerator, or mlx_unavailable) and that the export
backend imports without PyTorch and degrades to a clear message instead of crashing.
Hosted runners have no GPU/MLX, so this covers the "export unavailable, here is why"
path a Mac/Windows user without an accelerator sees; a real accelerator export is
validated separately. The job installs only a CPU PyTorch plus the backend import
deps (no unsloth, triton, or llama.cpp), so it runs in seconds with no GPU.
* Studio export: address Codex review (source-aware gating, GGUF LoRA token/MLX/guard)
Frontend (export-page):
- Gate LoRA and quantized-model restrictions on the active source. isAdapter /
isQuantized come from the selected checkpoint; in Local Model / Hugging Face
("model") source mode they were stale, so LoRA stayed wrongly enabled for a
direct base model (backend then rejects "No adapter to export") and a stale
"quantized" flag disabled every method for an unrelated, exportable model. Add
effectiveIsAdapter / effectiveIsQuantized (false outside checkpoint mode) and use
them in the method-reset effect and the MethodPicker disabled state.
- Hide the GGUF LoRA option on a macOS/MLX host (the backend rejects GGUF LoRA on
MLX), so users no longer pick it, wait through the load, and always fail. Disable
the "GGUF adapter" button on a Mac host and never send loraGguf there.
Backend (core/export/export.py):
- Pass the HF token into the GGUF LoRA conversion (save_pretrained_gguf), so a
gated/private base model's config fetch in convert_lora_to_gguf.py is
authenticated; without it the load can succeed but the conversion fails.
- Guard the save_pretrained_gguf capability check with getattr so an older Unsloth
model that lacks the method returns the clean "not supported" message instead of
an AttributeError that surfaces as a generic 500.
* Studio export: address 2nd Codex review (CI index, empty merged, test import)
- studio-export-capability-ci.yml: add --extra-index-url https://pypi.org/simple to
the torch install so torch's transitive deps still resolve; --index-url alone
replaces PyPI with only the CPU wheel index, which does not serve all of them.
- export-page handleStart: reject an empty merged selection (mirrors canExport), so
clicking the panel's Start button with every precision pill deselected no longer
submits mergedSelections: [] and launches an unintended default 16-bit export.
- test_export_imatrix_compressed: the torchao-registry test now reads unsloth/save.py
as text (like the other ast/string checks) instead of `import unsloth.save`, which
raised ModuleNotFoundError in the CPU studio-backend suite that has no unsloth
installed.
* Studio export: make comments succinct across the export changes
* Studio export: use load token for local GGUF LoRA export of gated bases
* Studio export: harden portable torchao path and gate multi-format Hub push
torchao (_unsloth_save_torchao):
- merge to an isolated temp staging dir so a co-selected 16-bit output at save_directory is not deleted
- narrow VLM detection to vision_config / ForVisionText2Text so T5/BART/Whisper are not misrouted
- forward trust_remote_code (from auto_map) to the reload so custom-code models export
Export UI:
- hide portable torchao formats on macOS/MLX (backend rejects quantized export there)
- restrict a Hub merged export to a single format (each writes to the repo root)
* Studio export: torchao tokenizer remote-code + XPU offload, scale GGUF timeout
torchao (_unsloth_save_torchao):
- honor auto_map in the staged tokenizer/processor configs (not just model.config) when
deriving trust_remote_code, so custom-code tokenizers reload after the merge
- offload single-device XPU models to CPU (and empty the XPU cache) before the reload, matching
the CUDA path, so an Intel GPU that fits the model once does not OOM on the second copy
Export orchestrator:
- scale the GGUF wait timeout by the number of requested quants so a multi-quant list export of a
large model does not time out at a flat 3600s
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio export: show portable torchao formats only on non-NVIDIA (CPU) hosts
Portable torchao FP8/INT8 is the fallback for hosts without the NVIDIA compressed-tensors path.
On an NVIDIA GPU the compressed-tensors FP8/FP4/INT formats are the intended path (llm-compressor
auto-installs), so hide the portable duplicates there; keep them on CPU / non-NVIDIA hosts and
continue hiding them on macOS/MLX.
* Studio export: report all output folders and the exported formats
- Multi-format merged export now collects every sibling output directory (one per selected
precision) instead of only the last; the success banner lists them all.
- Show the selected precision formats in the run summary (a Formats row, like GGUF Quantizations),
so the panel says what is being exported rather than just 'Merged Model'.
- Persist the selected formats in the run summary and seed them on mount, so navigating away and
back (or toggling the export method) restores the selection instead of resetting to 16-bit.
* Studio export: list all output formats, add GGUF LoRA target, default Q8_0, auto-select newest checkpoint
- Progress/summary panel now shows a Formats row with the selected merged
formats, and the success banner lists every output folder a multi-format
merged run creates (one line per format) instead of only the last one.
- Merged format selection is seeded from the active run, so navigating away
and back (or switching method cards) no longer resets it to 16-bit.
- GGUF / Llama.cpp now offers an Export target toggle (Full model or LoRA
adapter) for adapter checkpoints, reusing the LoRA GGUF export path.
- Removed the Auto GGUF LoRA output type and defaulted to Q8_0 in the UI,
the request model, and the backend defaults; the outtype list is now
Q8_0/F16/BF16/F32. Core save.py still accepts auto for external callers.
- When a finetune has no checkpoint selected, auto-select the newest one.
* Studio torchao export: robust reload class + optional VLM import
Two fixes to the portable torchao FP8/INT8 export reload, from review of the
narrowed VLM detection:
- Encoder-decoder seq2seq checkpoints (T5/BART/Whisper) are not causal LMs.
With the narrowed is_vlm test they now correctly skip the image-text class,
but fell through to AutoModelForCausalLM and failed to reload after the merge.
Reload them with their own architecture class from the config instead.
- AutoModelForImageTextToText was imported unconditionally at the top of the
torchao path, so on Transformers builds without that class the import aborted
every torchao export (even text-only). Import it lazily only for a VLM, with
the AutoModelForVision2Seq fallback used elsewhere in Unsloth.
* Studio: enable FP8/FP4 compressed export for newer-transformers models
The shipped llm-compressor 0.10.x pins transformers<=4.57.6, so FP8/FP4 export failed
for models needing a transformers 5.x sidecar (Qwen3.5, Gemma-4, Qwen3-Next): the
quantization subprocess crashed importing the removed TORCH_INIT_FUNCTIONS.
Run the quantization against a dedicated llm-compressor-main "shadow": a --target
package dir (transformers 5.10.2 + llm-compressor main + compressed-tensors) layered
over the existing torch. It installs --no-deps so torch is never touched (works on any
Studio torch build), is provisioned lazily and fingerprint-cached, and can be turned
off with UNSLOTH_DISABLE_LLMCOMPRESSOR_MAIN.
- transformers_version.py: provision + validate .venv_llmcompressor.
- export.py: route all compressed exports through the shadow when available; else keep
the workspace 0.10.x path and fail fast past its transformers ceiling.
- save.py: launch _compressed_quantize.py with a clean PYTHONPATH = shadow.
- _compressed_quantize.py: skip linear_attn / vision tower / MTP modules (matches the
RedHatAI and NVIDIA reference quants, and is required by the grouped schemes).
Verified all four schemes (fp8, w8a8, w4a16, mxfp4) on Qwen3.5-9B and Llama-3.2-1B, and
fp8 on Gemma-4, end to end through Studio.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix GGUF LoRA export tests
* Fix export CI expectations
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <112766706+wasimysaid@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* inference: add passthrough tool-call healing core (heal_gate, heal_openai_message, StreamToolCallHealer, nudge helpers)
Small GGUF models often emit tool calls as text (<tool_call>{...}</tool_call>,
Gemma <|tool_call>, <function=> XML) instead of structured tool_calls. Studio's
enable-tools loop already heals these, but the client-tool passthrough
(unsloth run --disable-tools, unsloth start agents) relays them verbatim, so
the agent sees prose and the turn dies.
This module is the shared response-side repair layer the passthrough routes
will call: promote parsed text-form calls to structured calls, but only for
function names the client actually declared; coerce arguments through the same
canonical-key healing as the tool loop; never touch the upstream request body
(llama-server KV/slot reuse stays byte-identical). StreamToolCallHealer is the
streaming buffer-and-repair state machine: prose forwards immediately, only a
partial-signal tail or a suspected tool block is held, false alarms flush
verbatim, and a 64 KiB bound caps memory. nudge_should_retry/nudge_messages
support an opt-in single-retry nudge for non-streaming routes (wired later).
Kill-switch: UNSLOTH_DISABLE_TOOL_CALL_HEALING=1. Reuses
core/tool_healing.parse_tool_calls_from_text, strip_tool_call_markup, and
tool_loop_controller.coerce_tool_arguments unchanged.
* inference: heal text-form tool calls on the OpenAI and Responses passthrough
Wire the passthrough healing core into /v1/chat/completions and /v1/responses,
default ON whenever the request declares client tools:
Non-streaming: heal_openai_message runs inside the existing response-mutation
loop; a promoted call flips finish_reason to tool_calls and nulls the content,
and the verbatim-bytes fast path still applies when nothing was healed.
/v1/responses non-streaming inherits this through openai_chat_completions.
Streaming: a StreamToolCallHealer per stream. Ordinary prose relays
byte-for-byte (a fast path keeps upstream bytes when the healer passes a chunk
through whole); once a tool signal appears, content is held, and at the
finish/[DONE] boundary either synthetic delta.tool_calls chunks replace the
markup (finish_reason rewritten to tool_calls, including the synthetic-finish
path) or a false alarm flushes the held text verbatim. Structured upstream
deltas put the healer to sleep after flushing anything held, so grammar-mode
responses stay byte-identical. The Responses stream feeds healed calls through
the same per-call state machinery as structured deltas (indexes live in a
disjoint range so a healed call can never merge into a structured call's
state), and the visible/reasoning split runs first so reasoning text is never
promoted. parallel_tool_calls=false caps healed calls on every path.
The upstream request body is never touched and healing issues no extra
generation, so llama-server slot/KV-cache reuse is unchanged. Opt-out per
request with auto_heal_tool_calls=false (Responses reads it from the
extra-body); requests without tools relay verbatim.
* inference: heal text-form tool calls on the Anthropic /v1/messages passthrough
Streaming: AnthropicPassthroughEmitter.enable_healing(allowed_tools) routes
content deltas through the shared StreamToolCallHealer. A promoted call closes
any open text block (only the safe prose prefix ever streamed into it), opens a
synthetic tool_use block with a fresh toolu_* id, carries one input_json_delta,
and closes; finish() then forces stop_reason to tool_use unless a truncation
(max_tokens) wins. Structured upstream deltas flush anything held and put the
healer to sleep, so grammar-mode responses are untouched, as is every stream
where enable_healing is never called (Studio's own loop, no-tools requests).
disable_parallel_tool_use caps healed calls too.
Non-streaming: the OpenAI message dict is healed BEFORE block building, so the
existing tool_use promotion loop and stop_reason line treat promoted calls
exactly like native ones (finish_reason length still maps to max_tokens). The
legacy tool-XML strip still runs on remaining text, so opted-out requests keep
today's cleanup behavior byte-for-byte.
auto_heal_tool_calls is now a typed field on AnthropicMessagesRequest
(default True, mirroring Chat Completions) and threads into both passthrough
calls. Healing never touches the upstream request body.
* inference: opt-in single-retry tool-call nudge on the non-streaming passthrough
When the model clearly tried to call a tool (a tool signal in the text) but
healing produced nothing usable, re-ask once: the retry body is the original
body plus an assistant turn (the model's own failed text) and a short user
nudge naming the declared tools. The prompt prefix stays byte-identical, so
llama-server reuses the slot's KV cache and only the two-message suffix is
prefilled. The retry replaces the original response only when it actually
yields a promotable or structured call; on any error or still-garbage output
the original response is returned unchanged. Exactly one retry, non-streaming
OpenAI and Anthropic passthroughs only (a stream has already emitted bytes).
OPT-IN per user decision: nudge_tool_calls=true per request (typed on both
ChatCompletionRequest and AnthropicMessagesRequest, lifted from the Responses
extra-body), or UNSLOTH_TOOL_CALL_NUDGE=1 to flip the process default.
auto_heal_tool_calls=false disables healing AND the nudge.
Also align the non-streaming heal on allow_incomplete=True: the response is
final, so a trailing unclosed tool block is a model failure worth repairing,
matching the enable-tools loop's drain semantics.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: never assume the upstream response shape in the nudge helpers
llama-server error bodies can carry message: null (or no choices at all), and
_last_assistant_text / response_has_promotable_calls / nudge_should_retry
called .get() on the message without a dict check, so a malformed upstream
response raised an AttributeError the surrounding except tuples did not catch,
failing the request instead of degrading to 'nothing to heal'. Route the shape
probing through one _first_choice_message helper that returns None for any
non-dict message, and add a parametrized test over the malformed shapes.
* inference: constrain healing by tool_choice, preserve length finish_reason, keep healed event order in Responses streams
Three review findings on the passthrough healer:
- heal_gate now honors the request's tool_choice: "none" disables healing
outright and a forced function narrows the promotion allowlist to that
one function, so healing can never contradict the request's tool-choice
constraint. Wired through the OpenAI chat (stream and non-stream),
Responses, and Anthropic (converted shape) passthroughs.
- The OpenAI non-streaming heal only upgrades finish_reason "stop" to
"tool_calls"; a truncated generation keeps "length" (the healed call
stays attached) matching the streaming and Anthropic paths.
- The Responses stream emits healer events in order instead of collapsing
all text ahead of the healed calls, so text after a healed call no longer
jumps ahead of the function_call item and output indexes are claimed in
the order the model produced them.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: all-or-nothing promotion when a response mixes declared and undeclared text-form calls
Promoting a subset used to strip ALL tool markup from the content, which
silently deleted the text of any call naming an undeclared tool. The heal
now declines entirely when any parsed call is unpromotable, so the whole
message relays verbatim (pre-PR behavior) and no bytes are ever lost. In
streaming, a declared call that completed before an undeclared one arrived
is already emitted; the late undeclared markup still flushes as raw text.
The nudge helpers mirror the same contract via a shared predicate.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* tests: wrap long lines in the Responses healing tests to the project style
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: span-exact healing, disjoint healed stream indexes, per-call Responses message items, allowlisted nudge acceptance
Four review findings on the passthrough healer:
- parse_tool_calls_from_text gains an optional with_spans return so healing
removes EXACTLY the promoted calls' markup. This supersedes the previous
all-or-nothing rule: declared calls promote and every unpromoted byte
(undeclared calls, unparseable closed blocks, suppressed alternate
formats such as a <function=...> block after a JSON call) relays as text.
The stream healer also processes one block per pass, so text between two
healed calls keeps its document position instead of trailing them.
- The OpenAI chat stream shifts native tool-call delta indexes past any
already-emitted healed calls; clients merge deltas by index, so a healed
call and a later native call can no longer merge into one.
- A healed call in the Responses stream closes the open message item and
trailing text opens a fresh one with a later output index, matching the
native stream shape; response.completed snapshots every message item
with its own text.
- The nudge retry only replaces the original response when the retry's
structured call names a DECLARED tool; a hallucinated undeclared call is
not an improvement.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stop the heal path folding trailing prose into a closed function call
parse_tool_calls_from_text(allow_incomplete=True) cut a <function=...> body only
at an end-anchored </function>, so a fully closed call followed by trailing prose
(<function=..>..</parameter></function> words) folded </parameter></function> and
the prose into the tool argument and deleted the prose from visible content. The
strict path (allow_incomplete=False) already cut at the real </function> via rfind.
Do the same in both modes: trim the body at the real </function> when present and
end the removal span there, falling back to the end-anchored strip and body_end
only when the call is genuinely truncated. Add a regression test.
* inference: one shared single-call budget for healed and native calls
Codex round 5: the parallel-call caps counted healed and native calls
separately, so a healed text-form call followed by a native structured
delta double-emitted on all three streaming surfaces when the client
disabled parallel calls.
- OpenAI SSE: once a healed call went out with parallel_tool_calls
false, native tool_call deltas are dropped instead of index-shifted.
- Anthropic emitter: native deltas skip block allocation when the
healed-plus-native count already filled the single slot, and healed
emission counts open native states too.
- Responses stream: native deltas that survived the chunk-level cap are
skipped once a healed call claimed the slot.
Also adds a span assertion for the closed-</function> trailing-prose
parse fixed in the previous commit.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: relay undeclared text-form calls as text on Anthropic non-streaming
heal_openai_message promotes only declared text-form tool calls and
span-trims just their markup, deliberately leaving every unpromoted byte
(undeclared text-form calls included) in the content to relay as text.
The Anthropic non-streaming builder then ran a blanket _TOOL_XML_RE strip
over that content unconditionally, deleting the undeclared block before
building the text part, so Anthropic clients silently lost a call the
OpenAI non-streaming path preserves. The strip was harmless when healing
was all-or-nothing but became data loss once healing turned span-exact.
Gate the legacy strip on whether healing promoted a call, matching the
OpenAI passthrough and the intent already stated in the comment above.
Add a route-level regression test for the mixed declared+undeclared case.
* inference: require fully declared nudge retries; keep unpromoted Anthropic text
Codex round 6, two findings:
- response_has_promotable_calls accepted a nudge retry when any one
structured call named a declared tool, so a mixed retry (hallucinated
undeclared call plus a declared one) replaced the original and the
caller forwarded the undeclared call, or with parallel_tool_calls
false could keep only it. All structured retry calls must be declared.
- The Anthropic non-streaming builder still ran the legacy _TOOL_XML_RE
strip after span-exact healing, deleting undeclared or malformed call
text that healing deliberately preserved. The legacy strip now runs
only when healing is off (no declared tools, or opted out), matching
the OpenAI passthrough.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* inference: keep unpromoted Anthropic text whenever healing is active
The previous commit skipped the legacy strip only when a call was
actually promoted, so an undeclared-only (or malformed-only) response
was still silently emptied: exactly the dead-turn shape this path
exists to fix, and inconsistent with the OpenAI passthrough, which
relays those bytes verbatim. Gate the strip on healing being active
instead; opt-out and no-tools requests keep the legacy strip.
* Fix schema-aware tool healing for PR #6801
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix passthrough healing ordering for PR #6801
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix stream finish ordering for PR #6801
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: wasimysaid <112766706+wasimysaid@users.noreply.github.com>
* replaced connect with start
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix
* Studio: build the coding-agent command from the selected server
The API keys panel showed a hardcoded `unsloth start claude`. `unsloth start`
defaults to 127.0.0.1:8888 and only mints a key for a loopback server, so a
non-default port or a tunnel/remote base would target the wrong server or fail
to mint. Build the command from the panel base/key (and emit a key for
non-loopback), matching the other snippets in the panel.
* CLI: keep `unsloth connect` as a hidden alias for `unsloth start`
Avoids breaking existing scripts and docs that still call `unsloth connect`.
* Tests: stub _unstarted_cleanup in same-task disconnect test
The test builds _SameTaskStreamingResponse via __new__, so set the attribute
that __call__ now reads.
* Match coding-agent command loopback check to the CLI 127.0.0.0/8 rule (#6613)
* Keep unsloth_cli.commands.connect importable as a deprecated shim (#6613)
* Format the new coding-agents panel strings and import per biome (#6613)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Drop the unsloth connect alias and shim; unsloth start is the only command (#6613)
* Route unsloth connect to unsloth start as a hidden backward-compatible alias (#6613)
* Forward unsloth run model-load flags to unsloth start (gguf-variant, context-length, load-in-4bit, tensor-parallel) (#6613)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Session-scope coding agent config in unsloth start
Configure each agent for the current session instead of writing the Studio endpoint, key, and default model into the user's own config. Codex, OpenCode, OpenClaw, and Hermes get a private config relocated through their config-path env vars (CODEX_HOME, OPENCODE_CONFIG overlay, OPENCLAW_CONFIG_PATH plus OPENCLAW_STATE_DIR, HERMES_HOME). Claude Code suppresses the attribution header for the session via the CLAUDE_CODE_ATTRIBUTION_HEADER env var plus a --settings overlay, with no ~/.claude write. --launch uses an ephemeral temp dir removed after the agent exits; --no-launch uses a stable Unsloth-owned dir and prints the matching export lines.
* Read relocated agent session config in Local Agent Guides CI
The contract crosscheck and the openclaw/hermes patch helpers now read each agent's config from the relocated path printed by unsloth start --no-launch (CODEX_HOME, OPENCODE_CONFIG, OPENCLAW_CONFIG_PATH, HERMES_HOME) instead of fixed home paths. The Claude attribution A/B toggles the header for the session only (shipped-config HIT vs vanilla MISS) instead of editing ~/.claude/settings.json.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Skip the POSIX-only --no-launch parser test on Windows
test_no_launch_output_is_parseable mirrors the #6547 bash CI parser, which greps export/unset lines and only runs on Linux/macOS runners. On Windows --no-launch prints PowerShell ($env: / Remove-Item), so the export-line assertion does not apply there. Cross-OS staging CI surfaced this.
* Size Claude Code's auto-compact window to the loaded model's context
Claude Code auto-compacts against its native (~600k token) window, so against a smaller local model it overflows the server's context (silent truncation) long before it compacts. Set CLAUDE_CODE_AUTO_COMPACT_WINDOW to the loaded model's real context length (the value codex/openclaw already get via model_context_window / contextWindow). Omitted when the model reports no context length.
* Pin OpenCode/Hermes context window and set 90% compaction across agents
Feed every agent the server-determined sequence length (the value /v1/models reports from runtime_context_length) and a ~90% compaction threshold. OpenCode: a custom-provider model with no limit defaults to context 0, which silently disables auto-compaction, so set limit.context/output and scale the compaction buffer to 10% of the window. Hermes: pin model.context_length (it otherwise falls back to a 256k default when the server's /v1/models omits the field) and set compression.threshold 0.9. Claude: add CLAUDE_AUTOCOMPACT_PCT_OVERRIDE=90 alongside the window. Codex (model_context_window) and OpenClaw (contextWindow) already carried the window and auto-manage off it.
* Add `unsloth start pi` recipe
Pi was the only agent without a built-in recipe, so the agent-guides CI
hand-wrote ~/.pi/agent/models.json. Add a first-class `pi` command mirroring
the others:
- write_pi_config writes the session-scoped OpenAI-compatible provider config
(key in the config, like openclaw/opencode).
- pi() launches `pi --provider unsloth --model <id>` (Pi defaults to the google
provider, so the provider/model are pinned on the command line) with HOME
relocated for the session. Pi has no config-dir env var and resolves ~/.pi off
$HOME, so HOME-scoping keeps the user's ~/.pi untouched.
Migrate the agent-guides CI off the hand-written config onto the
`unsloth start pi --no-launch` path (connection + file-edit), with a crosscheck
for the provider api, so the documented recipe is exercised.
* Harden unsloth start for Windows and WSL agent launches
Address the Codex review on PR 6613:
- write_pi_config now pins the loaded contextWindow and a sane maxTokens so Pi
compacts instead of overflowing a small Studio context (it otherwise assumes
its 128000 default), matching the other agents.
- pi() sets USERPROFILE (and HOMEDRIVE/HOMEPATH when present) alongside HOME on
native Windows, where Node resolves ~/.pi via USERPROFILE rather than HOME, so
the session no longer reads or writes the user's real ~/.pi.
- The WSLENV bridge flags path-valued vars with /p so a Windows npm shim under
/mnt receives translated paths, while scalar vars (the numeric context window)
pass through untranslated. WSLENV is deduped on the bare name.
- _print_env prints the launch command with PowerShell-safe quoting so the inline
--settings JSON survives copy-paste on native Windows --no-launch.
Add tests for the WSLENV path flagging, PowerShell quoting, the Pi context
window, and the Pi USERPROFILE relocation.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Set CLAUDE_CODE_NO_FLICKER for the Claude session
A local server streams in bursts, so Claude Code's full-screen TUI redraw
flickers between tokens. Disable it for the session via CLAUDE_CODE_NO_FLICKER,
alongside the other CLAUDE_CODE_* session env knobs.
* Add a normalized --yolo flag routed to each agent's auto-approve mode
It is easy to forget which agent spells "run tools without prompting" which way,
so `unsloth start` now accepts all three spellings as one option (--yolo,
--dangerously-skip-permissions, --dangerously-bypass-approvals-and-sandbox) and
routes to the agent's own mechanism:
- claude: --dangerously-skip-permissions
- codex: --dangerously-bypass-approvals-and-sandbox
- hermes: --yolo
- pi: --approve (Pi's only approval gate is project trust)
- opencode: a permission allow block in opencode.json (no CLI flag exists)
- openclaw: tools.exec security=full / ask=off / host=gateway (no CLI flag exists)
Because the option is parsed by `unsloth start`, the "wrong" spelling for an
agent still routes correctly instead of leaking through to the agent and erroring.
IS_SANDBOX is deliberately left unset for Claude so its root/sandbox safety gate
still applies. Adds routing, cross-routing, and per-config tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix review findings: IPv6 loopback command, pi USERPROFILE under WSL, yolo guard
From a 10-reviewer pass over the PR:
- studio/frontend agent-command.ts: normalize bracketed IPv6 hosts. URL.hostname
returns "[::1]" for http://[::1]:8888, which never matched the "::1" loopback
checks, so the copied command embedded the placeholder API key for a local IPv6
server instead of the bare auto-minting command. Now [::1] is treated as loopback
like the CLI's is_loopback_url, so the command matches the CLI contract.
- pi(): also relocate USERPROFILE (and HOMEDRIVE/HOMEPATH) when running under WSL
against a /mnt Windows shim, not just on native Windows. Windows Node resolves
~/.pi via USERPROFILE, and the WSLENV bridge translates the path, so pi no longer
falls back to the user's real ~/.pi in that case.
- _yolo_command_flags: use .get so a config-based agent (or a typo) yields no flag
instead of a latent KeyError.
Adds tests for the WSL pi USERPROFILE relocation, the yolo unmapped-agent guard,
and that opencode/openclaw --yolo stays config-only (no argv flag).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix round-2 review findings: WSLENV /p upgrade, agent help text
- _merge_wslenv now upgrades a user's pre-existing unflagged WSLENV entry (e.g. a
bare HOME or USERPROFILE) to the path-translated form (HOME/p) instead of leaving
it as-is, so a Windows agent shim under WSL receives the translated session path
rather than the raw Linux path.
- Generalize the `unsloth start` registration help to list all six agents (was only
"Claude Code, Codex").
Adds a test for the WSLENV unflagged-entry upgrade.
* Fix round-3 review findings: complete openclaw --yolo, refresh stale copy
- openclaw --yolo now also writes the host approvals file (exec-approvals.json with
defaults security=full / ask=off / askFallback=full) alongside the tools.exec
config. OpenClaw gates tool execution on both layers (the stricter wins), so the
config alone could still leave it prompting or denying. Mirrors `openclaw
exec-policy preset yolo`. ask=off means nothing is ever prompted, so the runtime
socket block is unnecessary.
- Studio API panel copy: clarify that a local server auto-mints the key while a
remote one embeds it in the command, and add pi to the swap hint.
- Local Agent Guides CI: drop the stale "pi has no start.py recipe" note now that
all six agents are driven via `unsloth start <agent> --no-launch`.
Adds the openclaw approvals-file assertions and a no-yolo openclaw test.
* start: parse claude --version with a regex so a format change does not drop optimization flags
* start: offer to install a missing agent (prompt then run its install command)
* start: auto-start a Studio server for --model when none is running, and stop it on exit
* inference: surface an actionable message when llama-server cannot compile a tool grammar
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix review findings: kill the auto-started server tree on Windows; apply the tool-grammar message to the OpenAI passthrough too
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* start: split --model org/repo:variant so a running session is not evicted
`unsloth start <agent> --model org/repo:QUANT` failed against an already-running
Studio server and, worse, killed whatever model another session had loaded.
/v1/models lists a loaded GGUF under its bare repo id (e.g. unsloth/Qwen3-1.7B-GGUF),
so _resolve_model never matched the `:QUANT`-suffixed request. It then POSTed
/api/inference/load with model_path=org/repo:QUANT, which (a) Hugging Face rejects
("Repo id must use alphanumeric chars, '-', '_' or '.'") and (b) evicts the model the
other session was using, so a second 'unsloth start' in a new tmux/terminal tore down
the first. Re-running the command then attached to the now-empty server, which is why
it 'worked the second time'.
Mirror the org/repo:QUANT -> org/repo + --gguf-variant QUANT shorthand that
'unsloth run' and llama.cpp already accept, splitting it in _connect before we match or
serve. Matching now resolves against the loaded bare repo id (no spurious reload, no
eviction), and any real load uses a valid repo id plus gguf_variant. An explicit
--gguf-variant still wins; local paths and Windows drive letters pass through untouched.
The auto-serve path likewise spawns 'unsloth run --model org/repo --gguf-variant QUANT'.
* start: harden auth-key handling, codex teardown, and CI transcript redaction
Three review findings:
1. CI could leak a live key. agent-guides-drive.sh printed the raw
'unsloth start --no-launch' transcript (which carries export UNSLOTH_API_KEY /
ANTHROPIC_AUTH_TOKEN lines) to the Actions log on both the failure path and the
success path before redact() ran. Add cat_redacted() and use it for those two
prints, so the key is scrubbed on the way to the log while the on-disk file stays
intact for the env parsing that follows.
2. Outages masqueraded as bad keys. _key_accepted caught a broad Exception and
returned False, so a 5xx or timeout while checking a cached key looked like a
rejection: it discarded a good key and minted extra ones (local) or reported 'no
saved key' (remote). Only treat HTTP 401/403 as a rejection; let other errors
propagate so a real outage surfaces.
3. Codex preflight could leave the auto-started server up. _require_gguf_for_codex
runs after _connect may have auto-started Studio but before _run installs its
teardown finally, so a preflight rejection (e.g. a transformers-backend model) left
the server holding the port/GPU until the atexit backstop. Tear it down explicitly
at the point of failure.
Tests: a 5xx on a saved key surfaces without minting; a non-GGUF codex preflight
tears down the auto-served server.
* start: fix IPv6/portless studio URLs, Pi config-dir isolation, and Pi install recipe
Four review findings:
1. Pi ignored the session config when PI_CODING_AGENT_DIR was already set. Pi's
getAgentDir() reads process.env.PI_CODING_AGENT_DIR before falling back to
$HOME/.pi/agent, so a value inherited from the user's shell sent Pi to their real
config and skipped our provider/key (the HOME relocation alone was not enough). Pin
PI_CODING_AGENT_DIR at the session's .pi/agent dir; it is path-valued so the WSL
bridge translates it automatically.
2. Pi install hint dropped Pi's documented --ignore-scripts. Pi's README installs with
'npm install -g --ignore-scripts @earendil-works/pi-coding-agent' and notes it needs
no install scripts, so accepting the prompt now follows that safe recipe.
3. Auto-start ignored a portless UNSLOTH_STUDIO_URL. unsloth run binds to
'parsed.port or 8888', so http://127.0.0.1 launched the child on 8888 but the health
poll (and the returned base) still used port 80, stalling until the startup timeout.
Normalize the base to host:8888 (IPv6-safe) before starting and polling.
4. API-panel command mistook IPv6 loopback for the bare default. The bare 'unsloth
start' only probes 127.0.0.1:8888 on the IPv4 stack, so http://[::1]:8888 must carry
an explicit UNSLOTH_STUDIO_URL. Drop ::1 from the bare-default host set while keeping
it a loopback host (URL emitted, no key needed).
Tests: PI_CODING_AGENT_DIR is set to the session dir; _effective_base normalizes
portless/IPv6 bases; a portless UNSLOTH_STUDIO_URL auto-serves on :8888.
* start: apply fresh-review findings across CLI, CI, and the API-panel command
From a fresh multi-reviewer pass over the merged head plus the latest Codex bot review:
1. Load knobs now always consult the server. _resolve_model matched on model id alone,
so --gguf-variant / --context-length / --no-load-in-4bit / --tensor-parallel were
silently ignored whenever the id was already loaded (asking for UD-Q4_K_XL kept a
Q8_0 serving). With any explicit knob the CLI defers to /api/inference/load, whose
already-loaded dedup answers without reloading when variant and settings match, so a
second session running the same command still attaches without evicting the first.
2. OpenCode --yolo and the session model pin now ride in OPENCODE_CONFIG_CONTENT. A
project's own opencode.json outranks OPENCODE_CONFIG, so a repo config could silently
override the session model and the --yolo permission block; OPENCODE_CONFIG_CONTENT
outranks project config. The API key stays in the private file, never in printed env.
3. The --no-launch recipe's last line is a self-contained one-liner (inline VAR=value
assignments before the command, conflicting vars blanked). People copy just the last
line, and a bare codex/claude there ran against the user's real ~/.codex or Anthropic
credentials with zero isolation, e.g. inheriting a pre-existing damaged ~/.codex
state DB and blaming the recipe. The CI drive script scrubs the key from the one
'invoking:' echo this adds.
4. The auto-serve log is 0600 and the parent handle is closed. It sat world-readable in
the shared tempdir under a predictable name while carrying the minted sk-unsloth-
key from the unsloth run banner.
5. _key_accepted fails with a clean message on outages. Non-auth errors (5xx, network,
timeout) surfaced as a raw traceback; 401/403 still mean a rejected key.
6. _effective_base strips URL paths, and https loopback targets never auto-serve.
http://127.0.0.1:8888/studio polled /studio/api/health (404) and https://127.0.0.1
polled the wrong scheme, both spinning until the 15-minute startup timeout.
7. API-panel command: only literal 127.0.0.1:8888 earns the bare command. localhost can
resolve to ::1, which the bare CLI never probes, so it keeps UNSLOTH_STUDIO_URL.
8. CI artifact sweep covers redacted-configs/ and agent-workdir/, not just logs/.
Tests: 125 CLI tests pass (new coverage for each fix), 156 backend tests pass, ruff
clean. Adds an unsloth connect alias regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* start: hand Pi a clean screen at launch
Pi paints inline from wherever the cursor sits: its first render assumes a
clean screen instead of clearing or entering the alternate screen itself
(current Pi never emits a clear at startup). Launched under unsloth start,
that left the session starting mid-scroll beneath the connection output.
Clear the screen (click.clear, cross-platform, no-op without a TTY) right
before the Studio banner so Pi opens exactly one line down on a clean
viewport. Launch path only: --no-launch recipes and piped output are never
wiped, and alternate-screen agents are left alone.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* start: auto-override hermes' 64K context floor for small model windows
Hermes refuses to initialize when the served model's context window is
under 64,000 tokens, and a second copy of the same check rejects the
compression model mid-session. write_hermes_config previously pinned the
real window, so any small local model (e.g. 40,960) failed at startup
with manual config.yaml instructions.
For windows below the floor the recipe now claims 65,536 in
model.context_length, scales compression.threshold so compaction still
fires at 90% of the real window, and sets
auxiliary.compression.context_length to cover the mid-session check.
Windows at or above the floor keep the exact previous behavior.
* ci: install pi with --ignore-scripts, matching the start.py hint
The pi cell predates the pi recipe in start.py and still installed the
package with lifecycle scripts enabled, so CI stopped exercising the
exact command users are prompted to run. npm_retry now passes extra
flags through, the pi branch mirrors the install hint verbatim, and the
stale no-recipe comment is refreshed.
* ci: fail loudly when a relocation var is missing from connect output
The empty-string guards ran after appending /config.toml or /config.yaml,
so they could never fire: crosscheck_contract silently skipped its
contract checks and patch_hermes_tools died on the root path with a bare
traceback. Check the raw variable first and guide_fail with the real
cause.
* staging: 6613 round 6 (https elision, no-launch home reuse, auto-start key fallback)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: shimmyshimmer <107991372+shimmyshimmer@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* fast_generate: clear error for vLLM-style inputs when fast_inference=False
When fast_inference=False, fast_generate falls back to HuggingFace
generate, and the wrapper already rejects vLLM-only usage (a
sampling_params or lora_request kwarg, or a string prompt). A vLLM prompt
dict ({'prompt':..., 'multi_modal_data':...}) or a SamplingParams passed
positionally slipped through and hit transformers.generate, raising a
cryptic 'SamplingParams object has no attribute update'. Detect both and
raise the same clear 'only supported with fast_inference=True' error.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate: also reject positional list of SamplingParams and list of vLLM prompt dicts
Address review feedback: the slow-mode guard missed SamplingParams passed inside a
positional list and a list of {"prompt": ...} dicts, both valid vLLM batched shapes
that leaked into transformers.generate. Fold the checks into small predicates and
extend the GPU-free test (now 7 reject + 3 pass).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* test_fast_generate_slow_guard: expose assertions via a test_ function so pytest collects them
The assertions lived in run(), only called from __main__, so pytest reported no tests
collected and CI skipped the coverage. Rename to test_fast_generate_slow_guard; the
standalone script entrypoint still works.
* fast_generate: reject vLLM tokenized/embeds prompt dicts in the slow-mode guard
vLLM also accepts prompt dicts keyed by prompt_token_ids or prompt_embeds, not just
prompt/multi_modal_data. Those slipped past the slow-mode guard and fell through to
HuggingFace generate with a cryptic error. Recognize all vLLM prompt-dict keys and
add a TokensPrompt test case (now 8 reject + 3 pass).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate slow-mode guard: catch vLLM prompts= keyword form
vLLM's generate names its first argument `prompts`, so a slow-mode call
like fast_generate(prompts="hi") or prompts=[{"prompt": ...}] bypassed the
guard and leaked into HuggingFace generate as an unexpected kwarg. Check
kwargs["prompts"] with the same _is_vllm_prompt predicate and add two test
cases.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate slow-mode guard: reject vLLM tokenized prompt kwargs
vLLM's legacy call shape passes tokens as prompt_token_ids= (and prompt_embeds=),
which are not HuggingFace generate arguments. In slow mode these bypassed the
guard and leaked into HF generate as unexpected kwargs. Reject their presence
with the same tokenize-first message and add a test case.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fast_generate slow-mode guard: treat prompts= as vLLM-only
prompts is a vLLM keyword, not a HuggingFace generate argument, so any value
passed as prompts= (including a bare token-id list, which _is_vllm_prompt
deliberately ignores for positional HF token ids) is a vLLM-style call. Reject
prompts= / prompt_token_ids= / prompt_embeds= on presence, and keep the
conservative _is_vllm_prompt check only for the positional arg.
* fast_generate slow-mode guard: reject vLLM prompt kwargs on presence
prompts / prompt_token_ids / prompt_embeds are vLLM-only keyword names that
HuggingFace generate does not accept, so a defaulted call like prompts=None
should raise the actionable slow-mode error instead of leaking a None kwarg
into HF generate. Check membership in kwargs rather than a non-None value.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Wasim Yousef Said <wasimysdev@gmail.com>
* report a complete load once llama-server is healthy
load_progress() derived its fraction purely from the llama-server's VmRSS over the GGUF shard total. With layers offloaded to VRAM (-ngl) the process releases the mmap'd weight pages after upload, so VmRSS sinks back well below the shard total: the fraction climbs toward ~1.0 during mmap, then collapses to a small value (~8%) once the weights are on the GPU. A fraction-driven progress bar therefore restarts and sticks there indefinitely even though the model is loaded and serving, which reads as a hang at "Starting model...".
Once the server is healthy the load is complete by definition, so report
fraction 1.0 (and bytes_loaded == bytes_total) in the ready phase regardless of resident set size. The VmRSS read is factored into _read_rss_bytes() with its original semantics preserved (0 on a missing VmRSS line, None when /proc is unavailable) so it can be unit-tested off Linux.
Fixes#5740
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* stub heavy deps in the load-progress test and guard a valueless VmRSS
Two review fixes:
1. The new test imported core.inference.llama_cpp at module top, which pulls in
loggers/structlog/httpx and fails collection with ModuleNotFoundError in the
lightweight backend test env when the file is run on its own. Stub loggers,
structlog and httpx via sys.modules.setdefault before the import, mirroring
test_llama_cpp_load_progress_matrix.py; setdefault keeps the real modules when installed. Verified the file now collects and passes with only pytest present.
2. Catch IndexError in _read_rss_bytes: a "VmRSS:" line with no value column
would make line.split()[1] raise and crash a load-progress poll. Return None
instead, with a test for the valueless line.
* Hold load-progress high-water mark and explain a never-healthy load (#5740)
load_progress() now holds a per-process VmRSS high-water mark, so the bar
no longer regresses to ~8% when -ngl offloads the weights and frees the
mmap pages mid-load.
A live server that never returns 200 on /health now gets a specific error
(context/VRAM too large, or a local proxy/VPN intercepting the loopback
probe) instead of the generic invalid-GGUF/out-of-memory message.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Hakan Baysal <hakan.baysal@trmix.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* feat: add mlx public trainer api
* test: cover mlx public trainer api
* fix: preserve mlx epoch trainer configs
* fix: pass mlx warmup ratio through config
* fix: align mlx trainer dataset order
* fix: keep mlx chat templates import-light
* fix: infer mlx trainer context length
* fix: mirror cuda mlx context defaults
* fix: align mlx notebook trainer defaults
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: keep mlx public helpers import-light
* refactor: reuse mlx optimizer normalization
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: address mlx review feedback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: tighten mlx training argument parity
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: align mlx trainer eos default
* Fix MLX trainer to accept DataCollatorForSeq2Seq and handle TokenizerWrapper in get_chat_template
* Trim redundant docstrings on internal MLX helpers
* MLX review fixes: Studio optimizer import-safe on non-MLX hosts, preserve explicit max_length, skip MLX tests before import
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* MLX review round 2: defer max_length to model context, optimizer alias fallback for older zoo, skip non-MLX test on missing GPU deps
* MLX review round 3: keep chat_templates importable without torch on MLX
* fix: preserve MLX trainer notebook shims
* fix: ignore CUDA tokenizer moves on MLX
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: harden MLX trainer shims
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: unwrap MLX scheduler enum args
* fix: coerce integral MLX epoch counts
* fix: spoof CUDA compatibility APIs on MLX
* fix: harden MLX notebook compatibility shims
* MLX: add torch.cuda.mem_get_info to the compatibility shim
Notebook memory cells call torch.cuda.mem_get_info()[0] directly (not gated by
is_available), so on MLX it raises without a shim. Return (free, total) bytes
from the MLX device stats, consistent with the other torch.cuda compat helpers,
and add a matching assertion to the compat-API test.
* MLX: use active memory for mem_get_info; fix BatchEncoding.to keyword device
Address review on the MLX compatibility shim:
- torch.cuda.mem_get_info() now derives free bytes from current active MLX
memory instead of the peak high-water mark, so a capacity check stays
accurate after a transient spike or a prior run.
- BatchEncoding.to(device=...) passed by keyword no longer forwards a positional
None alongside the keyword (which raised "multiple values for 'device'"), so
non-CUDA keyword moves like .to(device="cpu") delegate correctly.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* MLX: accept preserve_dataset_order; stub RL trainers with a clear error
Two fixes so unmigrated notebooks behave predictably on MLX (torch present):
- preserve_dataset_order is a real MLXTrainingConfig field but was missing from
the extra-argument allowlist, so passing it (as a config or trainer kwarg)
could be rejected as unknown on a zoo without the field. Add it to
_MLX_IMPLEMENTED_EXTRA_ARGUMENTS so the documented no-shuffle path is reachable.
- GRPO/DPO/ORPO (and KTO/PPO/Reward) have no MLX trainer yet. Retarget the ones
the installed trl exposes to a stub that raises a clear 'not supported on MLX'
error instead of importing the real torch/CUDA trainer and crashing deep
inside it. Only existing trainers are retargeted (no invented attributes),
idempotent across re-imports.
* MLX: make RL-trainer stubbing import-safe; back current-memory APIs with active memory
Address review on the MLX shims:
- The RL-trainer stub loop probed trl with getattr(_trl, name), which triggers
trl's lazy trainer import and pulls torch -- that can crash import unsloth on a
torch-free MLX install just to check existence. Decide what to stub from
trl.__all__ + already-materialized attrs (vars) instead; never resolve the real
trainer. All trl trainer names are in __all__, so they are still stubbed (even
torch-free), and the probe no longer imports torch.
- torch.cuda.memory_reserved / memory_allocated (the current, non-max APIs) were
aliased to peak max_memory_reserved. Back them with current active MLX memory so
cleanup / capacity checks see live usage; max_* keep the peak high-water mark.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* MLX: keep TRL's SFTConfig epoch default under the trl.SFTConfig alias
Unmigrated notebooks import SFTConfig from trl, which the MLX build aliases to
the public training-args class. TRL/HF SFTConfig defaults to num_train_epochs=3
(max_steps=-1); the native MLX config defaults to max_steps=60. So an SFTConfig
built without an explicit length silently ran 60 MLX steps instead of TRL's 3
epochs under the alias. Alias trl.SFTConfig to a thin subclass that seeds the
TRL epoch default only when neither max_steps nor num_train_epochs is given;
explicit lengths pass through untouched, and the native public args class keeps
its MLX default. Epoch mode is supported by the MLX trainer.
* MLX CI: keep the GGUF reload smoke under the job timeout
The RELOAD-GGUF-via-llama-cli step timed out at 300s. BF16 GGUF decode is
CPU-bound on the macOS runner (~10s+/token), so generating 24 tokens landed
right on the 300s cliff and killed the process. This step is a save/reload
integrity smoke (it only needs a few chars of output), so the token count is
incidental: generate 8 tokens with explicit threads and a small headroom on the
subprocess timeout, all env-tunable (UNSLOTH_GGUF_RELOAD_N / _THREADS /
_TIMEOUT). Cuts the reload well under the 25 minute job budget.
* MLX: broaden trainer stubs, real peak-memory reset, fix shim tests
Address review on the MLX public API:
- The SFTConfig identity tests asserted trl.SFTConfig is UnslothTrainingArguments,
but the alias now points at the _MLXSFTConfig subclass that preserves TRL's
epoch default, so the MLX suite failed before testing the shim. Assert
issubclass instead.
- torch.cuda.reset_peak_memory_stats was a no-op, so max_memory_reserved kept
earlier model-load peaks across a scoped run. Wire it to mx.reset_peak_memory
with the same core/metal fallback used for the reads.
- The unsupported-trainer stubs were a fixed list, so trainers outside it (a
newer RLOOTrainer) still routed to the real torch trainer. Derive the set from
trl.__all__ (every non-SFT *Trainer) so all non-SFT surfaces fail with a clear
MLX message; names come from __all__ so trl is never resolved.
- The non-MLX export smoke skipped only on missing bitsandbytes/triton; other
absent GPU deps (numpy/torch/unsloth-zoo, or _gpu_init re-raising ImportError)
made it fail on CPU hosts. Skip on any ImportError.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: keep MLX notebook compatibility minimal
* MLX CI: force CPU + small context for the GGUF reload smoke
The RELOAD-GGUF-via-llama-cli step timed out even at 8 tokens (>420s), so it is a
fixed hang, not per-token cost: on the paravirtual macOS runner GPU llama.cpp's
Metal backend stalls, and the gemma3 GGUF advertises a 32768 context that llama-cli
would otherwise fully allocate. Run llama-cli CPU-only (-ngl 0) with a small context
(-c 256); keep generation short. All env-tunable (UNSLOTH_GGUF_RELOAD_NGL / _CTX /
_N / _THREADS / _TIMEOUT). Also print llama.cpp's partial stdout/stderr on timeout so
a future hang is diagnosable instead of an opaque TimeoutExpired.
* MLX CI: export the reload-smoke GGUF as q8_0, not bf16
The GGUF reload via llama-cli timed out on the runner even CPU-only with a tiny
context and 8 tokens. Root cause is the format, not the flags: the smoke exported
quantization_method='not_quantized', which maps to a bf16 GGUF, and llama.cpp's
bf16 CPU decode is unusably slow on the paravirtual macOS runner. Export q8_0
(fast_quantized, the exporter default and what users deploy) instead -- llama.cpp
has optimized q8_0 CPU kernels, so the fresh-process reload loads and generates in
seconds. The reload stays CPU-only (-ngl 0) with a small context.
* test: clear TRL shim before availability check
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Add --with-llama-cpp-dir flag to install.ps1 and install.sh
Users can now pass --with-llama-cpp-dir /path/to/llama.cpp to the
installer to skip downloading or building llama.cpp and use a local
directory instead. A junction (Windows) or symlink (Linux/macOS) is
created at the canonical install location, bypassing both the prebuilt
download (Phase 3) and source build (Phase 4) steps in setup.ps1/setup.sh.
The path is passed via UNSLOTH_LOCAL_LLAMA_CPP_DIR env var which
setup.ps1 and setup.sh read directly.
Ported from the idea in unslothai/unsloth#4384, reimplemented against
current Studio architecture.
* test: add static wiring test for --with-llama-cpp-dir flag
Cross-checks install.sh, install.ps1, studio/setup.sh and studio/setup.ps1
so the flag's contract (parse -> UNSLOTH_LOCAL_LLAMA_CPP_DIR env var -> link
local dir, skip prebuilt download and source build) can't silently regress.
Wired into studio-backend-ci.yml alongside the other tests/sh installer tests.
* Address review feedback on --with-llama-cpp-dir flag
- setup.ps1: delete an existing junction/symlink via DirectoryInfo.Delete()
instead of a recursive remove, which can traverse the link and wipe the
user's real llama.cpp directory on PowerShell 5.1.
- setup.ps1: short-circuit the build chain when a local dir is linked so CMake
never runs inside the user's checkout when it lacks a Windows-layout binary.
- install.sh / setup.sh: resolve paths with CDPATH= cd -P so a set CDPATH
cannot corrupt the resolved path.
- install.sh: seed _WITH_LLAMA_CPP_DIR from UNSLOTH_LOCAL_LLAMA_CPP_DIR so an
exported env var (piped-install style) is honored instead of being clobbered.
- setup.sh: create the root llama-quantize shim when linking a local source
build so GGUF export's check_llama_cpp() still finds it.
- setup.sh / setup.ps1: drop a stale link before the custom-home ownership
assert so re-runs with the flag stay idempotent.
- test: pin the new linked-dir build short-circuit.
* Harden --with-llama-cpp-dir against Codex/Gemini review findings
- install.sh: error when --with-llama-cpp-dir is the final arg with no path,
matching the existing --package/--python post-loop guards (was a silent
fallback to the normal prebuilt/source install).
- studio/setup.sh: canonicalize LLAMA_CPP_DIR before the self-link no-op
compare. _RESOLVED_LOCAL is fully resolved while LLAMA_CPP_DIR was textual,
so a symlinked $HOME made the guard miss and the rm -rf could wipe the
user's real llama.cpp tree.
- studio/setup.sh: make the llama-quantize shim non-fatal; it writes through
the link into the user's tree, which may be read-only (shared/CI cache),
and under set -e a failed ln aborted an otherwise-good reuse.
- studio/setup.ps1: detect a broken junction via Get-Item -Force instead of
Test-Path so a dangling link from a prior run is removed and mklink can
relink to a new valid directory.
- studio/setup.ps1: use Copy-Item -LiteralPath so a source path containing
[ ] isn't treated as a wildcard in the junction copy fallback.
- tests: update the wiring assertions for the LiteralPath copy and the
canonicalized compare.
* Validate/reuse local llama.cpp tree and guard the in-use case
Addresses the second Codex pass on the --with-llama-cpp-dir flag:
- Validate the linked tree before disabling installs (setup.sh + setup.ps1):
reusing a local dir skips BOTH the prebuilt download and the source build,
so the dir must already contain a runnable llama-server (build/bin on
Linux/macOS, build\bin\Release\llama-server.exe on Windows). Bail out with a
clear message instead of linking an unbuilt/wrong-platform checkout and
leaving Studio with no usable binary.
- Treat a canonical-path target as already linked when it holds a build
(setup.sh + setup.ps1): point the flag at ~/.unsloth/llama.cpp itself and an
existing build is reused (skip prebuilt + source) rather than clobbered by
the staged prebuilt installer (which uses os.replace()/replace). An empty
canonical dir still falls through to the normal in-place install.
- Abort when an in-use llama.cpp can't be removed on Windows (setup.ps1):
Remove-Item -ErrorAction SilentlyContinue can silently leave a locked tree
in place; detect that and stop with the same active-process message + exit 3
the prebuilt path uses, instead of junctioning over a half-present dir.
Left as follow-up (already tracked by the PR author as a non-blocker): the
in-app "Update llama.cpp" updater does not yet recognize a local-link install
as externally managed; that fix belongs in studio/backend/utils/llama_cpp_update.py.
* Accept all backend llama-server layouts in --with-llama-cpp-dir validation
The linked-tree validation only accepted build/bin[/Release]/llama-server, but
LlamaCppBackend._layout_candidates() resolves a root-level llama-server first,
then build/bin, then build/bin/Release on Windows. A `make` build or a flat
release extract (binary at the dir root) was therefore rejected with a hard
installer failure even though Studio would have run it.
Validate the same candidate set the backend uses in both setup scripts, and add
wiring-test assertions so the check can't silently narrow again.
* Treat --with-llama-cpp-dir local links as externally managed
A --with-llama-cpp-dir install junctions/symlinks the canonical llama.cpp dir to
the user's own checkout, but two backend paths still treated it as a Studio-owned
tree:
- The in-app updater (llama_cpp_update) offered and could apply an official
prebuilt over the link, writing through it into the user's checkout (or
failing) and silently dropping the link the flag created.
- Orphan cleanup (LlamaCppBackend._kill_orphaned_servers) resolved the linked
root into its kill allowlist, so a llama-server the user launched from the same
checkout was classified as ours and killed on startup.
Detect the canonical dir being a symlink/junction (reparse point) and treat the
install as unmanaged: get_update_status reports unsupported, start_update refuses
with reason "local_link", and the linked root is left out of the orphan
allowlist. Adds behavioral tests (link vs plain dir, updater refusal, and the
spared-vs-killed orphan control).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Add behavioral shell test for --with-llama-cpp-dir linking
The existing tests/sh/test_with_llama_cpp_dir_flag.sh is a static grep of the
scripts. This adds a behavioral test that extracts the real link block from
studio/setup.sh (by content anchors, with a self-validating extraction) and runs
it against hermetic fake dirs, asserting the outcomes that matter:
- an external CMake build links and arms neither the prebuilt download nor the
source build
- a flat / make tree (root-level llama-server, no build/bin) is accepted too
- an unbuilt tree is rejected with a non-zero exit and no link left behind
- relinking over a stale link preserves the target's contents (no data loss)
- pointing at the canonical path is a no-op reuse, not a self-referential link
Symlink-identity checks run only where real symlinks exist (skipped on Windows
git-bash copy-mode); the link/skip/no-data-loss checks run everywhere. Wired into
studio-backend-ci.yml next to the static test.
* Install psutil in backend CI so orphan-cleanup tests run
The new orphan-cleanup tests import psutil for the process scan, but the Backend
CI deps step installed studio.txt plus a fixed extras list that omits it, so the
two tests failed with ModuleNotFoundError. Add psutil to both backend pytest dep
steps (kept in shared shape), and guard the import with pytest.importorskip so a
minimal env without psutil skips these tests instead of erroring.
---------
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* feat: Implementation of the Portuguese (Brazil) language and VRAM/RAM monitor.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/frontend/src/hooks/use-gpu-utilization.ts
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/backend/main.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/backend/main.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/backend/utils/hardware/hardware.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/backend/utils/hardware/hardware.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/frontend/src/features/settings/components/usage-examples.tsx
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update studio/backend/main.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/backend/utils/hardware/hardware.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Update studio/frontend/src/features/studio/sections/progress-section.tsx
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* fix: resolve automated review feedback on API shape
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix review issues for PR #6509: Cpu icon, VRAM percent, system polling
- model-inspector: use the exported CpuIcon (Cpu is not a Hugeicons export)
- app-sidebar: guard the VRAM percent on totalVram to avoid Infinity, and
reset the system poll cache only after each request settles so a slow probe
is reused instead of stacking overlapping requests
- use-gpu-info: populate CPU/RAM on hosts without a GPU
- progress-section: label GPUs by visible_ordinal instead of array index
- hub-page: base the RAM label on systemRamTotalGb
- usage-examples: emit JS sampling and tool options at the top level instead
of nesting them under extra_body (the JS SDK does not unwrap extra_body)
- main: read torch and transformers versions from package metadata instead of
importing the libraries on every system poll, and guard the VRAM math
against null values
- hardware: translate a leftover comment to English
* Harden /api/system: guard psutil.boot_time for PR #6509
Simulating restricted containers and some VMs (where psutil.boot_time can raise)
showed the /api/system endpoint would 500 on the unguarded boot_time call, the
same failure class already handled for cpu_freq, disk_usage, and Process. Wrap
boot_time and return uptime_seconds as null when it is unavailable so the sidebar
monitor degrades gracefully instead of breaking. Widen the uptime_seconds type to
number | null to match.
* Studio: make the sidebar hardware monitor a toggle (default on) for PR #6509
Adds a "Show hardware monitor" switch under Settings > Appearance > Layout,
backed by a localStorage preference (default on), mirroring the existing
useSidebarPin pattern. When turned off, the sidebar hides the VRAM/RAM meters
and useSystemInfo stops the 3s /api/system poll entirely, so no nvidia-smi /
SMI probes run while the monitor is disabled. Adds the en and pt-BR strings.
* Studio: default the sidebar hardware monitor to off (opt-in) for PR #6509
* Studio pt-BR: fix three small translation defects for PR #6509
- learningRateDescription: "5e-5 for CPT" -> "5e-5 para CPT" (leftover English)
- exportScopeRecents: "Recents" -> "Recentes" (untranslated)
- relativeMonthsAgo/relativeYearsAgo: add the missing space ("há {count} meses"/
"há {count} anos") so they no longer render as "há 3meses"
* Studio pt-BR: translate the last 10 fallback keys for PR #6509
Adds the settings.general.storage block (Armazenamento) and the
settings.chat.modelDisclaimer pair, so pt-BR now covers all en keys
(679/679) with no English fallbacks.
* Studio: hide sidebar VRAM row on CPU-only hosts for PR #6509
* Studio: tighten and trim code comments for PR #6509
* fix: UI issue in the stop button dialog box (fine-tuning)
* Studio pt-BR: translate 18 new keys from main merge (password dialog, GGUF export, dataset streaming) for PR #6509
* Rounding to GB
* Fix/adjust System resources tab for PR #6509
* Fix/adjust GPU monitor review items for PR #6509
* Fix/adjust remaining GPU monitor review items for PR #6509
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix/adjust MLX resource fallback for PR #6509
* floating window implementation
* resize for floating window
* Fix resource monitor review items
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Restore frontend optional dependency lock entries
* Make GPU selection tests hermetic
* Fix GPU monitor CI test failures
* Bound MLX GGUF reload smoke
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix MLX GGUF reload smoke exit
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: self-heal a pre-#6483-fix anyio>=4.14 stuck in existing installs
The <4.14 cap in constraints.txt/no-torch-runtime.txt only constrains new
anyio resolutions. An install made before that cap existed can already be
sitting on anyio 4.14+, and since it already satisfies mcp/fastmcp's
anyio>=4.5 floor, every later constrained install skips it as
already-satisfied -- so affected installs never recover and keep hitting
the cancel-scope RuntimeError on every request (#6797, a recurrence of
#6483). Force-reinstall anyio<4.14 whenever a stuck 4.14+ is detected.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: also repair anyio on the update fast path
setup.sh's _SKIP_PYTHON_DEPS and setup.ps1's $SkipPythonDeps skip
install_python_stack.py entirely once the installed package version already
matches PyPI latest, so an install stuck on anyio>=4.14 with an otherwise
up-to-date package never reaches the repair added in install_python_stack.py.
Probe anyio on that fast path too and fall through to the full dependency
pass when it's still >=4.14, mirroring the existing ROCm/CPU-torch override
right below it.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add customizable RAG embedding model setting and reorganize settings tabs
Chat with files, project sources, and knowledge bases previously always
embedded with unsloth/bge-small-en-v1.5. This adds a Settings option to
pick any Hugging Face embedding model (or local path), with HF search
autocomplete, server-side verification that the repo is actually an
embedding model, and a save anyway escape hatch for offline or local
models. The setting persists in app_settings and applies at runtime to
both the sentence-transformers and llama-server GGUF embedder backends
without a restart.
Also reorganizes the General settings tab: Documents & RAG sits above
Uploads, Helper LLM moved above the danger zone, and Model auto-switch
(OpenAI API) moved to the bottom of the API tab.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Support local model paths on the GGUF embedder and normalize default saves
Found by simulation testing of the embedding model setting:
Local paths saved as the embedding model now work on the llama-server
GGUF backend (the default backend on macOS and CPU). A path to a .gguf
file is used directly and a directory is scanned for a variant-matching
non-mmproj .gguf, with a clear error when none exists. Previously a
local path was sent to the HF hub API and failed with a repo lookup
error.
Saving the default model explicitly no longer stores an override, so
is_custom stays false and the UI does not show a reset button for the
default value.
* Address review: stale-vector handling, GGUF derivation, save-time guards
Review follow-ups, each verified by new tests:
Re-uploading a document after an embedding model change now re-indexes
instead of deduping by content hash. Documents record the embedder that
produced their vectors (lazy embedding_model column, NULL legacy rows
keep deduping) and a mismatch replaces the old document.
A vector width change no longer bricks the dense index. ensure_vec
drops and recreates chunks_vec when the dim changes (old vectors are in
a foreign space and only block inserts) and search_dense returns empty
on a width mismatch instead of surfacing a vec0 error, so lexical
search keeps working until documents are re-uploaded.
Saving a local sentence-transformers folder with no .gguf now returns
409 with a clear message when the install embeds via llama-server,
instead of failing at first index. force still saves.
A custom RAG_EMBEDDING_MODEL env without RAG_EMBED_GGUF_REPO now
derives the -GGUF companion repo instead of silently keeping the bge
GGUF on CPU and macOS installs.
The resolved GGUF path is tagged with the repo captured at entry, so a
setting change during a download cannot mark the old model as current.
GGUF repo detection matches gguf as a whole name segment rather than a
substring, hf_token is trimmed before verification, and the settings
combobox drops a redundant state mirror of its controlled value.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Shrink embedding model font to 11px in the input and dropdown
The combobox wrapper applies className to the outer input group, so the
size utility must target the inner input element; the previous text-xs
never reached it and the field rendered at the browser default.
* Show curated unsloth embedding models when the search field is empty
The empty-query listing was the global top-downloads page, which holds
no unsloth mirrors for the unsloth-first float to reorder, so the
dropdown opened on third-party models. Match the model picker: curated
unsloth listing when empty, whole-Hub search once a query is typed.
* Address review: settings resilience and index consistency
Keep the last known embedding model on settings store errors, remove the
re-entrant dim lock in the llama-server backend, accept local GGUF saves
and verify GGUF availability for HF repos on that backend, match local
path embedders exactly in model list filters, drop same-width stale
vectors from dense search, pin the embedder per ingestion job, and only
replace completed documents after the re-index succeeds.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Consolidate the GGUF repo derivation tests
* Trim to a single core embedding-model test
* Address review: GGUF repo saves and cache race
Accept a GGUF-named HF repo on the llama-server backend by verifying GGUF
availability instead of the sentence-transformers metadata gate, and guard
the settings cache with a generation counter so a read overlapping a save
cannot repopulate it with the pre-save value.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio RAG: fix RTL/Indic PDF corruption and dropped DOCX tables
The RAG parser prefers pymupdf4llm.to_markdown for PDFs, but that rebuilds text from
positioned glyphs and mangles complex-shaping scripts (RTL Arabic/Hebrew come back as
shaped Presentation Forms, Indic matras drop to U+FFFD) and can silently drop most of a
heavy-RTL page. _pdf now compares the Markdown against PyMuPDF's logical-order
get_text() per page and falls back to it when the Markdown looks corrupted (shaped
Presentation Forms or U+FFFD above a small floor/ratio) or holds far fewer letters than
the raw layer. Latin PDFs are unaffected and keep their Markdown tables/headings.
_docx walked document.paragraphs, which excludes table cells, so DOCX tables were
dropped entirely. It now walks body content in document order via iter_inner_content,
emitting each table row as pipe-joined cells (deduped across merged cells); the preview
locator already anchors on pipes.
Adds parser tests for the corruption and incompleteness fallbacks and for DOCX table
extraction. These mirror the chat document-extractor guard raised in the unslothai/
unsloth#5351 review; the RAG parser is a separate module and needed its own fix.
* RAG DOCX: keep empty table cells and collapse in-cell newlines
Skipping empty cells shifted later cells left and broke column alignment across rows;
a cell with internal paragraphs (newlines) also broke the pipe-joined row. Keep every
cell (dropping the row only when all are empty) and normalize each cell with
" ".join(split()) so multi-paragraph cells stay on one row. Adds a test for both.
* RAG DOCX: dedup merged table cells on the <w:tc> element directly
Store the shared <w:tc> lxml element in the seen set instead of its id(); it is
hashable and compares by the underlying node, so it dedups spanned/merged cells the
same way without relying on id(). Adds a merged-cell test.
* RAG DOCX: align merged cells, pad skipped grid columns, flatten nested tables
* RAG DOCX: walk cells in document order so nested tables keep in-cell position
* RAG DOCX: dedup vertically merged cells so a spanning label is indexed once
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Add a shared fits-on-device filter to the model selects
The chat model selector gains an Only show models that fit on this
device tick under its filter row, and the Hub page gains a matching
Fits device pill next to the sort menu. Both read one persisted
preference (unsloth_models_fit_on_device_only), so toggling either
applies to both.
The filter reuses the Recommended sort's existing fit math, extracted
into hfModelFitsDevice: size from safetensors metadata, GGUF param
count, or the repo name, against the 0.7 GPU + 0.7 RAM budget, with
unsizable models hidden. In the chat selector it extends the fit
filtering to the Trending and Recent sorts and to search results;
downloaded models stay visible regardless. An unknown device budget
keeps everything. The preference is cleared by Reset all local
preferences like the other picker toggles.
* Move the device-fit toggle into the sort dropdowns
* Tighten sort menu footer spacing and shorten the label
* Align the footer checkbox with the option text
* Make the footer checkbox circular with a smaller tick
* Clear menu highlight when the pointer leaves the options
* Address review: fit filter coverage and sizing
Exempt on-disk models from the Hub fit filter, apply it to the feed
trending rows and curated search results, size safetensors and MLX rows
by the quantized load estimate instead of checkpoint bytes, and replace
the native title hint with the app Tooltip.
* Make the whole device-fit row toggle the filter
pre-commit-ci[bot]