Unsloth is a local UI for training and running Gemma 4, Qwen3.6, DeepSeek, Kimi, GLM and other models. https://unsloth.ai/docs
  • Python 71.5%
  • TypeScript 22.7%
  • Shell 1.9%
  • PowerShell 1.6%
  • Rust 1.5%
  • Other 0.7%
Find a file
Daniel Han 3bfc83781d
Runtime MTP fallback for tensor parallelism (try MTP, recover if it crashes) (#6324)
* Disable MTP speculative decoding under tensor parallelism

Follow-up to #6040 (Studio tensor-parallel support).

MTP-draft speculative decoding plus --split-mode tensor crashes the CUDA
flash-attn kernel at decode time. The startup /health probe only checks that
llama-server comes up, so the existing MTP-drop fallback (keyed on startup
health) never fires and the server dies on the first generation instead.

Gate MTP off when a tensor attempt actually engages: this runs before the
VRAM planner (so no drafter memory is reserved) and before the speculative
flag build (so no --model-draft / --spec-type is emitted). Ngram modes use no
draft model and are kept, and mtp+ngram degrades to ngram rather than off. The
layer-split fallback re-runs with tensor_parallel False and restores MTP.

The reason is surfaced as spec_fallback_reason "tensor_parallel" so the
settings sheet explains why MTP is off instead of prompting a llama.cpp update.

Verified on unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL across 4x B200: the
load now emits --split-mode tensor with no MTP flags and generation completes
without the prior decode crash.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Make tensor-parallel MTP gate test format-independent

The assertion pinned the multi-line `speculative_type = (` form, but ruff
collapses it onto one line, so match `speculative_type =` instead.

* Recover from MTP+tensor-parallel crashes at runtime instead of banning MTP

MTP-draft speculative decoding under --split-mode tensor usually works, but can
crash llama-server's CUDA flash-attn kernel at decode time (the prompt-cache
checkpoint-restore path). The earlier fix statically disabled MTP whenever
tensor parallelism was on, which is not future-proof and gives up the MTP
speedup even though it normally works.

Replace the static ban with a try/recover, mirroring the existing load-time
MTP-drop fallback:

- Load-time decode probe: after the server passes /health under tensor +
  MTP, run one tiny /completion to exercise the draft path. A failure flips
  the load unhealthy so the existing fallback respawns with --spec-default.
  Catches a hard incompatibility that crashes on the first decode.

- Generation-time recovery: snapshot the load kwargs after a healthy load,
  and if llama-server exits mid-generation while MTP + tensor parallelism were
  active, quietly reload the same model with speculative decoding off (one
  single-flight background reload) and surface spec_fallback_reason=runtime_error.
  Catches the rare mid-generation crash the probe and load-time fallback miss.

No persistent ban: a later fresh load re-tries MTP, so this self-heals if a
future llama.cpp supports the combo. Verified on gemma-4-26B-A4B + 4x B200:
MTP runs normally, and killing llama-server mid-generation reloads it without
MTP and serves the next request cleanly.

* Address review feedback on the MTP runtime fallback

- Authenticate the decode probe: direct-stream mode runs llama-server with
  --api-key, so the unauthenticated /completion probe got a 401 and falsely
  dropped MTP. Attach the same bearer auth the other internal requests use.
- Re-check the cancel flag inside the recovery thread after the death poll,
  so an /unload that races the reload can't resurrect the dropped model.
- Schedule the no-MTP recovery on the connection-error paths it was missing:
  generate_chat_completion's ConnectError branch, the OpenAI passthrough
  typed (RemoteProtocolError/ReadError/CloseError) stream catch, and the
  Anthropic passthrough generic stream catch. Previously a server that died
  before reconnect, or a typed mid-stream error, skipped the reload.

* Cover every request path with the MTP+tensor crash recovery via a watchdog

The runtime MTP-crash recovery only fired from request handlers that
observed the failure, so the direct llama-server proxy endpoints
(/v1/completions, /v1/responses, the OpenAI/Anthropic passthrough
transports) -- and a crash with no request in flight -- could leave a
dead server. Add a single background watchdog, armed only on a healthy
MTP + tensor-parallel load, that polls the subprocess and routes an
unexpected death into the existing single-flight no-MTP reload. It is
stopped inside _kill_process (the one deliberate-termination chokepoint)
so a planned reload/unload is never mistaken for a crash, and re-checks
the stop flag after a detected exit to close the kill-vs-poll race. The
reload turns MTP off, so the replacement server arms no watchdog and the
fallback cannot loop; a later fresh load still re-tries MTP.

* Harden MTP+tensor crash recovery: stale-load race, pass-through MTP, requested mode

Address review findings on the runtime MTP-crash recovery:

- Stale-load race: the recovery thread snapshotted the crashed load, waited up
  to 5s for the process to confirm dead, then only checked the cancel flag
  before replaying load_model. A concurrent user load clears that flag, so the
  stale snapshot could reload the old model over the user's new one. Make the
  load lock re-entrant and run the staleness check (cancel + same process +
  unchanged snapshot) under it, atomically with the reload.

- Pass-through MTP: MTP can also be requested via a user --spec-type in
  extra_args or LLAMA_ARG_SPEC_TYPE, where Studio emits no spec flags and
  _speculative_type stays unset, so the probe/watchdog/recovery never engaged.
  Track _mtp_runtime_fallback_active from the actual launched config and gate on
  it; on the no-MTP reload, append a last-wins --spec-default so the replay drops
  MTP regardless of source (and the load-time fallback does the same).

- Requested mode: the off-reload reset _requested_spec_mode to off, so after a
  status refresh the UI showed a bare Off with the runtime-error note suppressed
  and would not retry MTP. Restore the original requested mode after the reload,
  matching the startup MTP fallback.

- Snapshot the extra_args list by value so a caller mutating it cannot corrupt
  the recovery snapshot.

Tests: test_tensor_parallel.py + test_llama_server_args.py green (303 passed).

* Trim verbose comments in the MTP+tensor crash recovery

Tighten the docstrings and inline comments added for the runtime MTP recovery
(watchdog, probe, reload, gating) to succinct one/two-line forms; no code
change (verified comment-only).

---------

Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-06-18 05:42:21 -07:00
.github Studio: remove the Windows VBS launcher to clear the Kaspersky false positive (#6326) 2026-06-16 04:00:18 -07:00
images images: use narrower Discord button and drop duplicate (#5552) 2026-05-18 05:00:59 -07:00
scripts Package scanners: cut false positives and make the CI gate blocking (#6355) 2026-06-16 01:46:15 -07:00
studio Runtime MTP fallback for tensor parallelism (try MTP, recover if it crashes) (#6324) 2026-06-18 05:42:21 -07:00
tests Reduce and tighten comments and docstrings across the test suite (#6429) 2026-06-18 01:07:09 -07:00
unsloth Shim removed vllm.transformers_utils.tokenizer so fast_inference works on vLLM >= 0.22 (#6390) 2026-06-17 17:25:41 -07:00
unsloth_cli CLI: add unsloth connect to point coding agents at a local Studio server (#6407) 2026-06-17 19:26:55 +01:00
.git-blame-ignore-revs chore(studio/frontend): normalize line endings to LF (#6012) 2026-06-12 03:51:59 -07:00
.gitattributes chore(studio/frontend): normalize line endings to LF (#6012) 2026-06-12 03:51:59 -07:00
.gitignore ci: advisory lockfile supply-chain audit (no install-script changes) (#5604) 2026-05-19 05:56:56 -07:00
.pre-commit-ci.yaml pre-commit CI config (#3565) 2025-11-07 14:44:18 -08:00
.pre-commit-config.yaml [pre-commit.ci] pre-commit autoupdate (#6343) 2026-06-15 12:43:26 -07:00
build.sh Add Studio web update banner and release version display (#5308) 2026-05-11 18:24:01 +04:00
cli.py Rename cli/ to unsloth_cli/ to fix namespace collision with stringzilla (#4393) 2026-03-17 20:40:21 -07:00
CODE_OF_CONDUCT.md Update CODE_OF_CONDUCT.md 2025-10-25 19:31:05 -07:00
CONTRIBUTING.md docs: repository cleanup (#5617) 2026-06-12 11:07:04 +01:00
COPYING Rename cli/ to unsloth_cli/ to fix namespace collision with stringzilla (#4393) 2026-03-17 20:40:21 -07:00
install.ps1 Studio: remove the Windows VBS launcher to clear the Kaspersky false positive (#6326) 2026-06-16 04:00:18 -07:00
install.sh Studio: remove the Windows VBS launcher to clear the Kaspersky false positive (#6326) 2026-06-16 04:00:18 -07:00
LICENSE Rename cli/ to unsloth_cli/ to fix namespace collision with stringzilla (#4393) 2026-03-17 20:40:21 -07:00
pyproject.toml Bug fixes 2026-06-13 04:43:27 -07:00
README.md Document install env vars in README advanced launch options (#5972) 2026-06-03 05:39:38 -07:00
unsloth-cli.py fix(unsloth-cli): route hub_path/hub_token correctly in --push_model save block (#6346) 2026-06-17 03:05:30 -07:00

Unsloth logo

Unsloth Studio lets you run and train models locally.

FeaturesQuickstartNotebooksDocumentation


unsloth studio ui homepage

Get started

macOS, Linux, WSL:

curl -fsSL https://unsloth.ai/install.sh | sh

Windows:

irm https://unsloth.ai/install.ps1 | iex

Community:

Features

Unsloth Studio (Beta) lets you run and train text, audio, embedding, vision models on Windows, Linux and macOS.

Inference

Training

  • Train and RL 500+ models up to 2x faster with up to 70% less VRAM, with no accuracy loss.
  • Custom Triton and mathematical kernels. See some collabs we did with PyTorch and Hugging Face.
  • Data Recipes: Auto-create datasets from PDF, CSV, DOCX etc. Edit data in a visual-node workflow.
  • Reinforcement Learning (RL): The most efficient RL library, using 80% less VRAM for GRPO, FP8 etc.
  • Supports full fine-tuning, RL, pretraining, 4-bit, 16-bit and, FP8 training.
  • Observability: Monitor training live, track loss and GPU usage and customize graphs.
  • Multi-GPU training is supported, with major improvements coming soon.

📥 Install

Unsloth can be used in two ways: through Unsloth Studio, the web UI, or through Unsloth Core, the code-based version. Each has different requirements.

Unsloth Studio (web UI)

Unsloth Studio (Beta) works on Windows, Linux, WSL and macOS.

  • CPU: Supported for Chat and Data Recipes currently
  • NVIDIA: Training works on RTX 30/40/50, Blackwell, DGX Spark, Station and more
  • macOS: Training, MLX and GGUF inference are ALL supported.
  • AMD: Chat + Data works. Train with Unsloth Core. Studio support is out soon.
  • Multi-GPU: Available now, with a major upgrade on the way

macOS, Linux, WSL:

curl -fsSL https://unsloth.ai/install.sh | sh

Use the same command to update.

Windows:

irm https://unsloth.ai/install.ps1 | iex

Use the same command to update.

Launch

unsloth studio -p 8888

For cloud or global access, add -H 0.0.0.0. By default, Unsloth is accessible only locally.

Docker

Use our Docker image unsloth/unsloth container. Run:

docker run -d -e JUPYTER_PASSWORD="mypassword" \
  -p 8888:8888 -p 8000:8000 -p 2222:22 \
  -v $(pwd)/work:/workspace/work \
  --gpus all \
  unsloth/unsloth

Developer, Nightly, Uninstall

To see developer, nightly and uninstallation etc. instructions, see advanced installation.

Unsloth Core (code-based)

Linux, WSL:

curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv unsloth_env --python 3.13
source unsloth_env/bin/activate
uv pip install unsloth --torch-backend=auto

Windows:

winget install -e --id Python.Python.3.13
winget install --id=astral-sh.uv  -e
uv venv unsloth_env --python 3.13
.\unsloth_env\Scripts\activate
uv pip install unsloth --torch-backend=auto

For Windows, pip install unsloth works only if you have PyTorch installed. Read our Windows Guide. You can use the same Docker image as Unsloth Studio.

AMD, Intel:

For RTX 50x, B200, 6000 GPUs: uv pip install unsloth --torch-backend=auto. Read our guides for: Blackwell and DGX Spark.
To install Unsloth on AMD and Intel GPUs, follow our AMD Guide and Intel Guide.

📒 Free Notebooks

Train for free with our notebooks. You can use our new free Unsloth Studio notebook to run and train models for free in a web UI. Read our guide. Add dataset, run, then deploy your trained model.

Model Free Notebooks Performance Memory use
Gemma 4 (E2B) ▶️ Start for free 1.5x faster 50% less
Qwen3.5 (4B) ▶️ Start for free 1.5x faster 60% less
gpt-oss (20B) ▶️ Start for free 2x faster 70% less
Qwen3.5 GSPO ▶️ Start for free 2x faster 70% less
gpt-oss (20B): GRPO ▶️ Start for free 2x faster 80% less
Qwen3: Advanced GRPO ▶️ Start for free 2x faster 70% less
embeddinggemma (300M) ▶️ Start for free 2x faster 20% less
Mistral Ministral 3 (3B) ▶️ Start for free 1.5x faster 60% less
Llama 3.1 (8B) Alpaca ▶️ Start for free 2x faster 70% less
Llama 3.2 Conversational ▶️ Start for free 2x faster 70% less
Orpheus-TTS (3B) ▶️ Start for free 1.5x faster 50% less

🦥 Unsloth News

  • Connections: Connect any API provider (OpenAI, Anthropic) or server (vLLM, Ollama). Guide
  • MTP: Run Qwen3.6 MTP in Unsloth. MTP settings are autoset specific to your hardware. Guide
  • API inference endpoint: Deploy and run local LLMs in Claude Code, Codex tools. Guide
  • Qwen3.6: Qwen3.6-35B-A3B can now be trained and run in Unsloth Studio. Blog
  • Gemma 4: Run and train Googles new models directly in Unsloth. Blog
  • Introducing Unsloth Studio: our new web UI for running and training LLMs. Blog
  • Qwen3.5 - 0.8B, 2B, 4B, 9B, 27B, 35-A3B, 112B-A10B are now supported. Guide + notebooks
  • Train MoE LLMs 12x faster with 35% less VRAM - DeepSeek, GLM, Qwen and gpt-oss. Blog
  • Embedding models: Unsloth now supports ~1.8-3.3x faster embedding fine-tuning. BlogNotebooks
  • New 7x longer context RL vs. all other setups, via our new batching algorithms. Blog
  • New RoPE & MLP Triton Kernels & Padding Free + Packing: 3x faster training & 30% less VRAM. Blog
  • 500K Context: Training a 20B model with >500K context is now possible on an 80GB GPU. Blog
  • FP8 & Vision RL: You can now do FP8 & VLM GRPO on consumer GPUs. FP8 BlogVision RL

📥 Advanced Installation

The below advanced instructions are for Unsloth Studio. For Unsloth Core advanced installation, view our docs.

Developer installs: macOS, Linux, WSL:

git clone https://github.com/unslothai/unsloth
cd unsloth
./install.sh --local
unsloth studio -p 8888

Then to update :

cd unsloth && git pull
./install.sh --local
unsloth studio -p 8888

Developer installs: Windows PowerShell:

git clone https://github.com/unslothai/unsloth.git
cd unsloth
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\install.ps1 --local
unsloth studio -p 8888

Then to update :

cd unsloth && git pull
./install.sh --local
unsloth studio -p 8888

Nightly: MacOS, Linux, WSL:

git clone https://github.com/unslothai/unsloth
cd unsloth
git checkout nightly
./install.sh --local
unsloth studio -p 8888

Then to launch every time:

unsloth studio -p 8888

Nightly: Windows:

Run in Windows Powershell:

git clone https://github.com/unslothai/unsloth.git
cd unsloth
git checkout nightly
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\install.ps1 --local
unsloth studio -p 8888

Then to launch every time:

unsloth studio -p 8888

Advanced launch options

Installer options can be passed as environment variables. On macOS, Linux and WSL place the variable after the pipe so the shell passes it to sh; on Windows set it with $env: before piping to iex.

Skip PyTorch (GGUF-only mode):

curl -fsSL https://unsloth.ai/install.sh | UNSLOTH_NO_TORCH=1 sh
$env:UNSLOTH_NO_TORCH=1; irm https://unsloth.ai/install.ps1 | iex

Pin the Python version:

curl -fsSL https://unsloth.ai/install.sh | UNSLOTH_PYTHON=3.12 sh
$env:UNSLOTH_PYTHON='3.12'; irm https://unsloth.ai/install.ps1 | iex

Install to a custom location with UNSLOTH_STUDIO_HOME:

curl -fsSL https://unsloth.ai/install.sh | UNSLOTH_STUDIO_HOME=/abs/path sh
$env:UNSLOTH_STUDIO_HOME='C:\path'; irm https://unsloth.ai/install.ps1 | iex

Cap Studio's native CPU thread pools on high-core hosts: UNSLOTH_CPU_THREADS=8 unsloth studio -p 8888.

Uninstall

The recommended way to fully remove Unsloth Studio is the matching uninstall script for your OS. It stops any running servers, removes the install dir, the launcher data dir, the desktop shortcut, and any platform-specific entries (macOS .app bundle + Launch Services on Mac; Start Menu, HKCU\Software\Unsloth registry key and user PATH entries on Windows):

  • MacOS, WSL, Linux: curl -fsSL https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.sh | sh
  • Windows (PowerShell): irm https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.ps1 | iex

If you only want to drop the install dir and keep the launcher/shortcut for a later reinstall, you can instead run rm -rf ~/.unsloth/studio (Mac/Linux/WSL) or Remove-Item -Recurse -Force "$HOME\.unsloth\studio" (Windows). The model cache at ~/.cache/huggingface is not touched by any of these.

For more info, see our docs.

Deleting model files

You can delete old model files either from the bin icon in model search or by removing the relevant cached model folder from the default Hugging Face cache directory. By default, HF uses:

  • MacOS, Linux, WSL: ~/.cache/huggingface/hub/
  • Windows: %USERPROFILE%\.cache\huggingface\hub\
Type Links
  Discord Join Discord server
  r/unsloth Reddit Join Reddit community
📚 Documentation & Wiki Read Our Docs
  Twitter (aka X) Follow us on X
🔮 Our Models Unsloth Catalog
✍️ Blog Read our Blogs

Citation

You can cite the Unsloth repo as follows:

@software{unsloth,
  author = {Daniel Han, Michael Han and Unsloth team},
  title = {Unsloth},
  url = {https://github.com/unslothai/unsloth},
  year = {2023}
}

If you trained a model with 🦥Unsloth, you can use this cool sticker!  

License

Unsloth uses a dual-licensing model of Apache 2.0 and AGPL-3.0. The core Unsloth package remains licensed under Apache 2.0, while certain optional components, such as the Unsloth Studio UI are licensed under the open-source license AGPL-3.0.

This structure helps support ongoing Unsloth development while keeping the project open source and enabling the broader ecosystem to continue growing.

Thank You to

  • The llama.cpp library that lets users run and save models with Unsloth
  • The Hugging Face team and their libraries: transformers and TRL
  • The Pytorch and Torch AO team for their contributions
  • NVIDIA for their NeMo DataDesigner library and their contributions
  • And of course for every single person who has contributed or has used Unsloth!