* test(studio): add e2e test for cpu-fallback overriding vulkan
* feat(studio): add UNSLOTH_LLAMA_CPP_BACKEND env var
* feat(studio): add UNSLOTH_LLAMA_CPP_BACKEND env var
* Preserve UNSLOTH_LLAMA_CPP_BACKEND=cpu across llama.cpp updates for PR #7228
The in-app updater rebuilt the installer command without --cpu-fallback and
only re-asserted Vulkan, so accepting a llama.cpp update after forcing CPU on
an Intel iGPU host re-ran host detection and routed back to the crashing Vulkan
bundle (#7213). Record install_kind in the prebuilt marker and re-assert
--cpu-fallback on update when the installed bundle is CPU.
Also make setup.sh's UNSLOTH_LLAMA_CPP_BACKEND check case-insensitive to match
setup.ps1, and add tests for the updater CPU preservation and the setup.sh flag
plumbing.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim and validate UNSLOTH_LLAMA_CPP_BACKEND, warn on unknown values for PR #7228
Trim surrounding whitespace and lowercase the value in both setup.sh and
setup.ps1, so values like ' cpu ' or 'CPU' still force the CPU-only prebuilt.
An unrecognized value (e.g. 'gpu') now prints a warning instead of silently
falling back to auto. Extend test_setup_llama_cpp_backend.py to cover both
scripts, including trimmed, empty and unknown values.
* Preserve arm64 CPU installs on update and honor CPU override in Windows prune for PR #7228
The update-path CPU preservation only matched install_kind ending in -cpu, so
arm64 CPU bundles (linux-arm64, windows-arm64) were re-routed to a GPU or source
build on update. Match the full set of CPU-only kinds instead.
Persisting install_kind also activated the previously inert Windows
mismatch-prune in setup.ps1: on a GPU host with UNSLOTH_LLAMA_CPP_BACKEND=cpu it
saw the windows-cpu marker as mismatched and deleted it every rerun. Normalize
the override once and make CPU expected so a deliberate CPU install is kept.
Extend the tests to cover both.
* Document legacy llama.cpp markers keep heal-to-GPU on update for PR #7228
Legacy prebuilt markers written before install_kind was persisted intentionally
do not force --cpu-fallback on update: the in-app updater lets them re-resolve
(heal to a GPU bundle) per the existing behavior from #6097, and only markers
that explicitly record a CPU install_kind are pinned to CPU. Add a comment and a
regression case documenting the boundary.
* Tighten llama.cpp CPU-fallback comments for PR #7228
* Fix Windows install-prune to keep valid Intel/fallback bundles for PR #7228
Persisting install_kind activated the setup.ps1 mismatch-prune, whose
expectedKinds was incomplete: the non-NVIDIA/non-AMD branch omitted
windows-vulkan (the Intel auto-route) and the GPU branches omitted the
windows-cpu/windows-arm64 fallback the installer uses when a GPU prebuilt is
missing. That made every setup rerun delete and re-download a valid Intel Vulkan
(or CPU-fallback) install. List all kinds the installer can produce per host so
only a bundle the host cannot run is pruned. Cover the full matrix in tests.
* Persist force_cpu marker flag so only forced CPU installs re-assert on update for PR #7228
* Add --force-cpu for deliberate CPU installs and warn on macOS for PR #7228
* Record force_cpu when reusing a matching CPU bundle for PR #7228
* Accept force_cpu keyword in installer test validator fakes for PR #7228
---------
Co-authored-by: danielhanchen <unslothai@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: add Vulkan llama.cpp support
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Address gemini's feedback
* Studio: move the Vulkan VRAM probe into a standalone script
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Improve Vulkan probe error reporting
* Resolve llama-server symlink so Vulkan build is detected
* Drop unreachable Vulkan fallback in GPU free-memory dispatcher
* Skip the Intel GPU probe when NVIDIA or ROCm is present
* Reserve host RAM headroom for Vulkan integrated GPUs
* Add a `UNSLOTH_FORCE_VULKAN` environment variable
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Honor GGML_VK_VISIBLE_DEVICES, reserve discrete Vulkan VRAM headroom, and clear Intel GPU on --cpu-fallback
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Route Intel and forced-Vulkan hosts to the upstream Vulkan prebuilt, add arm64 Vulkan, keep Vulkan out of RAG auto-detect
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Clear the fork release pin when routing a Vulkan host to the upstream repo
* Gate auto-Vulkan routing on no physical NVIDIA so hidden CUDA devices aren't used
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Pin Vulkan launches with --device Vulkan<i> instead of the raw GGML_VK_VISIBLE_DEVICES index space
* Let user --device override the Vulkan pin, and gate direct Vulkan asset picks on no physical NVIDIA
* Update RAG auto-backend test mocks for the _resolve_auto binary and Vulkan probes
* Keep the add_dll_directory handle alive through the Vulkan probe DLL loads
* Revert RAG auto Vulkan guard, guard multi-backend Vulkan detection, and preserve forced Vulkan across updates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use getattr for RTLD_GLOBAL in the Vulkan probe CDLL mode
* Skip CUDA/ROCm APU and datacenter GPU tuning on Vulkan builds
On a Vulkan llama.cpp build gpu_indices are ggml compact ordinals, not
CUDA/ROCm physical ids, so _amd_apu_wants_unified_memory and
_apply_datacenter_env were reading the wrong device. On a mixed AMD APU
plus discrete GPU host that could raise a spurious system-RAM shortfall
and block a valid discrete-GPU load. Gate all three call sites on
not is_vulkan_backend; the Vulkan path already reserves iGPU host
headroom and the backend ignores GGML_CUDA_* anyway.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Tighten Vulkan-guard comment in load_model
* Reduce comments in Vulkan support to be more succinct
* Resolve shell-wrapper llama-server entrypoint to the real lib dir
create_exec_entrypoint falls back to a #!/bin/sh wrapper at the install
root when it cannot symlink into build/bin. _find_llama_server_binary
returns that root entrypoint, but Path.resolve() does not follow a shell
wrapper, so _llama_lib_dir returned the install root and _is_vulkan_backend
missed libggml-vulkan.so -- silently skipping the Vulkan probe and --device
pin on an otherwise valid Vulkan install. Follow the wrapper's exec target
to build/bin. Regression test: test_shell_wrapper_entrypoint_resolves_to_real_lib_dir.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
Co-authored-by: danielhanchen <danielhanchen@gmail.com>
* Studio: source CPU llama.cpp prebuilts from the unslothai fork
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: reject unknown Linux CPU arches and keep ROCm-tooling hosts off the CPU prebuilt
* Studio: extend the resolve-prebuilt ROCm-tooling guard to Windows
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: let ROCm-SDK-only CPU hosts take the fork CPU prebuilt
* Studio: accept windows-arm64 prebuilt kind and refresh stale fork-routing comments
* Studio: correct stale fork-routing comments and --resolve-prebuilt help
* Refresh stale ggml-org routing comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
* Studio: treat data-center Blackwell (sm_100/sm_103) as Blackwell in llama.cpp prebuilt selection
_host_is_blackwell gated on _BLACKWELL_MIN_SM = 120, but data-center Blackwell
parts report a lower compute capability than consumer Blackwell: B100/B200 are
sm_100 and B300/GB300 are sm_103, while RTX 50 is sm_120 and DGX Spark is
sm_121. Because 100 and 103 are both < 120, every data-center Blackwell host was
classified as non-Blackwell, so two GPU-targeting paths never fired for a
B200/B300:
- the Linux blackwell_runtime_override that prefers the highest CUDA-major
runtime line shipping a bundle covering the host SMs (so a cu12x torch could
pin a cuda12 bundle over a native cuda13 one), and
- _drop_blackwell_incapable_windows_cuda, which removes cuda-12.4 builds that
load and validate but run Blackwell on a slow PTX-JIT path.
The result is a B200/B300 being handed a prebuilt that does not natively offload
its SM, i.e. the llama.cpp prebuilt is not really for the GPU. The Blackwell
floor is sm_100, so set _BLACKWELL_MIN_SM = 100. The toolkit floor (12.8) is
unchanged and already correct for sm_100/sm_103.
Surfaced loading unsloth/GLM-5.2-GGUF UD-IQ1_S on 8x B200.
Adds tests covering the sm_100/sm_103 classification, the Linux cuda13
preference for a data-center host, and the Windows cuda-12.4 drop.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Trim comments to be succinct (no behavior change)
* studio: require CUDA 12.9 for sm_103/sm_121 Blackwell prebuilts
sm_103 (B300/GB300) and sm_121 (DGX Spark) have no native compiler
target before CUDA 12.9; the family floor of 12.8 only covers
sm_100/101/120. Make the Windows-CUDA Blackwell filter SM-aware so a
legacy win-cuda-12.8 bundle is dropped on an sm_103/sm_121 host while
sm_100/sm_120 hosts keep the 12.8 floor.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: defer llama.cpp update probes and self-heal MLX on macOS
Two macOS startup problems shared one root area in the FastAPI lifespan:
- The llama.cpp capability + freshness probes ran inline before the server
yielded, so a cold/slow/flaky network on the GitHub freshness check blocked
'Application startup complete' (~34s on CI, longer in the field). Move both
probes to a daemon thread; app.state stays None until ready (status routes
already re-probe at request time). Opt out with UNSLOTH_DISABLE_UPDATE_CHECK=1.
- Train and Export were greyed out because mlx/mlx-lm/mlx-vlm arrive only
transitively and a resolver backtrack silently drops them, so CHAT_ONLY stayed
true. Add utils/mlx_repair.py: when Apple Silicon is detected without MLX,
reinstall mlx/mlx-lm/mlx-vlm by name on a daemon thread and re-run hardware
detection (opt out UNSLOTH_DISABLE_MLX_AUTOREPAIR=1). Surface a chat_only_reason
in /api/health plus a sidebar tooltip so a greyed Train/Export explains itself
instead of failing silently.
* Studio: guard model defaults against a None model name
load_model_defaults(None) called model_name.lower() with no guard, raising
'Error loading model defaults for None' before any model is selected. Return
an empty dict for a falsy/non-str name.
* Studio: drop obsolete upstream macOS + Windows Blackwell prebuilt pins
Both pins worked around gaps in ggml-org upstream prebuilts, but Studio now
routes every GPU host and all of macOS to the unslothai/llama.cpp fork
(published_repo_for_host), which ships the needed bundles, so both pins are
dead code on the default install path:
- macOS b9415: macOS always routes to the fork (its own macOS bundles), and
host_supports_macos_minos() is the backstop. The pin only fired under an
explicit --published-repo ggml-org override.
- Windows Blackwell b9360: Windows-NVIDIA routes to the fork, whose
windows-x64-cuda13 bundle covers Blackwell (manifest max_sm 120, toolkit
13.3), so the pin's self-disable check makes it dormant on every default
install; it could only activate under the same upstream override on a
13.0-13.2 driver.
Remove the pin constants, functions, and call sites. Keep the Blackwell
capability detection (_drop_blackwell_incapable_windows_cuda, _host_is_blackwell,
_windows_cuda_attempt_covers_blackwell) that still drops a non-sm_120 cuda-12.4
build on a Blackwell host. After this, an explicit --published-repo ggml-org
override on a Blackwell 13.0-13.2 host loses its GPU fallback and lands on CPU;
the default fork path is unaffected. Update the install selection-logic and
macOS-compat unit tests for the new no-pin behavior.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: walk back deeper on the macOS upstream prebuilt path
After removing the b9415 macOS pin, the explicit --published-repo ggml-org
upstream path still used the default 2-release fallback, so a pre-macOS-26 host
behind a run of macOS-26-only builds would exhaust two too-new plans (minos is
only checked post-download) and drop to a source build before reaching a
loadable older release. Walk back as deep as the fork macOS path
(DEFAULT_MAX_MACOS_RELEASE_FALLBACKS), turning the removed static pin into
dynamic discovery. Addresses review feedback on the macOS upstream fallback.
* Studio: pin transformers during MLX self-heal so it cannot break Studio
mlx-lm/mlx-vlm declare transformers>=5, but the single-env install pins
transformers==4.57.6. The self-heal used --upgrade with no constraint, so it
could upgrade transformers in the live venv and break the rest of Studio just to
make import mlx.core pass. Pin transformers to the installed version via a
constraint file: the resolver either finds an mlx build compatible with it or
fails (we stay chat-only), never upgrading transformers underneath Studio.
Addresses review feedback on the MLX repair install.
* Studio: harden MLX self-heal against an unsupported mlx-vlm
Pinning transformers alone made uv backtrack mlx-vlm to 0.3.9 (below unsloth-zoo's
mlx-vlm>=0.4.4), which imports but breaks VLM Train/Export -- so the self-heal
could clear chat-only onto a broken stack. Mirror the main installer: set
UV_OVERRIDE=overrides-darwin-arm64.txt so a current mlx-vlm coexists with the
transformers pin, require the same minimum versions unsloth-zoo declares, and
gate/validate on a full mlx_stack_available() check (not a bare import) so an
old or partial stack stays chat-only. Addresses PR review.
* Studio: filter Blackwell-incapable CUDA in resolve_upstream_asset_choice
resolve_upstream_asset_choice returned the first windows-cuda choice unfiltered,
so a Blackwell host could be handed an sm_120-incapable cuda-12.4 build while the
sibling planners drop it. Apply _drop_blackwell_incapable_windows_cuda here too
and fall through to the CPU bundle on a Blackwell host with no capable GPU asset.
Addresses PR review.
* Studio: re-poll health so MLX self-heal reaches an open UI
The sidebar cached the initial /api/health, so a successful background MLX
self-heal (chat_only flips false) did not re-enable Train/Export until a manual
reload. While chat-only for the recoverable mlx_unavailable reason, re-poll
/api/health and stop once Train/Export become available. Addresses PR review.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: make the disabled Train/Export tooltip reachable
The greyed Train/Export items pass a tooltip explaining why (e.g. MLX missing),
but a disabled <button> fires no pointer events and SidebarMenuButton only showed
tooltips while collapsed, so the explanation never appeared. Wrap a disabled
button in a focusable span and show its tooltip while expanded too; enabled items
keep the collapsed-only behavior. Addresses PR review.
* Studio: gate Train/Export on the full MLX stack, not bare mlx.core
detect_hardware enabled MLX training whenever `import mlx.core` worked, but the
MLX self-heal (utils/mlx_repair) treats a stack without mlx-lm/mlx-vlm at the
versions unsloth-zoo requires as inadequate. That asymmetry let the UI enable
Train/Export on exactly the partial/backtracked stack the self-heal is trying to
repair (greyed-in-but-broken VLM export). Gate on the same mlx_stack_available()
criterion so a partial stack stays chat-only (reason mlx_unavailable) and the
background repair restores it. Addresses PR review.
* Fix MLX repair and health auth for PR #6494
* Fix macOS upstream prebuilt fallback for PR #6494
* Fix MLX stack validation for PR #6494
* Fix MLX self-heal validation for PR #6494
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Review fixes: isolate hardware-state test, robust transformers pin
- test_chat_only_reason.py: detect_hardware() assigns module globals directly,
which monkeypatch does not revert; the autouse fixture now saves and restores
DEVICE/CHAT_ONLY/CHAT_ONLY_REASON/IS_ROCM so a chat-only verdict here cannot
leak into other backend tests (e.g. test_utils.py) on a GPU host.
- mlx_repair.py: read the transformers version from importlib.metadata instead of
importing transformers, so the install pin is not silently dropped when
transformers has valid metadata but fails to import.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix CI: model full MLX stack in dispatch tests, keep selection test offline
dispatch (macOS) job:
- detect_hardware now gates MLX on the full stack (mlx_stack_available imports
mlx_lm/mlx_vlm and checks dist versions), so faking only mlx.core makes the
apple_silicon_mlx profile resolve to CPU. The dispatch tests assert the routing
decision when the stack IS usable, so model a complete stack:
test_hardware_dispatch_matrix patches utils.mlx_repair.mlx_stack_available and
test_is_mlx_dispatch_gate patches hardware._has_usable_mlx_stack. The stack
predicate's own internals stay covered by test_mlx_repair.py.
Repo tests (CPU) job:
- test_no_cuda_attempt_on_published_path_for_13_1 fell through to a live
github_release_assets() upstream fetch after the Blackwell filter dropped every
published attempt, which the offline security scanner blocks. Stub that fetch so
the walk-back deterministically finds no usable CUDA build and raises
PrebuiltFallback without network.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Harden MLX self-heal: prepare transformers constraint inside the try
attempt_mlx_repair runs on a daemon thread, but _transformers_constraint_args was
called before the try. A failure there (e.g. tempfile.mkstemp on a full disk or a
bad TMPDIR) would propagate unhandled and silently kill the self-heal thread.
Move the call inside the try and initialize constraint_path so any such failure
is caught and leaves Studio chat-only instead of crashing the thread.
---------
Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
* Studio: offer the in-app llama.cpp update for source-build (markerless) installs
Source-build installs have no UNSLOTH_PREBUILT_INFO.json marker, so freshness
reported supported=False and the Update button never showed (notably on macOS,
where the fork shipped no prebuilt before b9585 and setup fell back to a source
build). When an install has no marker but an official prebuilt now exists for
the host, surface the update and let one click swap it in place.
- install_llama_prebuilt.py: published_repo_for_host() (the setup.sh host->repo
rule in Python) and a --resolve-prebuilt mode that reports whether a prebuilt
exists for this host without downloading.
- llama_cpp_update.py: markerless branch in get_update_status/start_update,
version-suppressed so source builds already newer than latest are not nagged;
fail-open throughout.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: run llama update detection off the event loop, expose source_build
The markerless source-build check probes the host and reads GitHub, so run
get_update_status and start_update in a worker thread to keep the API
responsive. Expose source_build in the status response so the banner can label
the source-build switch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Keep the llama-route auth stub out of sys.modules for the rest of the suite
test_llama_route.py replaced sys.modules['auth.authentication'] with a
bare stub at collection time and never restored it, so every later test
importing create_access_token got the stub: 17 failures across
test_desktop_auth, test_middleware, test_openai_tool_passthrough and
test_rag_preview on all four Backend CI Python versions. Import the
real module when its deps are available and only stub in minimal envs,
popping the stubs after the standalone route load either way.
* Studio: address review on the source-build update path
- published_repo_for_host: route CPU-only Windows to ggml-org too (mirrors
setup.ps1; the fork ships no win-cpu bundle), macOS always the fork.
- markerless detection compares/display the upstream llama_tag, not a possible
fork wrapper release_tag, so a source build is not wrongly judged newer.
- do not offer when there is no resolvable install root (a pinned
LLAMA_SERVER_PATH outside a managed dir): an apply would not take effect.
* Ignore version probes in the update tests' subprocess capture
The status polls in these tests trigger the new source-build detection,
which shells out to llama-server --version through the same patched
subprocess.run. On slow runners that probe lands after the installer
call and clobbers the single captured argv, failing the flag
assertions (seen on the 3.10/3.11 Backend CI jobs). Skip probe calls
in all three fakes so only the installer invocation is captured.
* Skip markerless re-detection while the update job is swapping the tree
On a source-build install the frontend polls update-status every 3s
during an apply, and each poll ran _source_build_status, which execs
the very llama-server binary the job is concurrently replacing. On
Windows that exec can hold the exe long enough to fail the installer's
os.replace; everywhere it is a per-poll subprocess spawn for a status
the poller does not read (it only consumes job progress). Gate the
markerless branch on the job not running; the marked path is probe-free
and still returns the live job state.
* Studio: tighten source-build update root, repo routing, and downgrade guard
Only manage a markerless install when the active binary lives under a
resolvable llama.cpp root (marker dir, UNSLOTH_LLAMA_CPP_PATH it sits in,
or a llama.cpp ancestor); a pinned LLAMA_SERVER_PATH or a PATH/system
binary is left alone so an apply cannot install where it would not take
effect. Gate start_update on the same suppression as detection so a
direct POST cannot downgrade a source build newer than the latest
prebuilt. Route Linux hosts with AMD tooling (rocminfo/amd-smi/hipconfig/
hipinfo) to the fork in --resolve-prebuilt, matching setup.sh, so a HIP
source build is not offered an upstream CPU prebuilt.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: cover inactive env root and pinned llama.cpp checkout in update root tests
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Naitik Pal