is_cuda_server() treats a co-located libggml-cuda.so* as proof the server is
CUDA-ready. That's normally true (llama.cpp dlopens the backend from beside the
binary), but an *interrupted* build (thermal/power shutdown -- common on the
NVIDIA-ARM laptops this path targets) can leave a half-linked libggml-cuda.so
next to the server: present, so is_cuda_server() matches, yet the backend fails
to load at runtime. The post-build path already wipes+rebuilds such a partial
.so, but the step-0 early-skip trusted it and never rebuilt -- so Studio could
report GGUF CUDA inference ready while running a broken/non-CUDA backend.
Gate the early-skip with cuda_server_probe(): 'llama-server --list-devices'
enumerates backends and exits (cheap, no server spin-up). Only a definitive
'flag supported, ran, but no CUDA device' triggers a clean rebuild; a timeout or
an old pin without --list-devices stays inconclusive and keeps trusting the .so,
so we never force a needless, thermally-expensive rebuild. Probe logic verified
against healthy/broken/unsupported/timeout stubs (0/1/2/2).
Addresses Codex review P2 (provision_llama_cuda.sh).
- find_nvcc now prefers the highest /usr/local/cuda-<ver> toolkit so a stale
unversioned `cuda` symlink or an older nvcc earlier on PATH can't win and
rebuild with CUDA 12.x (re-hitting the glibc>=2.41 / Blackwell clash this
script avoids); falls back to a PATH nvcc only when no versioned toolkit.
- Validate the GPU compute_cap is purely numeric before using it as
CMAKE_CUDA_ARCHITECTURES: some WSL GPU-PV / driver combos report "N/A",
which CMake rejects (aborting an otherwise-usable build) instead of letting
"native" autodetect.
- Gate the native-Linux aarch64 provisioner on _SKIP_GGUF_BUILD: when a non-root
user declines the sudo prompt (or lacks sudo) for GGUF deps, don't then run a
provisioner that does its own sudo apt-get installs.
- install.sh: gate the new aarch64 bitsandbytes block on SKIP_TORCH=false --
with --no-torch/UNSLOTH_NO_TORCH (GGUF-only install) it would have pulled
torch back into the venv through bitsandbytes' dependencies.
- studio worker: in the new Spark OOM-guard section, decide
PYTORCH_CUDA_ALLOC_CONF (expandable_segments) BEFORE the guard's first CUDA
touch -- get_device_properties initializes the CUDA allocator, after which
the env var is ignored, and the later `import unsloth`
(patch_dgx_spark_memory_config) is too late for the worker process. Uses
the same CUDA-free nvidia-smi name sniff, append-don't-override, and
UNSLOTH_NO_EXPANDABLE_SEGMENTS opt-out as the library patch. Live-verified
on the N1X: env set while torch.cuda.is_initialized() is still False.
- uninstall.ps1: only run `fuser -k 8888/tcp` in a probed WSL distro when an
Unsloth install actually exists there (checked BEFORE the rm deletes the
marker) -- an unrelated listener on 8888 (e.g. Jupyter) in a clean distro
must survive a Windows-side uninstall. The Unsloth-specific pkills stay
unconditional.
- install.ps1 + uninstall.ps1: persist the chosen WSL distro to
%LOCALAPPDATA%\Unsloth\wsl-distro.txt at install; uninstall reads it
(before removing the directory) and prepends it to the cleanup candidates,
so a custom UNSLOTH_WSL_DISTRO install is cleaned without the env var
being set again at uninstall time.
- provision_llama_cuda.sh: honor UNSLOTH_LLAMA_PR (numeric-validated,
best-effort fetch of pull/N/head after clone) so a provisioned tree
matches a PR pin the way setup.sh does; and require only llama-server in
the main cmake build (mirroring setup.sh), building the helper targets
(llama-cli/quantize/mtmd-cli/gguf-split) best-effort afterwards -- an
older UNSLOTH_LLAMA_TAG pin lacking a newer helper target no longer fails
the whole provision.
Not changed: the "--tauri rejection doesn't restore the venv rollback"
comment is incorrect -- the rejection returns through Exit-InstallFailure,
which itself calls Restore-StudioVenvRollback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- install.ps1: detect a PRE-EXISTING WSL1 distro up-front (kernel string +
libcuda probe inside the distro; encoding-proof vs UTF-16 `wsl -l -v`) and
convert it with `wsl --set-version 2`, failing early with instructions if
conversion does not take -- instead of completing a full install that only
fails at the final torch.cuda check (no GPU passthrough under WSL1).
- install.ps1: quote the distro name in the generated unsloth.cmd shim and in
the copy-pasteable hint commands so UNSLOTH_WSL_DISTRO values with spaces
("Ubuntu Preview") keep working.
- install.ps1: forward UNSLOTH_NO_LLAMA_CUDA=1 into the WSL install env; the
inner setup.sh otherwise defers its llama.cpp build to a background builder
this script then never dispatches (the same opt-out skips it), leaving no
llama-server and a misleading "building in background" footer. Also add
libcurl4-openssl-dev to the WSL bootstrap apt line.
- provision_llama_cuda.sh: install libcurl4-openssl-dev with the base tools --
_cmake_configure forces -DLLAMA_CURL=ON and on the deferred WSL path this
script is the only build path (setup.sh's GGUF dep install was skipped), so
configure failed on fresh hosts without the headers.
- provision_llama_cuda.sh: keep the pre-existing llama.cpp backup until the
fresh build is CONFIRMED (was: dropped right after a successful clone), and
restore it on configure/build failure or when no server binary was produced
-- a failed CUDA build no longer destroys a previously working (CPU) server.
- setup.sh: when provisioning fails and NO llama-server is present, set
_LLAMA_CPP_DEGRADED=true so the arm64 CPU-prebuilt last resort and the
installer failure exit fire instead of reporting a working install.
Round-2 comments verified already fixed in ad77ae6 (anchored to its parent
d161ff5): the torch probe already passes --reinstall; the WSL uninstall is
already scoped to /root only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
install.ps1 (WoA WSL fallback):
- report failure (non-zero) + restore the rolled-aside venv when the WSL GPU
install fails (torch.cuda absent) or when WSL needs enabling+reboot, instead of
returning success — so -File/Tauri callers don't see a broken install as complete
- on WSL success, Complete-StudioVenvRollback so the previous-venv backup isn't orphaned
- refuse under --tauri with a clear "use the CLI installer" message (the desktop
launcher resolves a Windows-venv backend, which a WSL-only install can't provide)
- reset $LASTEXITCODE before each wsl.exe / python probe (a stale 0 could mark WSL
ready / torch OK if the native command fails to launch)
- torch-availability probe: --reinstall so an already-installed CPU torch in a
migrated venv isn't accepted as "satisfied" (would wrongly skip the WSL path)
- treat a null HKCU PATH as empty (fresh profile) so shim PATH update can't throw
- keep apt stderr visible inside WSL (only stdout -> /dev/null) for diagnosability
scripts/uninstall.ps1:
- scope WSL cleanup to /root (the fallback's install location); stop deleting
/home/*/.unsloth, which could erase an unrelated WSL user's own Unsloth/cache
studio/setup.sh:
- direct (non-install.ps1) WSL installs now provision CUDA llama.cpp themselves
instead of being left with no GGUF server: install.ps1 exports
UNSLOTH_WSL_LLAMA_DEFERRED=1, and the aarch64+NVIDIA provision block runs under
WSL only when that marker is absent
- mark a provisioner-built llama.cpp as Studio-owned in custom-STUDIO_HOME mode so
the next setup's _assert_studio_owned_or_absent doesn't abort
- glibc>=2.41 check: also match a future major>2 (e.g. 3.0)
studio/scripts/provision_llama_cuda.sh:
- install base tools (cmake/git/curl) in their own apt transaction before the
best-effort gcc-14/g++-14 (unavailable on Ubuntu 22.04 / Debian 12, where bundling
them aborted the whole transaction and left no build tools)
- back up an existing (e.g. CPU-only) llama.cpp before the destructive clone and
restore it on clone failure, so a failed clone doesn't leave the user with no server
- honor a pinned llama.cpp ref via UNSLOTH_LLAMA_TAG instead of always tracking main
unsloth/models/_utils.py:
- set PYTORCH_CUDA_ALLOC_CONF (expandable_segments) via a CUDA-free Spark detector
(nvidia-smi, not torch.cuda.get_device_name) so it takes effect before CUDA/the
caching allocator initialize — previously it was a silent no-op on auto-detected Spark
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A non-numeric or 0 override silently fell through to `cmake -j0`, which
builds with ALL cores -- the opposite of the thermal-headroom default and a
shutdown risk on NVIDIA-ARM laptops. Validate it's a positive integer; ignore
anything else and auto-compute.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An interrupted CUDA build (e.g. a thermal/power shutdown mid-compile -- which
this machine class hits) can leave a partially-linked libggml-cuda.so. On the
next run cmake does not relink it, so linking llama-server fails with undefined
ggml_cuda_op_* references and the script gives up with no server. Mirror the
existing configure retry-clean: if `cmake --build` fails, wipe build/,
reconfigure, and rebuild clean once before giving up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A full -j(nproc) CUDA build trips power/thermal shutdowns on thermally
constrained NVIDIA-ARM laptops (observed on the N1X "RTX Spark": a full-core
build, especially alongside other load, shuts the machine down). nice lowers
CPU *scheduling* priority but not heat -- power/heat scale with the number of
active compile jobs -- so default to ~half the cores instead: still ~2.5x
faster than a tiny -j4, but leaves real headroom. Still mem-capped (~1.5 GB
per nvcc job) and overridable via UNSLOTH_LLAMA_BUILD_JOBS (raise on a
well-cooled box, lower if it still trips). Tiny boxes (<=4 cores) use all.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Building at -j(nproc) saturates every core (load ~25 on a 20-core box),
which starved a concurrently launched `unsloth studio` / training session
during the build's few-minute window. Wrap the cmake build in
`nice -n 19` (+ `ionice -c 3` when available): full speed when the box is
idle, but instant yield to foreground work. Also trims this file's comments.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The CUDA llama.cpp compile is the slow step of the WSL GPU setup. The job
count now defaults to the full core count (nproc) instead of being capped,
which is ~5x faster on a 20-core box (-j4 -> -j20). To stay safe on
unified-memory machines, where nvcc jobs (~1.5 GB each) could OOM-kill a
full-parallel build, jobs are capped at mem/1.5GB when that is lower than
nproc. UNSLOTH_LLAMA_BUILD_JOBS=N still overrides for thermal throttling.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three coupled fixes so the Windows-on-ARM + NVIDIA (DGX Spark / N1X) WSL path
builds the GPU llama-server reliably and never wastes time on a CPU build:
1. install.ps1 -- the post-install CUDA provision was launched as a WSL-side
`nohup setsid ... &`. That does NOT survive: WSL shuts the distro's VM down
once the launching wsl.exe session exits, killing the detached build (observed
on a fresh distro: no build log, only a CPU server left behind). Fetch the
provision script in a quick session, then run the build anchored to a DETACHED
Windows-side process (Start-Process wsl.exe, no -Wait) that holds the VM up for
the whole build while install.ps1 returns immediately.
2. provision_llama_cuda.sh -- a pre-existing build/ can carry an incompatible
CMake cache (the Studio installer stages its build in llama.cpp.build.NNNN then
relocates it, leaving a cache with stale absolute source/build paths and
GGML_CUDA=OFF), so reconfiguring for CUDA fails ("CMakeCache directory is
different" / "source does not match"). Try to reuse build/ first (incremental
resume), and if configure fails, wipe build/ and configure clean once. Verified
live on the failing scenario: stale cache detected, wiped, clean CUDA configure.
(setup.sh's skip of the CPU source build on this path is the companion commit;
together the fresh-install path builds only the CUDA server, in the background.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Current llama.cpp ships the CUDA backend as a dynamically-loaded plugin
(libggml-cuda.so* next to the binary), NOT a load-time dependency, so
ldd llama-server | grep libggml-cuda is a false negative: it reports no
CUDA on a perfectly good CUDA build. That made both is_cuda_server()
(provision_llama_cuda.sh) and _have_cuda_llama_server() (setup.sh) force a
needless full rebuild every run.
Fix both: keep the ldd check (old monolithic builds) and additionally treat
the presence of libggml-cuda.so* beside the binary as the CUDA signal. A
CPU-only build has no such backend, so this stays correct for the CPU case.
Verified on an N1X/sm_121 WSL build: llama-server --list-devices shows
CUDA0 JMJWOA-Generic-GPU and serves on the GPU, while ldd lists no
libggml-cuda; the new check correctly returns CUDA-present.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When the installer is launched from a Windows shell, WSL interop leaks the Windows
PATH (/mnt/c/... entries, with spaces) into the build environment, which can make
cmake/gcc/git resolve to Windows tools or otherwise confuse the CUDA build. Prepend
the CUDA toolkit + standard Linux dirs so the Linux toolchain always wins; keep the
original PATH after so nvidia-smi etc. still resolve.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A full -j(nproc) CUDA build is power/thermal-heavy on laptops (e.g. N1X) and can
trip a thermal/power shutdown mid-build. Allow lowering the job count; cmake
--build is incremental so re-running resumes from where it stopped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A clean-slate reinstall on an ARM64+NVIDIA box surfaced three follow-on gaps in
the WSL path. All fixes are additive, best-effort, and confined to the $torchOk
success branch of the WSL fallback, so they only run on the ARM64+NVIDIA machines
that reach it -- no other platform is affected.
install.ps1:
- Self-heal Studio's web-server deps. install_python_stack.py installs the Studio
UI deps (fastapi/uvicorn/structlog/starlette) in a late step; if that run is cut
short, torch+unsloth land but the server stack is missing and `unsloth studio`
dies at launch on ModuleNotFoundError. Import-check the stack after the torch.cuda
probe and, if absent, install it WITHOUT re-pinning huggingface-hub/transformers/
datasets, so the verified GPU torch path is never disturbed.
- Seed pip into the (uv-managed, pip-less) venv via ensurepip so save_pretrained_gguf
-> check_pip() works regardless of how Studio is launched.
- Auto-build a CUDA llama-server for GGUF inference in the background via the new
provision script (below), so GGUF chat/tool-calling lights up a few minutes after
install with zero manual steps. Opt out with UNSLOTH_NO_LLAMA_CUDA=1.
studio/scripts/provision_llama_cuda.sh (new):
- Idempotent, best-effort (always exits 0). Builds a CUDA llama.cpp into
~/.unsloth/llama.cpp (Studio's resolver path). Generic across NVIDIA Linux/WSL
incl. aarch64 (DGX Spark, N1X): derives the arch from the GPU's compute_cap,
installs gcc-14 + CUDA 13.3 only when nvcc is missing (gcc-15 is rejected by nvcc;
CUDA <13.3 hits the glibc>=2.41 rsqrt header clash), and builds the full target set
(llama-server llama-cli llama-quantize llama-mtmd-cli llama-gguf-split) so it
satisfies both Studio inference and save_pretrained_gguf without a later rebuild.
Validated on an NVIDIA N1X (sm_121): training, GPU inference, GGUF q4_k_m export,
`unsloth studio` via both Desktop + Start Menu shortcuts (HTTP 200), GGUF chat at
121 tok/s (BLACKWELL_NATIVE_FP4=1) and OpenAI-style tool-calling.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
