Sixth review round; every item reproduced against the live scripts first.
The WSL fallback tolerated a nonzero inner exit (the optional llama.cpp
step legitimately fails), so a run whose installer died mid-way could pass
the torch and CLI probes on a stale venv from a previous install and be
reported as success. setup.sh now stamps /root/.unsloth/.install-ok after
the core venv and Studio deps complete, just before its tolerated
llama-only nonzero exit; install.ps1 clears the stamp before the run and
requires it to exist afterwards (existence only, no mtime compare, so
WSL/Windows clock skew cannot bite). uninstall.sh removes the stamp and
the downloaded installer file so the trailing rmdir can still prune.
Root login shells reset PATH via /etc/profile and drop /usr/lib/wsl/lib,
the only location of nvidia-smi under WSL2 GPU-PV, so every bare
nvidia-smi probe in setup.sh and the provisioner could silently misreport
"no GPU". Both now resolve nvidia-smi explicitly (PATH, then
/usr/lib/wsl/lib, then /usr/bin) via a shared-resolver pattern, and the
provisioner's driver-major and compute_cap reads use the resolved path.
My round-five uninstall fix inserted the CUDA-build kill block after the
llama.cpp tree was already removed, so a live cmake/nvcc kept running
against deleted paths; the block now runs before the removal.
uninstall.ps1 gated its legacy marker-less WSL cleanup on the process
PROCESSOR_ARCHITECTURE, which reports AMD64 under an x64-emulated
PowerShell on ARM64, skipping exactly the machines the fallback installs
on. It now uses the same triple detection as install.ps1 (OSArchitecture,
Win32_Processor.Architecture 12, machine-level registry arch), factored
into one helper used at both gate sites.
The nvidia-smi capture helper retried twice with a 60s timeout
everywhere, so off WSL a hung nvidia-smi stalled three successive
detect_host probes for about two minutes each; the generous retry now
applies only under WSL (where GPU-PV load slowness is real) and bare
metal keeps a single short attempt.
The generated WSL Desktop launcher hardcoded port 8888 for launch, health
poll, and browser open, so with Jupyter or a second Studio on 8888 the
poll waited on the wrong server forever; it now scans 8888..8908 with a
TcpListener, mirroring the native launcher's free-port window, and passes
the winner via -p everywhere.
Verified: bash -n on all shell scripts, Python AST parse, PowerShell AST
parse on install.ps1, uninstall.ps1, and the generated launcher; the
launcher port scan exercised free, busy, and exhausted cases; the capture
helper unit-tested for WSL and bare-metal attempt/timeout splits; sh test
battery matches the branch baseline.
Fifth review round; each item traced through the live scripts before fixing.
A provisioner fresh clone that failed to produce a server was left behind as
a markerless git tree; under a custom UNSLOTH_STUDIO_HOME the next run's
ownership assert refuses the unmarked dir and aborts the whole install until
the user deletes it by hand. _restore_prev now removes a clone this script
created when no server came out of it (backed-up dirs restore as before).
The CUDA provision gate ignored --with-llama-cpp-dir linked mode, so a
linked user tree with a CPU-only server could be checked out to a pinned
ref, rebuilt in place, or moved aside entirely and replaced by a fresh
clone. The gate now skips linked local dirs.
uninstall.sh removed the CUDA build artifacts without stopping a running
detached build; _pkill_studio only matches Studio roots, so live cmake/nvcc
kept burning thermals, recreated build files, and defeated the trailing
rmdir. The runner, provisioner, and llama.cpp-path processes now get
TERM-then-KILL with the same escape helper and grace the Studio kill uses.
The worker's memory-fraction guard classified Spark purely from device
props, so UNSLOTH_FORCE_DGX_SPARK=1 on an unlisted name got no fraction
guard (and the fraction env was dead), while FORCE=0 could not disable it;
the guard now honors the same force semantics as the detectors.
UNSLOTH_LLAMA_TAG / UNSLOTH_LLAMA_PR were interpolated into the runner
script's single-quoted exports unvalidated while every sibling forward has
an allow-list; they now get the INSTALL_REF ref allow-list and a digits-only
check respectively (own-machine robustness, not a trust boundary).
On WSL-fallback success with a custom UNSLOTH_STUDIO_HOME, the installer
deleted the rolled-aside custom-root venv right after telling the user that
root is not used by the WSL install; a custom root now restores the previous
venv instead (the WSL shim does not depend on the Windows venv), while the
default root keeps dropping the vestigial backup.
Verified: bash -n on all three shell scripts, AST parse on worker.py,
PowerShell AST parse on install.ps1, icon suites pass, sh battery matches
the branch baseline. Two resurfaced anchors (build/bin backup, --package
forwarding) confirmed already fixed at head.
Fourth review round; each item verified against the live scripts and the CUDA
compatibility documentation before fixing.
The provisioner installed (and, since the stale-toolkit change, preferred)
CUDA 13.3 without ever consulting the driver, but cu13 binaries need a 580+
driver and minor-version compatibility never crosses majors, so a GH200-class
host on a 5xx driver got an unloadable llama-server that the structural
acceptance check then stamped as ready. The driver's supported CUDA major is
now read from nvidia-smi and enforced three ways: the stale-toolkit upgrade
only fires when the driver can run cu13, a fresh install on a sub-13 driver
bails to the existing no-toolkit message instead of installing 13.3, and a
final guard swaps a too-new selected toolkit for the newest one the driver
supports (or refuses to build). Spark-class hosts (580+ drivers) behave
exactly as before; unparseable output keeps the previous behavior.
The WSL install pipeline ended in curl | sh, so a failed download fed sh an
empty stdin and exited 0; on a rerun the stale venv then passed the torch
probe and the installer reported success without ever running. install.sh is
now downloaded to a file with exit 86 as the never-ran sentinel, checked
before any probe (rollback + non-zero). The --package splice moved onto the
file invocation.
When the Studio web-server dep repair failed its re-verify, the installer
still created shims and reported success; the missing set includes typer, so
even the plain unsloth CLI dies. A failed repair now routes to the existing
failure path (rollback + non-zero), mirroring the CLI-missing case.
If all three provision-script resolutions fail (unpackaged wheel + GitHub
unreachable), the provision block silently skipped and, with the CPU build
now deferred on native Spark hosts, the install could report success with no
GGUF server; that case is now marked degraded so the CPU-prebuilt last
resort and failure exit fire.
flex_attention.py's Spark sniff gets the same /usr/lib/wsl/lib/nvidia-smi
fallback as the other two detectors (grep confirms these are the only three),
and uninstall.sh removes the remaining WSL-side build artifacts
(run_llama_build.sh, llama_cuda_build.log, the shortcut-skip marker) so the
.unsloth directory can actually be removed.
Verified: bash -n on all three shell scripts, AST parse on flex_attention.py,
PowerShell AST parse on install.ps1, the toolkit-picker awk exercised against
a fake /usr/local tree (driver 12 picks cuda-12.8 over 13.0, driver 11 picks
none), icon suites pass, sh test battery matches the branch baseline.
Third review round; each item re-verified against the live scripts.
A stale CUDA < 13 toolkit was kept forever: the 13.3 install was gated on
nvcc being absent, so a host with CUDA 12.x failed the sm_121 configure (or
the glibc >= 2.41 rsqrt clash) on every rerun and always exited with the CPU
server. When apt can provide 13.3 the provisioner now installs it alongside a
stale toolkit; find_nvcc's sort -V prefers the new install, and a failed
install leaves the old toolkit as the last resort, so non-Spark hosts that
build fine on cu12x are unaffected.
llama.cpp pins only applied to fresh clones; an existing checkout rebuilt
whatever commit it had while the log claimed a release pin. Existing
checkouts now fetch and check out the pinned (or resolved-latest) ref, best
effort with the current commit as fallback, and the UNSLOTH_LLAMA_PR handling
moved out of the fresh-clone branch so it applies to both paths.
The WSL fallback silently dropped a non-default --package and reported
success with stock unsloth; it is now spliced into the curl | sh invocation
(the name is regex-validated at parse time).
setup.sh's CUDA provision gate used raw nvidia-smi and ignored the
_setup_nvidia_usable computation that honors CUDA_VISIBLE_DEVICES=""/-1, so a
mixed-GPU host that hid its NVIDIA card still got a system CUDA install; the
gate now requires the flag. On native Linux Spark hosts without nvcc, setup.sh
also no longer does the multi-minute CPU source build that the CUDA provision
in the same run immediately replaces (mirroring the existing WSL deferral
arm); provision failure still cascades to the CPU-prebuilt last resort.
uninstall.ps1's distro extraction truncated quoted names at the first space
(-d "Ubuntu Preview" matched as "Ubuntu"), wrongly keeping or removing
shortcuts; the regex now matches a full quoted token first. And the profile
icon (%USERPROFILE%\.unsloth\unsloth.ico) was removed unconditionally while
the sweep above deliberately keeps launchers for non-evidenced WSL installs,
blanking their icons; removal is now gated on no surviving Unsloth shortcut,
mirroring uninstall.sh's _drop_shared_icon_if_unused guard.
Verified: bash -n on both shell scripts, PowerShell AST parse on both ps1
files, the new distro regex proven on spaced and unspaced names, icon suites
pass, sh test battery matches the branch baseline.
Second review round on the Windows-on-ARM + NVIDIA path; each item verified
against the live code (and torch where relevant) before fixing.
The native-CUDA probe ran uv --dry-run against the venv interpreter without
checking its architecture. uv resolves for the interpreter's platform tags, so
an x64-emulated python resolved existing win_amd64 CUDA wheels and "proved" a
native wheel WoA cannot use, skipping the WSL fallback entirely. The probe now
requires platform.machine() ARM64 from the venv python first; anything else
keeps the WSL routing.
The Studio worker appended PYTORCH_CUDA_ALLOC_CONF next to its memory-fraction
logic, 550 lines after detect_hardware() had already initialized CUDA, where
the allocator config is latched (verified on torch 2.9.1: expandable_segments
set after get_device_properties is a no-op in memory snapshots). The CUDA-free
Spark sniff now runs immediately before detect_hardware(), and it honors the
documented UNSLOTH_FORCE_DGX_SPARK=1/0 override the library detectors support,
closing the older force-flag item on the same block.
setup.sh's _have_cuda_llama_server accepted any co-located libggml-cuda.so,
re-opening the interrupted-relink hole the provisioner's completion stamp was
added to close: in exactly that state setup.sh skipped provisioning and
reported CUDA ready over the old CPU binary. The split-.so branch now also
requires the stamp; monolithic ldd-linked builds are unaffected.
The provisioner builds llama-quantize but never created the repo-root shim
that unsloth_zoo's check_llama_cpp needs (it only searches the root, which is
why setup.sh symlinks it in all three of its own paths). The success branch
now mirrors that symlink.
CMAKE_CUDA_ARCHITECTURES=native needs CMake >= 3.24, but this script installs
distro cmake (Ubuntu 22.04 apt ships 3.22), so the N/A-compute_cap fallback
aborted configure, wiped build/, and aborted again. The fallback now omits the
flag and lets ggml's version-guarded CMake defaults pick the arches.
Fresh clones tracked ggml-org master, bypassing setup.sh's newest-release pin
policy (its own header warns master bypasses the pin). An unset or "latest"
ref now resolves to the newest release tag via the GitHub API, keeping the
default-branch clone as the best-effort fallback when the API is unreachable.
install.sh writes the WSL shortcut icon to the Windows profile
(%USERPROFILE%\.unsloth\unsloth.ico) because the WoA icon broker cannot read
AppData\Local, but both uninstall.sh cleanup sites only cleaned the
AppData\Local icon. Both now clean the profile icon and drop the directory
when empty.
Verified: bash -n on all four shell scripts, AST parse on worker.py,
PowerShell AST parse on both ps1 files, the icon suites pass, and the sh test
battery matches the branch baseline (test_install_host_defaults.sh fails
identically on the clean tree).
Review round on the Windows-on-ARM + NVIDIA WSL2 path; each item reproduced
against the live scripts before fixing.
provision_llama_cuda.sh now serializes with install_llama_prebuilt.py on the
same <parent>/.<name>.install.lock file (its filelock backend is flock(2), so
shell flock interoperates; append-mode open so the Python O_EXCL fallback's
PID file is never truncated). The detached background builder could otherwise
race an installer rerun or `unsloth studio update`, both of which mv/rm -rf
inside the llama.cpp dir. Losing the 2h wait exits 0: another provisioner is
already doing the job.
The step-0 early-skip trusted a co-located libggml-cuda.so alone, which
wrongly skips one case: an in-place rebuild interrupted after the .so links
but before llama-server relinks leaves new .so + old CPU server. A completion
stamp (build/bin/.unsloth-cuda-ok) written only after the script's own final
CUDA check closes that window; skip now requires ldd evidence or the stamp.
The rejected functional --list-devices probe stays rejected: the stamp does
not gamble thermals on an env-fragile probe.
The WSL shortcut skip (install.ps1 owns the canonical WoA .lnk) was only a
transient env var, so the first `unsloth studio update`, whose wsl.exe shim
carries no env into install.sh --shortcuts-only, recreated the duplicate
blank-icon shortcut. The skip is now also persisted as
/root/.unsloth/.skip-wsl-windows-shortcut, checked by install.sh and removed
with the install by both uninstallers.
--with-llama-cpp-dir (and UNSLOTH_LOCAL_LLAMA_CPP_DIR) were parsed but
silently ignored on the WSL fallback path, which builds its own llama.cpp
inside the distro. Reject with guidance (UNSLOTH_LLAMA_TAG / UNSLOTH_LLAMA_PR
pin the WSL-side build), mirroring the --local reject.
uninstall.sh's Windows shortcut sweep only removed wsl.exe-target .lnks, so
the WoA fallback shortcuts (powershell.exe + launch-studio-wsl.ps1) survived
while their launcher dir was deleted, leaving dangling shortcuts. The
owner-matched cleanup now removes them first.
uninstall.ps1 swept every "Unsloth Studio (*.lnk" as legacy, but install.sh
creates exactly that per-distro name for current WSL installs, and the WSL
cleanup below only removes evidenced distros. The sweep now keeps a live
wsl.exe launcher whose distro is not in the same evidence set, so a surviving
WSL install keeps its shortcut; everything else is still swept.
Verified: bash -n on all three shell scripts, PowerShell AST parse on both
ps1 files, flock mutual-exclusion and stamp skip/rebuild decisions exercised
standalone, and the uninstall icon suites (sh + ps1) pass. The
test_install_host_defaults.sh failure pre-exists on the branch merge base.
Three valid findings from the 06-22 Codex review:
1. provision_llama_cuda.sh: when $LLAMA_DIR holds a .git checkout (a prior CPU
source build), the whole-dir backup was skipped, so a failed CUDA rebuild's
'rm -rf build' destroyed the working CPU server with nothing to restore --
leaving NO llama-server despite the 'keeps the existing server' promise (a
thermal shutdown mid-build is a real failure mode on this hardware). Back up
build/bin before the rebuild and restore it on total failure; idempotent and
self-cleaning (never overwrites a freshly built server). Verified both paths.
2. uninstall.ps1: 'fuser -k 8888/tcp' killed ANY listener on 8888 (Jupyter et al.
default to it), not just Studio. Now only kills a PID whose /proc/cmdline is
under /root/.unsloth -- matching the adjacent pkill scoping.
3. setup.sh: the 'defer to background CUDA build' branch fired even on a direct
in-WSL 'unsloth studio update', where install.ps1 never launched a background
builder -- so the footer claimed a build was running while nothing built. Gate
it on UNSLOTH_WSL_LLAMA_DEFERRED=1 (set only by install.ps1, and already read
elsewhere in setup.sh); a direct run now falls through to a real CPU build.
bash -n + PS parse clean; the common install.ps1 WoA path (prebuilt success,
deferred flag set) is unaffected.
The cuda_server_probe() added in 27bc44c gated the step-0 rebuild-skip on a
runtime 'llama-server --list-devices' check. In a real cold install on the N1X
this BACKFIRED: the background provision runs step-0 while the install is still
under heavy load (torch download, frontend build), and under WSL2 GPU-PV the
CUDA backend's init transiently fails under load (the same flakiness cycle-21
worked around for nvidia-smi). --list-devices then enumerated devices but no
CUDA, so the probe declared the freshly-validated PREBUILT 'broken', wiped it
(rm -rf build), and kicked off a CUDA-13.3 toolkit install + source build -- the
exact thermal-risk + wasted-prebuilt outcome cycle-21 eliminated. (Confirmed the
prebuilt is fine: --list-devices shows CUDA0 in a normal shell, even with
LD_LIBRARY_PATH stripped -- the probe failure was purely load-induced.)
Restore the load-insensitive structural check: a co-located libggml-cuda.so* is
trusted, because the prebuilt resolver validates what it installs and an
interrupted SOURCE build is already caught by the build-failure wipe+rebuild in
section 6. The Codex P2's half-linked-.so concern is real but narrow, and a
runtime probe that can gamble the machine's thermals on an env/load-fragile GPU
call is the wrong trade on this hardware.
is_cuda_server() treats a co-located libggml-cuda.so* as proof the server is
CUDA-ready. That's normally true (llama.cpp dlopens the backend from beside the
binary), but an *interrupted* build (thermal/power shutdown -- common on the
NVIDIA-ARM laptops this path targets) can leave a half-linked libggml-cuda.so
next to the server: present, so is_cuda_server() matches, yet the backend fails
to load at runtime. The post-build path already wipes+rebuilds such a partial
.so, but the step-0 early-skip trusted it and never rebuilt -- so Studio could
report GGUF CUDA inference ready while running a broken/non-CUDA backend.
Gate the early-skip with cuda_server_probe(): 'llama-server --list-devices'
enumerates backends and exits (cheap, no server spin-up). Only a definitive
'flag supported, ran, but no CUDA device' triggers a clean rebuild; a timeout or
an old pin without --list-devices stays inconclusive and keeps trusting the .so,
so we never force a needless, thermally-expensive rebuild. Probe logic verified
against healthy/broken/unsupported/timeout stubs (0/1/2/2).
Addresses Codex review P2 (provision_llama_cuda.sh).
- find_nvcc now prefers the highest /usr/local/cuda-<ver> toolkit so a stale
unversioned `cuda` symlink or an older nvcc earlier on PATH can't win and
rebuild with CUDA 12.x (re-hitting the glibc>=2.41 / Blackwell clash this
script avoids); falls back to a PATH nvcc only when no versioned toolkit.
- Validate the GPU compute_cap is purely numeric before using it as
CMAKE_CUDA_ARCHITECTURES: some WSL GPU-PV / driver combos report "N/A",
which CMake rejects (aborting an otherwise-usable build) instead of letting
"native" autodetect.
- Gate the native-Linux aarch64 provisioner on _SKIP_GGUF_BUILD: when a non-root
user declines the sudo prompt (or lacks sudo) for GGUF deps, don't then run a
provisioner that does its own sudo apt-get installs.
- install.sh: gate the new aarch64 bitsandbytes block on SKIP_TORCH=false --
with --no-torch/UNSLOTH_NO_TORCH (GGUF-only install) it would have pulled
torch back into the venv through bitsandbytes' dependencies.
- studio worker: in the new Spark OOM-guard section, decide
PYTORCH_CUDA_ALLOC_CONF (expandable_segments) BEFORE the guard's first CUDA
touch -- get_device_properties initializes the CUDA allocator, after which
the env var is ignored, and the later `import unsloth`
(patch_dgx_spark_memory_config) is too late for the worker process. Uses
the same CUDA-free nvidia-smi name sniff, append-don't-override, and
UNSLOTH_NO_EXPANDABLE_SEGMENTS opt-out as the library patch. Live-verified
on the N1X: env set while torch.cuda.is_initialized() is still False.
- uninstall.ps1: only run `fuser -k 8888/tcp` in a probed WSL distro when an
Unsloth install actually exists there (checked BEFORE the rm deletes the
marker) -- an unrelated listener on 8888 (e.g. Jupyter) in a clean distro
must survive a Windows-side uninstall. The Unsloth-specific pkills stay
unconditional.
- install.ps1 + uninstall.ps1: persist the chosen WSL distro to
%LOCALAPPDATA%\Unsloth\wsl-distro.txt at install; uninstall reads it
(before removing the directory) and prepends it to the cleanup candidates,
so a custom UNSLOTH_WSL_DISTRO install is cleaned without the env var
being set again at uninstall time.
- provision_llama_cuda.sh: honor UNSLOTH_LLAMA_PR (numeric-validated,
best-effort fetch of pull/N/head after clone) so a provisioned tree
matches a PR pin the way setup.sh does; and require only llama-server in
the main cmake build (mirroring setup.sh), building the helper targets
(llama-cli/quantize/mtmd-cli/gguf-split) best-effort afterwards -- an
older UNSLOTH_LLAMA_TAG pin lacking a newer helper target no longer fails
the whole provision.
Not changed: the "--tauri rejection doesn't restore the venv rollback"
comment is incorrect -- the rejection returns through Exit-InstallFailure,
which itself calls Restore-StudioVenvRollback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- install.ps1: detect a PRE-EXISTING WSL1 distro up-front (kernel string +
libcuda probe inside the distro; encoding-proof vs UTF-16 `wsl -l -v`) and
convert it with `wsl --set-version 2`, failing early with instructions if
conversion does not take -- instead of completing a full install that only
fails at the final torch.cuda check (no GPU passthrough under WSL1).
- install.ps1: quote the distro name in the generated unsloth.cmd shim and in
the copy-pasteable hint commands so UNSLOTH_WSL_DISTRO values with spaces
("Ubuntu Preview") keep working.
- install.ps1: forward UNSLOTH_NO_LLAMA_CUDA=1 into the WSL install env; the
inner setup.sh otherwise defers its llama.cpp build to a background builder
this script then never dispatches (the same opt-out skips it), leaving no
llama-server and a misleading "building in background" footer. Also add
libcurl4-openssl-dev to the WSL bootstrap apt line.
- provision_llama_cuda.sh: install libcurl4-openssl-dev with the base tools --
_cmake_configure forces -DLLAMA_CURL=ON and on the deferred WSL path this
script is the only build path (setup.sh's GGUF dep install was skipped), so
configure failed on fresh hosts without the headers.
- provision_llama_cuda.sh: keep the pre-existing llama.cpp backup until the
fresh build is CONFIRMED (was: dropped right after a successful clone), and
restore it on configure/build failure or when no server binary was produced
-- a failed CUDA build no longer destroys a previously working (CPU) server.
- setup.sh: when provisioning fails and NO llama-server is present, set
_LLAMA_CPP_DEGRADED=true so the arm64 CPU-prebuilt last resort and the
installer failure exit fire instead of reporting a working install.
Round-2 comments verified already fixed in ad77ae6 (anchored to its parent
d161ff5): the torch probe already passes --reinstall; the WSL uninstall is
already scoped to /root only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
install.ps1 (WoA WSL fallback):
- report failure (non-zero) + restore the rolled-aside venv when the WSL GPU
install fails (torch.cuda absent) or when WSL needs enabling+reboot, instead of
returning success — so -File/Tauri callers don't see a broken install as complete
- on WSL success, Complete-StudioVenvRollback so the previous-venv backup isn't orphaned
- refuse under --tauri with a clear "use the CLI installer" message (the desktop
launcher resolves a Windows-venv backend, which a WSL-only install can't provide)
- reset $LASTEXITCODE before each wsl.exe / python probe (a stale 0 could mark WSL
ready / torch OK if the native command fails to launch)
- torch-availability probe: --reinstall so an already-installed CPU torch in a
migrated venv isn't accepted as "satisfied" (would wrongly skip the WSL path)
- treat a null HKCU PATH as empty (fresh profile) so shim PATH update can't throw
- keep apt stderr visible inside WSL (only stdout -> /dev/null) for diagnosability
scripts/uninstall.ps1:
- scope WSL cleanup to /root (the fallback's install location); stop deleting
/home/*/.unsloth, which could erase an unrelated WSL user's own Unsloth/cache
studio/setup.sh:
- direct (non-install.ps1) WSL installs now provision CUDA llama.cpp themselves
instead of being left with no GGUF server: install.ps1 exports
UNSLOTH_WSL_LLAMA_DEFERRED=1, and the aarch64+NVIDIA provision block runs under
WSL only when that marker is absent
- mark a provisioner-built llama.cpp as Studio-owned in custom-STUDIO_HOME mode so
the next setup's _assert_studio_owned_or_absent doesn't abort
- glibc>=2.41 check: also match a future major>2 (e.g. 3.0)
studio/scripts/provision_llama_cuda.sh:
- install base tools (cmake/git/curl) in their own apt transaction before the
best-effort gcc-14/g++-14 (unavailable on Ubuntu 22.04 / Debian 12, where bundling
them aborted the whole transaction and left no build tools)
- back up an existing (e.g. CPU-only) llama.cpp before the destructive clone and
restore it on clone failure, so a failed clone doesn't leave the user with no server
- honor a pinned llama.cpp ref via UNSLOTH_LLAMA_TAG instead of always tracking main
unsloth/models/_utils.py:
- set PYTORCH_CUDA_ALLOC_CONF (expandable_segments) via a CUDA-free Spark detector
(nvidia-smi, not torch.cuda.get_device_name) so it takes effect before CUDA/the
caching allocator initialize — previously it was a silent no-op on auto-detected Spark
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A non-numeric or 0 override silently fell through to `cmake -j0`, which
builds with ALL cores -- the opposite of the thermal-headroom default and a
shutdown risk on NVIDIA-ARM laptops. Validate it's a positive integer; ignore
anything else and auto-compute.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An interrupted CUDA build (e.g. a thermal/power shutdown mid-compile -- which
this machine class hits) can leave a partially-linked libggml-cuda.so. On the
next run cmake does not relink it, so linking llama-server fails with undefined
ggml_cuda_op_* references and the script gives up with no server. Mirror the
existing configure retry-clean: if `cmake --build` fails, wipe build/,
reconfigure, and rebuild clean once before giving up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A full -j(nproc) CUDA build trips power/thermal shutdowns on thermally
constrained NVIDIA-ARM laptops (observed on the N1X "RTX Spark": a full-core
build, especially alongside other load, shuts the machine down). nice lowers
CPU *scheduling* priority but not heat -- power/heat scale with the number of
active compile jobs -- so default to ~half the cores instead: still ~2.5x
faster than a tiny -j4, but leaves real headroom. Still mem-capped (~1.5 GB
per nvcc job) and overridable via UNSLOTH_LLAMA_BUILD_JOBS (raise on a
well-cooled box, lower if it still trips). Tiny boxes (<=4 cores) use all.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Building at -j(nproc) saturates every core (load ~25 on a 20-core box),
which starved a concurrently launched `unsloth studio` / training session
during the build's few-minute window. Wrap the cmake build in
`nice -n 19` (+ `ionice -c 3` when available): full speed when the box is
idle, but instant yield to foreground work. Also trims this file's comments.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The CUDA llama.cpp compile is the slow step of the WSL GPU setup. The job
count now defaults to the full core count (nproc) instead of being capped,
which is ~5x faster on a 20-core box (-j4 -> -j20). To stay safe on
unified-memory machines, where nvcc jobs (~1.5 GB each) could OOM-kill a
full-parallel build, jobs are capped at mem/1.5GB when that is lower than
nproc. UNSLOTH_LLAMA_BUILD_JOBS=N still overrides for thermal throttling.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three coupled fixes so the Windows-on-ARM + NVIDIA (DGX Spark / N1X) WSL path
builds the GPU llama-server reliably and never wastes time on a CPU build:
1. install.ps1 -- the post-install CUDA provision was launched as a WSL-side
`nohup setsid ... &`. That does NOT survive: WSL shuts the distro's VM down
once the launching wsl.exe session exits, killing the detached build (observed
on a fresh distro: no build log, only a CPU server left behind). Fetch the
provision script in a quick session, then run the build anchored to a DETACHED
Windows-side process (Start-Process wsl.exe, no -Wait) that holds the VM up for
the whole build while install.ps1 returns immediately.
2. provision_llama_cuda.sh -- a pre-existing build/ can carry an incompatible
CMake cache (the Studio installer stages its build in llama.cpp.build.NNNN then
relocates it, leaving a cache with stale absolute source/build paths and
GGML_CUDA=OFF), so reconfiguring for CUDA fails ("CMakeCache directory is
different" / "source does not match"). Try to reuse build/ first (incremental
resume), and if configure fails, wipe build/ and configure clean once. Verified
live on the failing scenario: stale cache detected, wiped, clean CUDA configure.
(setup.sh's skip of the CPU source build on this path is the companion commit;
together the fresh-install path builds only the CUDA server, in the background.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Current llama.cpp ships the CUDA backend as a dynamically-loaded plugin
(libggml-cuda.so* next to the binary), NOT a load-time dependency, so
ldd llama-server | grep libggml-cuda is a false negative: it reports no
CUDA on a perfectly good CUDA build. That made both is_cuda_server()
(provision_llama_cuda.sh) and _have_cuda_llama_server() (setup.sh) force a
needless full rebuild every run.
Fix both: keep the ldd check (old monolithic builds) and additionally treat
the presence of libggml-cuda.so* beside the binary as the CUDA signal. A
CPU-only build has no such backend, so this stays correct for the CPU case.
Verified on an N1X/sm_121 WSL build: llama-server --list-devices shows
CUDA0 JMJWOA-Generic-GPU and serves on the GPU, while ldd lists no
libggml-cuda; the new check correctly returns CUDA-present.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When the installer is launched from a Windows shell, WSL interop leaks the Windows
PATH (/mnt/c/... entries, with spaces) into the build environment, which can make
cmake/gcc/git resolve to Windows tools or otherwise confuse the CUDA build. Prepend
the CUDA toolkit + standard Linux dirs so the Linux toolchain always wins; keep the
original PATH after so nvidia-smi etc. still resolve.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A full -j(nproc) CUDA build is power/thermal-heavy on laptops (e.g. N1X) and can
trip a thermal/power shutdown mid-build. Allow lowering the job count; cmake
--build is incremental so re-running resumes from where it stopped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A clean-slate reinstall on an ARM64+NVIDIA box surfaced three follow-on gaps in
the WSL path. All fixes are additive, best-effort, and confined to the $torchOk
success branch of the WSL fallback, so they only run on the ARM64+NVIDIA machines
that reach it -- no other platform is affected.
install.ps1:
- Self-heal Studio's web-server deps. install_python_stack.py installs the Studio
UI deps (fastapi/uvicorn/structlog/starlette) in a late step; if that run is cut
short, torch+unsloth land but the server stack is missing and `unsloth studio`
dies at launch on ModuleNotFoundError. Import-check the stack after the torch.cuda
probe and, if absent, install it WITHOUT re-pinning huggingface-hub/transformers/
datasets, so the verified GPU torch path is never disturbed.
- Seed pip into the (uv-managed, pip-less) venv via ensurepip so save_pretrained_gguf
-> check_pip() works regardless of how Studio is launched.
- Auto-build a CUDA llama-server for GGUF inference in the background via the new
provision script (below), so GGUF chat/tool-calling lights up a few minutes after
install with zero manual steps. Opt out with UNSLOTH_NO_LLAMA_CUDA=1.
studio/scripts/provision_llama_cuda.sh (new):
- Idempotent, best-effort (always exits 0). Builds a CUDA llama.cpp into
~/.unsloth/llama.cpp (Studio's resolver path). Generic across NVIDIA Linux/WSL
incl. aarch64 (DGX Spark, N1X): derives the arch from the GPU's compute_cap,
installs gcc-14 + CUDA 13.3 only when nvcc is missing (gcc-15 is rejected by nvcc;
CUDA <13.3 hits the glibc>=2.41 rsqrt header clash), and builds the full target set
(llama-server llama-cli llama-quantize llama-mtmd-cli llama-gguf-split) so it
satisfies both Studio inference and save_pretrained_gguf without a later rebuild.
Validated on an NVIDIA N1X (sm_121): training, GPU inference, GGUF q4_k_m export,
`unsloth studio` via both Desktop + Start Menu shortcuts (HTTP 200), GGUF chat at
121 tok/s (BLACKWELL_NATIVE_FP4=1) and OpenAI-style tool-calling.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
