ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

WSL fallback: self-heal Studio server deps, seed pip, auto-build CUDA llama.cpp

Browse source 
A clean-slate reinstall on an ARM64+NVIDIA box surfaced three follow-on gaps in
the WSL path. All fixes are additive, best-effort, and confined to the $torchOk
success branch of the WSL fallback, so they only run on the ARM64+NVIDIA machines
that reach it -- no other platform is affected.

install.ps1:
- Self-heal Studio's web-server deps. install_python_stack.py installs the Studio
  UI deps (fastapi/uvicorn/structlog/starlette) in a late step; if that run is cut
  short, torch+unsloth land but the server stack is missing and `unsloth studio`
  dies at launch on ModuleNotFoundError. Import-check the stack after the torch.cuda
  probe and, if absent, install it WITHOUT re-pinning huggingface-hub/transformers/
  datasets, so the verified GPU torch path is never disturbed.
- Seed pip into the (uv-managed, pip-less) venv via ensurepip so save_pretrained_gguf
  -> check_pip() works regardless of how Studio is launched.
- Auto-build a CUDA llama-server for GGUF inference in the background via the new
  provision script (below), so GGUF chat/tool-calling lights up a few minutes after
  install with zero manual steps. Opt out with UNSLOTH_NO_LLAMA_CUDA=1.

studio/scripts/provision_llama_cuda.sh (new):
- Idempotent, best-effort (always exits 0). Builds a CUDA llama.cpp into
  ~/.unsloth/llama.cpp (Studio's resolver path). Generic across NVIDIA Linux/WSL
  incl. aarch64 (DGX Spark, N1X): derives the arch from the GPU's compute_cap,
  installs gcc-14 + CUDA 13.3 only when nvcc is missing (gcc-15 is rejected by nvcc;
  CUDA <13.3 hits the glibc>=2.41 rsqrt header clash), and builds the full target set
  (llama-server llama-cli llama-quantize llama-mtmd-cli llama-gguf-split) so it
  satisfies both Studio inference and save_pretrained_gguf without a later rebuild.

Validated on an NVIDIA N1X (sm_121): training, GPU inference, GGUF q4_k_m export,
`unsloth studio` via both Desktop + Start Menu shortcuts (HTTP 200), GGUF chat at
121 tok/s (BLACKWELL_NATIVE_FP4=1) and OpenAI-style tool-calling.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Daniel Han 2026-06-02 06:06:25 -07:00
commit 25c11b3e98
2 changed files with 198 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

75
install.ps1
View file

@ -1533,6 +1533,50 @@ shell.Run cmd, 0, False
& wsl.exe -d $distro -u root -- /root/.unsloth/studio/unsloth_studio/bin/python -c "import torch,sys; sys.exit(0 if torch.cuda.is_available() else 3)" *> $null
$torchOk = ($LASTEXITCODE -eq 0)
} catch {} finally { $ErrorActionPreference = $prevEapChk }
# Self-heal Studio's web-server deps. install.sh installs them in a late step
# (install_python_stack.py "studio deps", step 8). If that step is cut short --
# an interrupted download, or a transient resolver hiccup -- torch + unsloth still
# land, but the server stack (fastapi/uvicorn/structlog/starlette) is missing and
# `unsloth studio` dies at launch with ModuleNotFoundError. If the stack can't
# import, install it WITHOUT disturbing the working ML stack: we deliberately do
# NOT pin huggingface-hub / transformers / datasets here, so the GPU torch path
# we just verified stays intact (those pins live in studio.txt for a fresh env).
if ($torchOk) {
$_studioPy = "/root/.unsloth/studio/unsloth_studio/bin/python"
$_serverOk = $false
$prevEapS = $ErrorActionPreference; $ErrorActionPreference = "Continue"
try {
& wsl.exe -d $distro -u root -- $_studioPy -c "import structlog, fastapi, uvicorn, starlette" *> $null
$_serverOk = ($LASTEXITCODE -eq 0)
} catch {} finally { $ErrorActionPreference = $prevEapS }
if (-not $_serverOk) {
substep "Studio web-server deps incomplete (install.sh step cut short) -- installing them now..." "Cyan"
# Mirrors studio/backend/requirements/studio.txt MINUS the huggingface-hub
# pin (protected above). Prefer uv (matches install.sh); fall back to pip.
$_deps = 'typer fastapi uvicorn matplotlib pandas nest_asyncio pyjwt easydict addict "structlog>=24.1.0" diceware ddgs "cryptography>=42.0.0" "httpx>=0.27.0" "fastmcp>=3.0.2"'
$_repair = 'PY=/root/.unsloth/studio/unsloth_studio/bin/python; UV="$(command -v uv 2>/dev/null || echo /root/.local/bin/uv)"; if [ -x "$UV" ] || command -v uv >/dev/null 2>&1; then "$UV" pip install --python "$PY" ' + $_deps + '; else "$PY" -m pip install ' + $_deps + '; fi'
$prevEapR = $ErrorActionPreference; $ErrorActionPreference = "Continue"
try { & wsl.exe -d $distro -u root -- bash -lc $_repair } catch {} finally { $ErrorActionPreference = $prevEapR }
$prevEapS2 = $ErrorActionPreference; $ErrorActionPreference = "Continue"
try {
& wsl.exe -d $distro -u root -- $_studioPy -c "import structlog, fastapi, uvicorn, starlette" *> $null
$_serverOk = ($LASTEXITCODE -eq 0)
} catch {} finally { $ErrorActionPreference = $prevEapS2 }
if ($_serverOk) { substep "Studio web-server deps installed." "Green" }
else { substep "(could not auto-install Studio server deps; 'unsloth studio' may fail to start)" "Yellow" }
}
# GGUF export robustness: the venv is uv-managed and ships no `pip`, but
# unsloth-zoo's exporter calls check_pip() and only finds `uv pip` when uv is
# on PATH (true for the login-shell launcher, not for every code path).
# Seeding pip into the venv makes `save_pretrained_gguf` work regardless.
$prevEapP = $ErrorActionPreference; $ErrorActionPreference = "Continue"
try {
& wsl.exe -d $distro -u root -- $_studioPy -m pip --version *> $null
if ($LASTEXITCODE -ne 0) {
& wsl.exe -d $distro -u root -- $_studioPy -m ensurepip --upgrade *> $null
}
} catch {} finally { $ErrorActionPreference = $prevEapP }
}
if ($torchOk) {
step "done" "Unsloth Studio installed in WSL '$distro' -- GPU ready (torch.cuda available)." "Green"
# Native Windows `unsloth` shim: forward every `unsloth ...` into the WSL GPU env so the user
@ -1587,12 +1631,39 @@ shell.Run cmd, 0, False
$sc.Save()
}
step "shortcuts" "created Desktop + Start Menu shortcuts (launch WSL Studio + open browser)" "Green"
# Refresh the shell icon cache so the brand-new .lnk icons render immediately instead of
# showing blank (Explorer caches per-.lnk icons; programmatically-created links often need a poke).
# Make the brand-new .lnk icons render immediately instead of blank. Explorer caches
# per-.lnk icons, so a freshly-created shortcut often shows blank until the shell is told
# to re-read it. ie4uinit -show alone is unreliable; also broadcast SHChangeNotify so
# Explorer refreshes the icons without needing a restart or re-login.
try { & "$env:SystemRoot\System32\ie4uinit.exe" -show 2>$null } catch {}
try {
if (-not ("UnslothShell.Notify" -as [type])) {
Add-Type -Namespace UnslothShell -Name Notify -MemberDefinition '[System.Runtime.InteropServices.DllImport("shell32.dll")] public static extern void SHChangeNotify(int eventId, uint flags, System.IntPtr item1, System.IntPtr item2);'
}
# SHCNE_ASSOCCHANGED (0x08000000) with SHCNF_IDLIST (0) -> flush shell icon associations.
[UnslothShell.Notify]::SHChangeNotify(0x08000000, 0, [System.IntPtr]::Zero, [System.IntPtr]::Zero)
} catch {}
} catch {
substep "(could not create shortcuts: $($_.Exception.Message))" "Yellow"
}
# GGUF *inference* needs a CUDA-linked llama-server. There is no published
# aarch64+CUDA llama.cpp prebuilt (NVIDIA DGX Spark / N1X), so build one into
# ~/.unsloth/llama.cpp (Studio's resolver path) IN THE BACKGROUND: the user gets
# Studio + training immediately, and GGUF inference lights up a few minutes later
# with zero manual steps. Best-effort; opt out with UNSLOTH_NO_LLAMA_CUDA=1.
if ($env:UNSLOTH_NO_LLAMA_CUDA -ne '1') {
$prevEapL = $ErrorActionPreference; $ErrorActionPreference = "Continue"
try {
$_llamaUrl = "https://raw.githubusercontent.com/unslothai/unsloth/main/studio/scripts/provision_llama_cuda.sh"
$_provCmd = 'mkdir -p /root/.unsloth; if curl -fsSL "' + $_llamaUrl + '" -o /root/.unsloth/provision_llama_cuda.sh && [ -s /root/.unsloth/provision_llama_cuda.sh ]; then chmod +x /root/.unsloth/provision_llama_cuda.sh; nohup setsid bash /root/.unsloth/provision_llama_cuda.sh > /root/.unsloth/llama_cuda_build.log 2>&1 < /dev/null & echo PROV_STARTED; else echo PROV_NOSCRIPT; fi'
$_provOut = & wsl.exe -d $distro -u root -- bash -lc $_provCmd 2>$null
if ("$_provOut" -match 'PROV_STARTED') {
step "llama.cpp" "building CUDA llama.cpp for GGUF inference in the background (a few min); log: ~/.unsloth/llama_cuda_build.log" "Green"
} else {
substep "(GGUF inference needs a CUDA llama.cpp build; build later: wsl -d $distro -u root -- bash ~/.unsloth/provision_llama_cuda.sh)" "Yellow"
}
} catch {} finally { $ErrorActionPreference = $prevEapL }
}
} else {
step "wsl" "WSL Studio install did not finish cleanly (torch.cuda not detected; inner exit $wslRc) -- see log above." "Yellow"
substep "retry, or launch manually: wsl -d $distro -u root -- bash -lic 'unsloth studio -p 8888'" "Cyan"

125
studio/scripts/provision_llama_cuda.sh Normal file
View file

@ -0,0 +1,125 @@
#!/usr/bin/env bash
# Provision a CUDA-enabled llama.cpp for Unsloth Studio GGUF *inference*.
#
# Builds into ~/.unsloth/llama.cpp (the dir Unsloth Studio's llama-server
# resolver checks: <dir>/build/bin/llama-server). Best-effort and idempotent:
# safe to re-run, never hard-fails the caller (always exits 0).
#
# Why this exists: torch ships its own bundled CUDA runtime, so training +
# GGUF *export* work without a system CUDA toolkit. But GGUF *inference* needs
# a CUDA-linked llama-server, and on NVIDIA ARM machines (NVIDIA DGX Spark /
# GB10, N1X "RTX" laptops) there is no published aarch64+CUDA prebuilt, so we
# build one. Handles the known gotchas on these platforms:
# * nvcc rejects gcc-15 -> force gcc-14 / g++-14 as the host compiler
# * glibc >= 2.41 vs CUDA < 13.3 -> install CUDA 13.3 (rsqrt header clash)
# * sm_121 (Blackwell) GPUs -> derive arch from the GPU's compute_cap
#
# Opt out entirely with UNSLOTH_NO_LLAMA_CUDA=1 (handled by the caller).
set -uo pipefail
LLAMA_DIR="${UNSLOTH_LLAMA_CPP_PATH:-$HOME/.unsloth/llama.cpp}"
SERVER="$LLAMA_DIR/build/bin/llama-server"
log() { printf ' - %s\n' "$*"; }
is_cuda_server() { [ -x "$1" ] && ldd "$1" 2>/dev/null | grep -qi 'libggml-cuda'; }
# 0. Already provisioned?
if is_cuda_server "$SERVER"; then
log "CUDA llama-server already present: $SERVER"
exit 0
fi
# 1. Require an NVIDIA GPU (this script is only meaningful with one).
if ! command -v nvidia-smi >/dev/null 2>&1; then
log "no nvidia-smi found; skipping CUDA llama.cpp build"
exit 0
fi
SUDO=""; [ "$(id -u)" -ne 0 ] && SUDO="sudo"
HAVE_APT=0; command -v apt-get >/dev/null 2>&1 && HAVE_APT=1
# 2. Base toolchain. gcc-14 is required because nvcc rejects gcc-15.
if [ "$HAVE_APT" -eq 1 ]; then
$SUDO apt-get update -y >/dev/null 2>&1 || true
$SUDO apt-get install -y --no-install-recommends \
build-essential cmake git curl ca-certificates gcc-14 g++-14 >/dev/null 2>&1 || true
fi
# 3. Locate nvcc; install the CUDA toolkit if missing.
find_nvcc() { command -v nvcc 2>/dev/null || ls /usr/local/cuda*/bin/nvcc 2>/dev/null | sort -V | tail -1; }
NVCC="$(find_nvcc)"
if [ -z "$NVCC" ] && [ "$HAVE_APT" -eq 1 ]; then
log "CUDA toolkit (nvcc) not found - installing CUDA 13.3 (matches torch cu13x; avoids glibc>=2.41 rsqrt clash)"
# shellcheck disable=SC1091
. /etc/os-release 2>/dev/null || true
case "$(uname -m)" in
aarch64) NV_ARCH=sbsa ;;
x86_64) NV_ARCH=x86_64 ;;
*) NV_ARCH="" ;;
esac
case "${ID:-}${VERSION_ID:-}" in
ubuntu24.04) NV_DISTRO=ubuntu2404 ;;
ubuntu22.04) NV_DISTRO=ubuntu2204 ;;
debian12) NV_DISTRO=debian12 ;;
*) NV_DISTRO="" ;;
esac
if [ -n "$NV_ARCH" ] && [ -n "$NV_DISTRO" ]; then
KR=/tmp/cuda-keyring.deb
if curl -fsSL "https://developer.download.nvidia.com/compute/cuda/repos/$NV_DISTRO/$NV_ARCH/cuda-keyring_1.1-1_all.deb" -o "$KR" 2>/dev/null; then
$SUDO dpkg -i "$KR" >/dev/null 2>&1 || true
$SUDO apt-get update -y >/dev/null 2>&1 || true
$SUDO apt-get install -y cuda-toolkit-13-3 >/dev/null 2>&1 \
|| $SUDO apt-get install -y cuda-toolkit >/dev/null 2>&1 || true
fi
fi
NVCC="$(find_nvcc)"
fi
if [ -z "$NVCC" ]; then
log "could not provision a CUDA toolkit. Training + GGUF export still work;"
log "GGUF *inference* in Studio will be unavailable until a CUDA toolkit exists."
log "Re-run this script after installing one to enable GGUF inference."
exit 0
fi
CUDA_HOME="$(dirname "$(dirname "$NVCC")")"
export PATH="$CUDA_HOME/bin:$PATH"
export CUDAToolkit_ROOT="$CUDA_HOME"
# 4. Host compiler: prefer gcc-14 / g++-14 (nvcc rejects 15).
HCC=gcc; command -v gcc-14 >/dev/null 2>&1 && HCC=gcc-14
HCXX=g++; command -v g++-14 >/dev/null 2>&1 && HCXX=g++-14
export CC="$HCC" CXX="$HCXX" CUDAHOSTCXX="$HCXX"
# 5. CUDA arch from the GPU's compute capability (e.g. "12.1" -> 121). Fallback: native.
CC_CAP="$(nvidia-smi --query-gpu=compute_cap --format=csv,noheader 2>/dev/null | head -1 | tr -d ' .')"
if [ -n "$CC_CAP" ]; then CUDA_ARCH="$CC_CAP"; else CUDA_ARCH="native"; fi
# 6. Clone + build into ~/.unsloth/llama.cpp.
mkdir -p "$(dirname "$LLAMA_DIR")"
if [ ! -d "$LLAMA_DIR/.git" ]; then
rm -rf "$LLAMA_DIR"
git clone --depth 1 https://github.com/ggml-org/llama.cpp "$LLAMA_DIR" >/dev/null 2>&1 \
|| { log "git clone failed"; exit 0; }
fi
cd "$LLAMA_DIR" || exit 0
log "building CUDA llama.cpp (arch=$CUDA_ARCH, host=$HCXX) - this takes a few minutes..."
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
-DGGML_CUDA=ON -DGGML_CUDA_F16=ON \
-DCMAKE_CUDA_ARCHITECTURES="$CUDA_ARCH" \
-DCMAKE_CUDA_HOST_COMPILER="$HCXX" \
-DLLAMA_CURL=ON >/dev/null 2>&1 || { log "cmake configure failed"; exit 0; }
# Build the full set unsloth-zoo's GGUF exporter expects too (llama-mtmd-cli,
# llama-gguf-split), so a pre-provisioned build satisfies both Studio inference
# AND save_pretrained_gguf without triggering a --clean-first rebuild later.
cmake --build build -j"$(nproc)" --target \
llama-server llama-cli llama-quantize llama-mtmd-cli llama-gguf-split >/dev/null 2>&1 \
|| { log "cmake build failed"; exit 0; }
if is_cuda_server "$SERVER"; then
log "CUDA llama-server ready: $SERVER"
else
log "build finished but CUDA llama-server could not be confirmed"
fi
exit 0