Use prebuilt llama.cpp for unsloth studio setup (#4562)
* Use prebuilt llama.cpp for unsloth studio setup * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix 3 issues that cause unnecessary fallback to source build 1. Make filelock import optional -- environments without filelock (e.g. minimal installs) crashed at import time instead of gracefully skipping the lock. 2. Use already-verified converter script from the hydrated source tree instead of re-downloading from raw.githubusercontent.com with no checksum. Adds symlink with copy fallback for the legacy filename. 3. Initialize $SkipPrebuiltInstall in setup.ps1 before first use to prevent potential uninitialized variable errors. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Keep network fallback in ensure_converter_scripts Prefer the local verified copy from the hydrated source tree, but retain the original network download as a fallback if the file is missing. Create the legacy hyphenated filename as a symlink with a copy fallback instead of writing a second full copy. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix 4 bugs in source-build fallback and binary_env paths - setup.ps1: Replace git pull + checkout FETCH_HEAD with fetch + checkout -B to avoid detached HEAD state that breaks re-runs. Use pinned tag in both fetch and clone paths. - setup.sh: Move rm -rf after cmake/git prerequisite checks so a missing tool no longer deletes the existing install. Add --branch tag to clone. - install_llama_prebuilt.py: Add binary_path.parent to Linux LD_LIBRARY_PATH in binary_env() so bundled .so files in build/bin are found even without RPATH, matching the existing Windows PATH logic. - Add test for binary_env LD_LIBRARY_PATH on Linux. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Handle unresolved "latest" tag in source-build fallback clone When tag resolution fails and the requested tag is "latest", both setup scripts now omit --branch from git clone so the default branch is cloned instead of failing on a nonexistent "latest" branch/tag. Similarly, the PS1 fetch path fetches the default ref when the tag is "latest". * Resolve actual latest ggml-org tag instead of using literal "latest" When both Python tag resolution attempts fail and the requested tag is "latest", query the GitHub API for the actual latest release tag from ggml-org/llama.cpp (e.g. b8508) instead of passing the literal string "latest" to git clone --branch, which would fail since no such branch/tag exists. setup.sh uses curl + python json parsing; setup.ps1 uses Invoke-RestMethod. Both fall back to the raw requested tag if the API call also fails. * Try Unsloth release repo before ggml-org when resolving latest tag When falling back to the GitHub API to resolve "latest", query the Unsloth release repo (unslothai/llama.cpp) first since it has the prebuilt binaries pinned to tested tags. Only fall back to ggml-org/llama.cpp if the Unsloth repo query fails. * Add comprehensive sandbox tests for PR #4562 bug fixes 35 tests covering all fixes across platforms: - binary_env cross-platform (Linux LD_LIBRARY_PATH, Windows PATH, macOS DYLD_LIBRARY_PATH) with edge cases (dedup, ordering, existing paths) - resolve_requested_llama_tag (concrete, latest, None, empty) - setup.sh logic via subprocess: prereq check ordering (cmake/git missing preserves install), pinned tag in clone, fetch+checkout -B pattern, fetch failure warns instead of aborting - "latest" tag resolution fallback chain (Unsloth API -> ggml-org -> raw) with mock curl: success, failure, malformed JSON, empty body, empty tag_name, env overrides - Source code pattern verification for both .sh and .ps1 files All 138 tests pass in isolated uv venv. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add binary_path.parent to macOS DYLD_LIBRARY_PATH in binary_env macOS prebuilt .dylib files are overlaid into build/bin (same as Linux), but binary_env only added install_dir to DYLD_LIBRARY_PATH. Add binary_path.parent so the loader can find sibling dylibs even without embedded loader paths. Mirrors the existing fix for Linux LD_LIBRARY_PATH and the Windows PATH pattern. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Guard --branch when resolved tag is "latest"; fix broken test assertion When all API fallbacks fail and the tag stays as literal "latest", omit --branch from git clone (clones default branch instead of failing). Both setup.sh and setup.ps1 now check for "latest" before passing --branch to git clone/fetch. Also fix test_setup_ps1_clone_uses_branch_tag which used Python tuple syntax (assert "x", "y" in z) that always passes. Changed to assert "x" in z and "y" in z. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix macOS DYLD trailing colon, install_lock no-op, and debug log - binary_env macOS: use dedupe_existing_dirs instead of raw string concatenation. Eliminates trailing colon in DYLD_LIBRARY_PATH (which causes dyld to search CWD for libraries) and deduplicates when binary_path.parent == install_dir. Now consistent with the Linux and Windows branches. - install_lock: when filelock is not installed, use os.O_CREAT|O_EXCL as a fallback exclusive file lock with timeout, instead of yielding with no locking. Prevents concurrent installs from corrupting each other's staging directories. - setup.ps1: remove [DEBUG] log line that printed to every user on every Windows setup run. * Add stale-lock detection and atomic clone-then-swap install_lock fallback (no filelock): write PID to lock file and check if the holder process is still alive on contention. Dead PIDs (ProcessLookupError) and unreadable lock files trigger immediate cleanup. Live processes owned by other users (PermissionError) are correctly recognized as alive -- the lock is not removed. setup.sh/setup.ps1 source-build: clone into a temporary directory first, then swap into place only on success. If git clone fails, the existing install is preserved instead of being deleted by the premature rm -rf. * Remove redundant upstream_tag != release_tag check load_approved_release_checksums compared checksums.upstream_tag against the Unsloth release_tag, which are different namespaces (upstream ggml-org tag vs Unsloth published tag). This only worked because both happened to be "b8508" by convention. Would break if Unsloth ever uses a different release naming scheme. The existing check at parse_approved_release_checksums (line 950) already validates the release_tag field correctly. * Fix lock TOCTOU race and build-in-temp-dir swap install_lock fallback: add os.fsync(fd) after writing PID to ensure the PID is visible to racing processes before they check. Treat empty lock files (PID not yet written) as "wait and retry" instead of stale, closing the window where two processes could both see an empty file, both unlink it, and both acquire the lock. setup.sh/setup.ps1 source-build: clone AND build in a temp directory (LLAMA_CPP_DIR.build.$$). Only swap into the final LLAMA_CPP_DIR after the build succeeds. If clone or cmake or build fails, the temp dir is cleaned up and the existing working install is preserved. Previously, rm -rf ran after clone but before build, destroying the existing install even if the build later failed. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Daniel Han <danielhanchen@gmail.com>
This commit is contained in:
DoubleMathew
GitHub
parent
cc1be75621
commit
f4d8a246bf
7 changed files with 6044 additions and 62 deletions
3395
studio/install_llama_prebuilt.py
Executable file
3395
studio/install_llama_prebuilt.py
Executable file
File diff suppressed because it is too large
Load diff
226
studio/setup.ps1
226
studio/setup.ps1
|
|
@ -503,7 +503,6 @@ if ($DriverMaxCuda) {
|
|||
$isCompat = ($tkMaj -lt $drMajorCuda) -or ($tkMaj -eq $drMajorCuda -and $tkMin -le $drMinorCuda)
|
||||
if ($isCompat) {
|
||||
# Also verify the toolkit supports our GPU architecture
|
||||
Write-Host " [DEBUG] Checking CUDA compatibility: toolkit=$tkMaj.$tkMin arch=sm_$CudaArch" -ForegroundColor Magenta
|
||||
$archOk = $true
|
||||
if ($CudaArch) {
|
||||
$archOk = Test-NvccArchSupport -NvccExe $candidateNvcc -Arch $CudaArch
|
||||
|
|
@ -1296,6 +1295,93 @@ if ($LASTEXITCODE -ne 0) {
|
|||
$ErrorActionPreference = $prevEAP_t5
|
||||
Write-Host "[OK] Transformers 5.x pre-installed to .venv_t5/" -ForegroundColor Green
|
||||
|
||||
# ==========================================================================
|
||||
# PHASE 3.4: Prefer prebuilt llama.cpp bundles before source build
|
||||
# ==========================================================================
|
||||
$UnslothHome = Join-Path $env:USERPROFILE ".unsloth"
|
||||
if (-not (Test-Path $UnslothHome)) { New-Item -ItemType Directory -Force $UnslothHome | Out-Null }
|
||||
$LlamaCppDir = Join-Path $UnslothHome "llama.cpp"
|
||||
$NeedLlamaSourceBuild = $false
|
||||
$SkipPrebuiltInstall = $false
|
||||
$RequestedLlamaTag = if ($env:UNSLOTH_LLAMA_TAG) { $env:UNSLOTH_LLAMA_TAG } else { "latest" }
|
||||
$HelperReleaseRepo = if ($env:UNSLOTH_LLAMA_RELEASE_REPO) { $env:UNSLOTH_LLAMA_RELEASE_REPO } else { "unslothai/llama.cpp" }
|
||||
$resolveOutput = & python "$PSScriptRoot\install_llama_prebuilt.py" --resolve-install-tag $RequestedLlamaTag --published-repo $HelperReleaseRepo 2>&1
|
||||
$resolveExit = $LASTEXITCODE
|
||||
$ResolvedLlamaTag = if ($resolveOutput) { ($resolveOutput | Select-Object -Last 1).ToString().Trim() } else { "" }
|
||||
if ($resolveExit -ne 0 -or [string]::IsNullOrWhiteSpace($ResolvedLlamaTag)) {
|
||||
Write-Host ""
|
||||
Write-Host "[WARN] Failed to resolve an installable prebuilt llama.cpp tag via $HelperReleaseRepo" -ForegroundColor Yellow
|
||||
if ($resolveOutput) {
|
||||
$resolveOutput | ForEach-Object { Write-Host $_ }
|
||||
}
|
||||
$fallbackOutput = & python "$PSScriptRoot\install_llama_prebuilt.py" --resolve-llama-tag $RequestedLlamaTag 2>$null
|
||||
$fallbackExit = $LASTEXITCODE
|
||||
$ResolvedLlamaTag = if ($fallbackExit -eq 0 -and $fallbackOutput) {
|
||||
($fallbackOutput | Select-Object -Last 1).ToString().Trim()
|
||||
} elseif ($RequestedLlamaTag -eq "latest") {
|
||||
# Try Unsloth release repo first, then fall back to ggml-org upstream
|
||||
$resolvedLatest = $null
|
||||
try {
|
||||
$latestRelease = Invoke-RestMethod -Uri "https://api.github.com/repos/$HelperReleaseRepo/releases/latest" -ErrorAction Stop
|
||||
$resolvedLatest = $latestRelease.tag_name
|
||||
} catch {}
|
||||
if (-not $resolvedLatest) {
|
||||
try {
|
||||
$latestRelease = Invoke-RestMethod -Uri "https://api.github.com/repos/ggml-org/llama.cpp/releases/latest" -ErrorAction Stop
|
||||
$resolvedLatest = $latestRelease.tag_name
|
||||
} catch {}
|
||||
}
|
||||
if ($resolvedLatest) { $resolvedLatest } else { $RequestedLlamaTag }
|
||||
} else {
|
||||
$RequestedLlamaTag
|
||||
}
|
||||
$NeedLlamaSourceBuild = $true
|
||||
$SkipPrebuiltInstall = $true
|
||||
}
|
||||
|
||||
Write-Host ""
|
||||
Write-Host "Resolved llama.cpp release tag: $ResolvedLlamaTag" -ForegroundColor Gray
|
||||
|
||||
if ($env:UNSLOTH_LLAMA_FORCE_COMPILE -eq "1") {
|
||||
Write-Host ""
|
||||
Write-Host "[WARN] UNSLOTH_LLAMA_FORCE_COMPILE=1 -- skipping prebuilt llama.cpp install" -ForegroundColor Yellow
|
||||
$NeedLlamaSourceBuild = $true
|
||||
} else {
|
||||
Write-Host ""
|
||||
Write-Host "Installing prebuilt llama.cpp bundle (preferred path)..." -ForegroundColor Cyan
|
||||
if (Test-Path $LlamaCppDir) {
|
||||
Write-Host "Existing llama.cpp install detected -- validating staged prebuilt update before replacement" -ForegroundColor Gray
|
||||
}
|
||||
if ($SkipPrebuiltInstall) {
|
||||
Write-Host "[WARN] Skipping prebuilt install because prebuilt tag resolution failed -- falling back to source build" -ForegroundColor Yellow
|
||||
} else {
|
||||
$prebuiltArgs = @(
|
||||
"$PSScriptRoot\install_llama_prebuilt.py",
|
||||
"--install-dir", $LlamaCppDir,
|
||||
"--llama-tag", $ResolvedLlamaTag,
|
||||
"--published-repo", $HelperReleaseRepo
|
||||
)
|
||||
if ($env:UNSLOTH_LLAMA_RELEASE_TAG) {
|
||||
$prebuiltArgs += @("--published-release-tag", $env:UNSLOTH_LLAMA_RELEASE_TAG)
|
||||
}
|
||||
$prevEAPPrebuilt = $ErrorActionPreference
|
||||
$ErrorActionPreference = "Continue"
|
||||
& python @prebuiltArgs
|
||||
$prebuiltExit = $LASTEXITCODE
|
||||
$ErrorActionPreference = $prevEAPPrebuilt
|
||||
|
||||
if ($prebuiltExit -eq 0) {
|
||||
Write-Host "[OK] Prebuilt llama.cpp installed and validated" -ForegroundColor Green
|
||||
} else {
|
||||
if (Test-Path $LlamaCppDir) {
|
||||
Write-Host "[WARN] Prebuilt update failed; existing install was restored or cleaned before source build fallback" -ForegroundColor Yellow
|
||||
}
|
||||
Write-Host "[WARN] Prebuilt llama.cpp path unavailable or failed validation -- falling back to source build" -ForegroundColor Yellow
|
||||
$NeedLlamaSourceBuild = $true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# ==========================================================================
|
||||
# PHASE 3.5: Install OpenSSL dev (for HTTPS support in llama-server)
|
||||
# ==========================================================================
|
||||
|
|
@ -1303,42 +1389,46 @@ Write-Host "[OK] Transformers 5.x pre-installed to .venv_t5/" -ForegroundColor G
|
|||
# ShiningLight.OpenSSL.Dev includes headers + libs that cmake can find.
|
||||
$OpenSslAvailable = $false
|
||||
|
||||
# Check if OpenSSL dev is already installed (look for include dir)
|
||||
$OpenSslRoots = @(
|
||||
'C:\Program Files\OpenSSL-Win64',
|
||||
'C:\Program Files\OpenSSL',
|
||||
'C:\OpenSSL-Win64'
|
||||
)
|
||||
$OpenSslRoot = $null
|
||||
foreach ($root in $OpenSslRoots) {
|
||||
if (Test-Path (Join-Path $root 'include\openssl\ssl.h')) {
|
||||
$OpenSslRoot = $root
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if ($OpenSslRoot) {
|
||||
$OpenSslAvailable = $true
|
||||
Write-Host "[OK] OpenSSL dev found at $OpenSslRoot" -ForegroundColor Green
|
||||
} else {
|
||||
Write-Host ""
|
||||
Write-Host "Installing OpenSSL dev (for HTTPS in llama-server)..." -ForegroundColor Cyan
|
||||
$HasWinget = $null -ne (Get-Command winget -ErrorAction SilentlyContinue)
|
||||
if ($HasWinget) {
|
||||
winget install -e --id ShiningLight.OpenSSL.Dev --accept-package-agreements --accept-source-agreements
|
||||
# Re-check after install
|
||||
foreach ($root in $OpenSslRoots) {
|
||||
if (Test-Path (Join-Path $root 'include\openssl\ssl.h')) {
|
||||
$OpenSslRoot = $root
|
||||
$OpenSslAvailable = $true
|
||||
Write-Host "[OK] OpenSSL dev installed at $OpenSslRoot" -ForegroundColor Green
|
||||
break
|
||||
}
|
||||
if ($NeedLlamaSourceBuild) {
|
||||
# Check if OpenSSL dev is already installed (look for include dir)
|
||||
$OpenSslRoots = @(
|
||||
'C:\Program Files\OpenSSL-Win64',
|
||||
'C:\Program Files\OpenSSL',
|
||||
'C:\OpenSSL-Win64'
|
||||
)
|
||||
$OpenSslRoot = $null
|
||||
foreach ($root in $OpenSslRoots) {
|
||||
if (Test-Path (Join-Path $root 'include\openssl\ssl.h')) {
|
||||
$OpenSslRoot = $root
|
||||
break
|
||||
}
|
||||
}
|
||||
if (-not $OpenSslAvailable) {
|
||||
Write-Host "[WARN] OpenSSL dev not available -- llama-server will be built without HTTPS" -ForegroundColor Yellow
|
||||
|
||||
if ($OpenSslRoot) {
|
||||
$OpenSslAvailable = $true
|
||||
Write-Host "[OK] OpenSSL dev found at $OpenSslRoot" -ForegroundColor Green
|
||||
} else {
|
||||
Write-Host ""
|
||||
Write-Host "Installing OpenSSL dev (for HTTPS in llama-server)..." -ForegroundColor Cyan
|
||||
$HasWinget = $null -ne (Get-Command winget -ErrorAction SilentlyContinue)
|
||||
if ($HasWinget) {
|
||||
winget install -e --id ShiningLight.OpenSSL.Dev --accept-package-agreements --accept-source-agreements
|
||||
# Re-check after install
|
||||
foreach ($root in $OpenSslRoots) {
|
||||
if (Test-Path (Join-Path $root 'include\openssl\ssl.h')) {
|
||||
$OpenSslRoot = $root
|
||||
$OpenSslAvailable = $true
|
||||
Write-Host "[OK] OpenSSL dev installed at $OpenSslRoot" -ForegroundColor Green
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if (-not $OpenSslAvailable) {
|
||||
Write-Host "[WARN] OpenSSL dev not available -- llama-server will be built without HTTPS" -ForegroundColor Yellow
|
||||
}
|
||||
}
|
||||
} else {
|
||||
Write-Host "[SKIP] OpenSSL dev install -- prebuilt llama.cpp already validated" -ForegroundColor Yellow
|
||||
}
|
||||
|
||||
# ==========================================================================
|
||||
|
|
@ -1351,9 +1441,7 @@ if ($OpenSslRoot) {
|
|||
# - llama-server: for GGUF model inference (with HTTPS if OpenSSL available)
|
||||
# - llama-quantize: for GGUF export quantization
|
||||
# Prerequisites (git, cmake, VS Build Tools, CUDA Toolkit) already installed in Phase 1.
|
||||
$UnslothHome = Join-Path $env:USERPROFILE ".unsloth"
|
||||
if (-not (Test-Path $UnslothHome)) { New-Item -ItemType Directory -Force $UnslothHome | Out-Null }
|
||||
$LlamaCppDir = Join-Path $UnslothHome "llama.cpp"
|
||||
$OriginalLlamaCppDir = $LlamaCppDir
|
||||
$BuildDir = Join-Path $LlamaCppDir "build"
|
||||
$LlamaServerBin = Join-Path $BuildDir "bin\Release\llama-server.exe"
|
||||
|
||||
|
|
@ -1376,7 +1464,10 @@ if (Test-Path $LlamaServerBin) {
|
|||
}
|
||||
}
|
||||
|
||||
if ((Test-Path $LlamaServerBin) -and -not $NeedRebuild) {
|
||||
if (-not $NeedLlamaSourceBuild) {
|
||||
Write-Host ""
|
||||
Write-Host "[OK] Using validated prebuilt llama.cpp install at $LlamaCppDir" -ForegroundColor Green
|
||||
} elseif ((Test-Path $LlamaServerBin) -and -not $NeedRebuild) {
|
||||
Write-Host ""
|
||||
Write-Host "[OK] llama-server already exists at $LlamaServerBin" -ForegroundColor Green
|
||||
} elseif (-not $HasCmakeForBuild) {
|
||||
|
|
@ -1432,29 +1523,49 @@ if ((Test-Path $LlamaServerBin) -and -not $NeedRebuild) {
|
|||
|
||||
# -- Step A: Clone or pull llama.cpp --
|
||||
|
||||
$UseConcreteRef = ($ResolvedLlamaTag -ne "latest" -and -not [string]::IsNullOrWhiteSpace($ResolvedLlamaTag))
|
||||
|
||||
if (Test-Path (Join-Path $LlamaCppDir ".git")) {
|
||||
Write-Host " llama.cpp repo already cloned, pulling latest..." -ForegroundColor Gray
|
||||
git -C $LlamaCppDir pull 2>&1 | Out-Null
|
||||
Write-Host " Syncing llama.cpp to $ResolvedLlamaTag..." -ForegroundColor Gray
|
||||
if ($UseConcreteRef) {
|
||||
git -C $LlamaCppDir fetch --depth 1 origin $ResolvedLlamaTag 2>&1 | Out-Null
|
||||
} else {
|
||||
git -C $LlamaCppDir fetch --depth 1 origin 2>&1 | Out-Null
|
||||
}
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
Write-Host " [WARN] git pull failed -- using existing source" -ForegroundColor Yellow
|
||||
Write-Host " [WARN] git fetch failed -- using existing source" -ForegroundColor Yellow
|
||||
} else {
|
||||
git -C $LlamaCppDir checkout -B unsloth-llama-build FETCH_HEAD 2>&1 | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
$BuildOk = $false
|
||||
$FailedStep = "git checkout"
|
||||
} else {
|
||||
git -C $LlamaCppDir clean -fdx 2>&1 | Out-Null
|
||||
}
|
||||
}
|
||||
} else {
|
||||
Write-Host " Cloning llama.cpp..." -ForegroundColor Gray
|
||||
if (Test-Path $LlamaCppDir) { Remove-Item -Recurse -Force $LlamaCppDir }
|
||||
git clone --depth 1 https://github.com/ggml-org/llama.cpp.git $LlamaCppDir 2>&1 | Out-Null
|
||||
Write-Host " Cloning llama.cpp @ $ResolvedLlamaTag..." -ForegroundColor Gray
|
||||
$buildTmp = "$LlamaCppDir.build.$PID"
|
||||
if (Test-Path $buildTmp) { Remove-Item -Recurse -Force $buildTmp }
|
||||
$cloneArgs = @("clone", "--depth", "1")
|
||||
if ($UseConcreteRef) {
|
||||
$cloneArgs += @("--branch", $ResolvedLlamaTag)
|
||||
}
|
||||
$cloneArgs += @("https://github.com/ggml-org/llama.cpp.git", $buildTmp)
|
||||
git @cloneArgs 2>&1 | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
$BuildOk = $false
|
||||
$FailedStep = "git clone"
|
||||
if (Test-Path $buildTmp) { Remove-Item -Recurse -Force $buildTmp }
|
||||
}
|
||||
# Use temp dir for build; swap into $LlamaCppDir only after build succeeds
|
||||
if ($BuildOk) {
|
||||
$LlamaCppDir = $buildTmp
|
||||
$BuildDir = Join-Path $LlamaCppDir "build"
|
||||
}
|
||||
}
|
||||
|
||||
# -- Step B: cmake configure --
|
||||
# Clean stale CMake cache to prevent previous CUDA settings from leaking
|
||||
# into a CPU-only rebuild (or vice versa).
|
||||
$CmakeCacheFile = Join-Path $BuildDir "CMakeCache.txt"
|
||||
if (Test-Path $CmakeCacheFile) {
|
||||
Remove-Item -Recurse -Force $BuildDir
|
||||
}
|
||||
|
||||
if ($BuildOk) {
|
||||
Write-Host ""
|
||||
|
|
@ -1555,6 +1666,21 @@ if ((Test-Path $LlamaServerBin) -and -not $NeedRebuild) {
|
|||
}
|
||||
}
|
||||
|
||||
# Swap temp build dir into final location (only if we built in a temp dir)
|
||||
if ($BuildOk -and $LlamaCppDir -ne $OriginalLlamaCppDir) {
|
||||
if (Test-Path $OriginalLlamaCppDir) { Remove-Item -Recurse -Force $OriginalLlamaCppDir }
|
||||
Move-Item $LlamaCppDir $OriginalLlamaCppDir
|
||||
$LlamaCppDir = $OriginalLlamaCppDir
|
||||
$BuildDir = Join-Path $LlamaCppDir "build"
|
||||
$LlamaServerBin = Join-Path $BuildDir "bin\Release\llama-server.exe"
|
||||
} elseif (-not $BuildOk -and $LlamaCppDir -ne $OriginalLlamaCppDir) {
|
||||
# Build failed -- clean up temp dir, preserve existing install
|
||||
if (Test-Path $LlamaCppDir) { Remove-Item -Recurse -Force $LlamaCppDir }
|
||||
$LlamaCppDir = $OriginalLlamaCppDir
|
||||
$BuildDir = Join-Path $LlamaCppDir "build"
|
||||
$LlamaServerBin = Join-Path $BuildDir "bin\Release\llama-server.exe"
|
||||
}
|
||||
|
||||
# Restore ErrorActionPreference
|
||||
$ErrorActionPreference = $prevEAP
|
||||
|
||||
|
|
|
|||
127
studio/setup.sh
127
studio/setup.sh
|
|
@ -341,10 +341,98 @@ else
|
|||
echo "✅ Python dependencies up to date — skipping"
|
||||
fi
|
||||
|
||||
# ── 7. WSL: pre-install GGUF build dependencies ──
|
||||
# ── 7. Prefer prebuilt llama.cpp bundles before any source build path ──
|
||||
UNSLOTH_HOME="$HOME/.unsloth"
|
||||
mkdir -p "$UNSLOTH_HOME"
|
||||
LLAMA_CPP_DIR="$UNSLOTH_HOME/llama.cpp"
|
||||
LLAMA_SERVER_BIN="$LLAMA_CPP_DIR/build/bin/llama-server"
|
||||
_NEED_LLAMA_SOURCE_BUILD=false
|
||||
_LLAMA_FORCE_COMPILE="${UNSLOTH_LLAMA_FORCE_COMPILE:-0}"
|
||||
_REQUESTED_LLAMA_TAG="${UNSLOTH_LLAMA_TAG:-latest}"
|
||||
_HELPER_RELEASE_REPO="${UNSLOTH_LLAMA_RELEASE_REPO:-unslothai/llama.cpp}"
|
||||
_RESOLVE_LLAMA_LOG="$(mktemp)"
|
||||
set +e
|
||||
python "$SCRIPT_DIR/install_llama_prebuilt.py" \
|
||||
--resolve-install-tag "$_REQUESTED_LLAMA_TAG" \
|
||||
--published-repo "$_HELPER_RELEASE_REPO" >"$_RESOLVE_LLAMA_LOG" 2>&1
|
||||
_RESOLVE_LLAMA_STATUS=$?
|
||||
set -e
|
||||
if [ "$_RESOLVE_LLAMA_STATUS" -eq 0 ]; then
|
||||
_RESOLVED_LLAMA_TAG="$(tail -n 1 "$_RESOLVE_LLAMA_LOG" | tr -d '\r')"
|
||||
else
|
||||
_RESOLVED_LLAMA_TAG=""
|
||||
fi
|
||||
if [ -z "$_RESOLVED_LLAMA_TAG" ]; then
|
||||
echo ""
|
||||
echo "⚠️ Failed to resolve an installable prebuilt llama.cpp tag via $_HELPER_RELEASE_REPO"
|
||||
cat "$_RESOLVE_LLAMA_LOG" >&2 || true
|
||||
set +e
|
||||
_RESOLVED_LLAMA_TAG="$(python "$SCRIPT_DIR/install_llama_prebuilt.py" --resolve-llama-tag "$_REQUESTED_LLAMA_TAG" 2>/dev/null)"
|
||||
_RESOLVE_UPSTREAM_STATUS=$?
|
||||
set -e
|
||||
if [ "$_RESOLVE_UPSTREAM_STATUS" -ne 0 ] || [ -z "$_RESOLVED_LLAMA_TAG" ]; then
|
||||
if [ "$_REQUESTED_LLAMA_TAG" = "latest" ]; then
|
||||
# Try Unsloth release repo first, then fall back to ggml-org upstream
|
||||
_RESOLVED_LLAMA_TAG="$(curl -fsSL "https://api.github.com/repos/${_HELPER_RELEASE_REPO}/releases/latest" 2>/dev/null | python -c "import sys,json; print(json.load(sys.stdin)['tag_name'])" 2>/dev/null)" || _RESOLVED_LLAMA_TAG=""
|
||||
if [ -z "$_RESOLVED_LLAMA_TAG" ]; then
|
||||
_RESOLVED_LLAMA_TAG="$(curl -fsSL https://api.github.com/repos/ggml-org/llama.cpp/releases/latest 2>/dev/null | python -c "import sys,json; print(json.load(sys.stdin)['tag_name'])" 2>/dev/null)" || _RESOLVED_LLAMA_TAG=""
|
||||
fi
|
||||
fi
|
||||
if [ -z "$_RESOLVED_LLAMA_TAG" ]; then
|
||||
_RESOLVED_LLAMA_TAG="$_REQUESTED_LLAMA_TAG"
|
||||
fi
|
||||
fi
|
||||
_NEED_LLAMA_SOURCE_BUILD=true
|
||||
_SKIP_PREBUILT_INSTALL=true
|
||||
fi
|
||||
rm -f "$_RESOLVE_LLAMA_LOG"
|
||||
|
||||
echo ""
|
||||
echo "Resolved llama.cpp release tag: $_RESOLVED_LLAMA_TAG"
|
||||
|
||||
if [ "$_LLAMA_FORCE_COMPILE" = "1" ]; then
|
||||
echo ""
|
||||
echo "⚠️ UNSLOTH_LLAMA_FORCE_COMPILE=1 -- skipping prebuilt llama.cpp install"
|
||||
_NEED_LLAMA_SOURCE_BUILD=true
|
||||
else
|
||||
echo ""
|
||||
echo "Installing prebuilt llama.cpp bundle (preferred path)..."
|
||||
if [ -d "$LLAMA_CPP_DIR" ]; then
|
||||
echo "Existing llama.cpp install detected -- validating staged prebuilt update before replacement"
|
||||
fi
|
||||
if [ "${_SKIP_PREBUILT_INSTALL:-false}" = true ]; then
|
||||
echo "⚠️ Skipping prebuilt install because prebuilt tag resolution failed -- falling back to source build"
|
||||
else
|
||||
_PREBUILT_CMD=(
|
||||
python "$SCRIPT_DIR/install_llama_prebuilt.py"
|
||||
--install-dir "$LLAMA_CPP_DIR"
|
||||
--llama-tag "$_RESOLVED_LLAMA_TAG"
|
||||
--published-repo "$_HELPER_RELEASE_REPO"
|
||||
)
|
||||
if [ -n "${UNSLOTH_LLAMA_RELEASE_TAG:-}" ]; then
|
||||
_PREBUILT_CMD+=(--published-release-tag "$UNSLOTH_LLAMA_RELEASE_TAG")
|
||||
fi
|
||||
set +e
|
||||
"${_PREBUILT_CMD[@]}"
|
||||
_PREBUILT_STATUS=$?
|
||||
set -e
|
||||
|
||||
if [ "$_PREBUILT_STATUS" -eq 0 ]; then
|
||||
echo "✅ Prebuilt llama.cpp installed and validated"
|
||||
else
|
||||
if [ -d "$LLAMA_CPP_DIR" ]; then
|
||||
echo "⚠️ Prebuilt update failed; existing install was restored or cleaned before source build fallback"
|
||||
fi
|
||||
echo "⚠️ Prebuilt llama.cpp path unavailable or failed validation -- falling back to source build"
|
||||
_NEED_LLAMA_SOURCE_BUILD=true
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── 8. WSL: pre-install GGUF build dependencies for fallback source builds ──
|
||||
# On WSL, sudo requires a password and can't be entered during GGUF export
|
||||
# (runs in a non-interactive subprocess). Install build deps here instead.
|
||||
if grep -qi microsoft /proc/version 2>/dev/null; then
|
||||
if [ "$_NEED_LLAMA_SOURCE_BUILD" = true ] && grep -qi microsoft /proc/version 2>/dev/null; then
|
||||
echo ""
|
||||
echo "⚠️ WSL detected -- installing build dependencies for GGUF export..."
|
||||
_GGUF_DEPS="pciutils build-essential cmake curl git libcurl4-openssl-dev"
|
||||
|
|
@ -402,22 +490,19 @@ if grep -qi microsoft /proc/version 2>/dev/null; then
|
|||
fi
|
||||
fi
|
||||
|
||||
# ── 8. Build llama.cpp binaries for GGUF inference + export ──
|
||||
# ── 9. Build llama.cpp binaries for GGUF inference + export when prebuilt install fails ──
|
||||
# Builds at ~/.unsloth/llama.cpp — a single shared location under the user's
|
||||
# home directory. This is used by both the inference server and the GGUF
|
||||
# export pipeline (unsloth-zoo).
|
||||
# - llama-server: for GGUF model inference
|
||||
# - llama-quantize: for GGUF export quantization (symlinked to root for check_llama_cpp())
|
||||
UNSLOTH_HOME="$HOME/.unsloth"
|
||||
mkdir -p "$UNSLOTH_HOME"
|
||||
LLAMA_CPP_DIR="$UNSLOTH_HOME/llama.cpp"
|
||||
LLAMA_SERVER_BIN="$LLAMA_CPP_DIR/build/bin/llama-server"
|
||||
if [ "${_SKIP_GGUF_BUILD:-}" = true ]; then
|
||||
if [ "$_NEED_LLAMA_SOURCE_BUILD" = false ]; then
|
||||
:
|
||||
elif [ "${_SKIP_GGUF_BUILD:-}" = true ]; then
|
||||
echo ""
|
||||
echo "Skipping llama-server build (missing dependencies)"
|
||||
echo " Install the missing packages and re-run setup to enable GGUF inference."
|
||||
else
|
||||
rm -rf "$LLAMA_CPP_DIR"
|
||||
{
|
||||
# Check prerequisites
|
||||
if ! command -v cmake &>/dev/null; then
|
||||
|
|
@ -432,7 +517,13 @@ rm -rf "$LLAMA_CPP_DIR"
|
|||
echo "Building llama-server for GGUF inference..."
|
||||
|
||||
BUILD_OK=true
|
||||
run_quiet_no_exit "clone llama.cpp" git clone --depth 1 https://github.com/ggml-org/llama.cpp.git "$LLAMA_CPP_DIR" || BUILD_OK=false
|
||||
_CLONE_BRANCH_ARGS=()
|
||||
if [ "$_RESOLVED_LLAMA_TAG" != "latest" ] && [ -n "$_RESOLVED_LLAMA_TAG" ]; then
|
||||
_CLONE_BRANCH_ARGS=(--branch "$_RESOLVED_LLAMA_TAG")
|
||||
fi
|
||||
_BUILD_TMP="${LLAMA_CPP_DIR}.build.$$"
|
||||
rm -rf "$_BUILD_TMP"
|
||||
run_quiet_no_exit "clone llama.cpp" git clone --depth 1 "${_CLONE_BRANCH_ARGS[@]}" https://github.com/ggml-org/llama.cpp.git "$_BUILD_TMP" || BUILD_OK=false
|
||||
|
||||
if [ "$BUILD_OK" = true ]; then
|
||||
# Skip tests/examples we don't need (faster build)
|
||||
|
|
@ -571,21 +662,29 @@ rm -rf "$LLAMA_CPP_DIR"
|
|||
CMAKE_GENERATOR_ARGS="-G Ninja"
|
||||
fi
|
||||
|
||||
run_quiet_no_exit "cmake llama.cpp" cmake $CMAKE_GENERATOR_ARGS -S "$LLAMA_CPP_DIR" -B "$LLAMA_CPP_DIR/build" $CMAKE_ARGS || BUILD_OK=false
|
||||
run_quiet_no_exit "cmake llama.cpp" cmake $CMAKE_GENERATOR_ARGS -S "$_BUILD_TMP" -B "$_BUILD_TMP/build" $CMAKE_ARGS || BUILD_OK=false
|
||||
fi
|
||||
|
||||
if [ "$BUILD_OK" = true ]; then
|
||||
run_quiet_no_exit "build llama-server" cmake --build "$LLAMA_CPP_DIR/build" --config Release --target llama-server -j"$NCPU" || BUILD_OK=false
|
||||
run_quiet_no_exit "build llama-server" cmake --build "$_BUILD_TMP/build" --config Release --target llama-server -j"$NCPU" || BUILD_OK=false
|
||||
fi
|
||||
|
||||
# Also build llama-quantize (needed by unsloth-zoo's GGUF export pipeline)
|
||||
if [ "$BUILD_OK" = true ]; then
|
||||
run_quiet_no_exit "build llama-quantize" cmake --build "$LLAMA_CPP_DIR/build" --config Release --target llama-quantize -j"$NCPU" || true
|
||||
# Symlink to llama.cpp root — check_llama_cpp() looks for the binary there
|
||||
run_quiet_no_exit "build llama-quantize" cmake --build "$_BUILD_TMP/build" --config Release --target llama-quantize -j"$NCPU" || true
|
||||
fi
|
||||
|
||||
# Swap only after build succeeds -- preserves existing install on failure
|
||||
if [ "$BUILD_OK" = true ]; then
|
||||
rm -rf "$LLAMA_CPP_DIR"
|
||||
mv "$_BUILD_TMP" "$LLAMA_CPP_DIR"
|
||||
# Symlink to llama.cpp root -- check_llama_cpp() looks for the binary there
|
||||
QUANTIZE_BIN="$LLAMA_CPP_DIR/build/bin/llama-quantize"
|
||||
if [ -f "$QUANTIZE_BIN" ]; then
|
||||
ln -sf build/bin/llama-quantize "$LLAMA_CPP_DIR/llama-quantize"
|
||||
fi
|
||||
else
|
||||
rm -rf "$_BUILD_TMP"
|
||||
fi
|
||||
|
||||
if [ "$BUILD_OK" = true ]; then
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue