Make the Studio installers (sh + ps1) resilient to transient uv download failures (#6281)
* Make Studio installer resilient to transient uv download failures
Updating an existing Studio install via install.sh could hard-fail and roll
back when a wheel download (torch, unsloth) hit a transient connection reset:
x Failed to download unsloth==2026.6.6
error decoding response body -> error reading a body from connection
-> connection reset
restoring previous environment after failed install...
Root cause: that error chain is a mid-stream HTTP/2 body read failure. uv did
not retry this class until 0.8.16 (astral-sh/uv#15675, h2 was shadowing the
underlying IO error), but the installer pinned UV_MIN_VERSION=0.7.22, so a stale
uv got zero retries and a single blip aborted the whole update under set -e.
Fix (installer only, backwards compatible, no change on success):
- Raise UV_MIN_VERSION to 0.8.16 so stale uv is upgraded to a version that
retries HTTP/2 streaming body errors.
- Export UV_HTTP_RETRIES=5 and UV_HTTP_TIMEOUT=180 (override-preserving :=).
- Add run_install_cmd_retry (retry-with-backoff around run_install_cmd) and use
it for the network-heavy uv pip install steps (torch, unsloth, unsloth-zoo
from git, ROCm torch repair, no-torch runtime deps). Local editable overlays
and venv creation are left to fail fast.
run_install_cmd_retry preserves the final exit code on permanent failure, so the
existing set -e rollback trap still fires.
* Apply the same transient-download resilience to the Windows installer
install.ps1 is the native-Windows installer and had the identical issue as
install.sh: it pinned $UvMinVersion=0.7.22 (below uv 0.8.16, which is where uv
started retrying HTTP/2 streaming body errors), set no UV_HTTP_* defaults, and
ran each 'uv pip install' once via Invoke-InstallCommand, so a single connection
reset aborted the update and triggered the Exit-InstallFailure rollback.
install.ps1:
- Raise $UvMinVersion to 0.8.16.
- Default $env:UV_HTTP_RETRIES=5 and $env:UV_HTTP_TIMEOUT=180 (preserving overrides).
- Add Invoke-InstallCommandRetry and use it for the network-heavy uv pip install
steps (torch, unsloth, unsloth-zoo from git, ROCm torch, no-torch runtime deps).
Local editable overlays and venv creation stay single-shot.
install.sh:
- Align UNSLOTH_INSTALL_RETRIES sanitization with the PowerShell version: a
non-positive-integer value now falls back to the default of 3 instead of
silently disabling retries (set =1 to disable). Keeps both installers identical.
* Adopt pre-marker Studio llama.cpp and sidecar dirs on update
After the uv retry fix, an update now reaches studio/setup.sh, whose
Studio-owned ownership guard rejects a llama.cpp or sidecar venv created by an
earlier install that predates the .unsloth-studio-owned marker:
ERROR: .../llama.cpp already exists and is not marked as a Studio-owned
llama.cpp install.
The marker and UNSLOTH_PREBUILT_INFO.json were introduced in the same commit,
so a directory from before that point carries neither signal and a legitimate
self-update fails for anyone who installed earlier (reported on issue #6274).
Fold a one-time adoption into _assert_studio_owned_or_absent (setup.sh) and
Assert-StudioOwnedOrAbsent (setup.ps1): when a custom-home directory lacks the
marker, backfill it and proceed only when there is positive evidence it belongs
to an established Studio home -- the directory carries UNSLOTH_PREBUILT_INFO.json,
or STUDIO_HOME already holds Studio's CLI shim or studio.conf from a prior run.
Both installers write the shim and studio.conf only after invoking setup, so a
fresh install into a dirty custom home (the case the guard protects) does not
have them yet and is still rejected. The venv marker is excluded because install
writes it before setup and so cannot tell a prior install from a fresh one.
* Review fixes: restrict llama.cpp adoption to dir-local evidence; restore install.sh +x
Addresses the PR review on the marker-migration change.
P1 - the adoption helper keyed on root-level Studio sentinels ($STUDIO_HOME/bin
/unsloth, share/studio.conf), so once a home was recognized every unmarked child
passed to the guard became adoptable, and an unrelated directory at a
Studio-managed path could be silently marked and overwritten. Base adoption on
evidence inside the directory instead:
- UNSLOTH_PREBUILT_INFO.json, written by the prebuilt llama.cpp installer (the
default path, in place well before the marker), or
- a top-level llama-quantize symlink, written by source builds (a plain
llama.cpp checkout keeps the binary under build/bin, not a root symlink).
A foreign llama.cpp now stays rejected even inside an established Studio home,
and sidecar venvs (no such fingerprint) stay subject to the strict guard; their
marker has been written since the guard was introduced, so a real custom install
already carries it.
P2 - restore the executable bit on install.sh; a stray mode change to 100644
would break ./install.sh --local on Unix.
On Windows the prebuilt metadata is the signal; source builds are git checkouts
indistinguishable from a user clone, so they are left to the strict guard.
* Bound UNSLOTH_INSTALL_RETRIES / _DELAY before numeric use
An oversized all-digit override (e.g. a fat-fingered
"99999999999999999999") passed the digit-only validation and then reached the
numeric comparison: POSIX `[ -ge ]` errored with "Illegal number" mid-loop and
could spin instead of falling back, and PowerShell's `[int]` cast threw an
Int32 overflow under $ErrorActionPreference = "Stop" before any install ran.
Sanitize with a length guard + range check (sh) and [int]::TryParse with bounds
(ps1), so out-of-range or oversized values fall back to the default. Bounds:
1..100 retries, 0..3600s base delay.
* Studio installers: scope llama.cpp adoption to prebuilt metadata; reject leading-zero retry delay
setup.sh: drop the top-level llama-quantize symlink as an ownership-adoption signal, leaving UNSLOTH_PREBUILT_INFO.json as the sole fingerprint. The shared ownership guard runs immediately before a destructive replace / rm -rf, and a bare root llama-quantize symlink is user-creatable (a user can keep their own llama.cpp build with such a convenience symlink at a custom UNSLOTH_STUDIO_HOME), so the old check could adopt and then delete a user directory. This matches the Windows installer, which already keeps markerless source builds strict. Pre-marker prebuilt installs still adopt via the metadata file, so the original update fix is preserved.
install.sh: reject leading-zero values for UNSLOTH_INSTALL_RETRY_DELAY. A value like 08 or 09 passed the range check but then hit the backoff doubling $((_ricr_delay * 2)), where a non-octal leading zero is a fatal arithmetic error mid-retry. The 0?* pattern routes such values to the default; bare 0 stays valid.
* Tighten the comments added in this PR
* Condense the comments in this PR
This commit is contained in:
Daniel Han
GitHub
parent
c20f25afac
commit
375350e0b6
4 changed files with 149 additions and 41 deletions
74
install.ps1
74
install.ps1
|
|
@ -482,6 +482,37 @@ function Install-UnslothStudio {
|
|||
}
|
||||
}
|
||||
|
||||
# Retry Invoke-InstallCommand on transient uv download failures with backoff.
|
||||
# Returns the last exit code on permanent failure so rollback still fires.
|
||||
function Invoke-InstallCommandRetry {
|
||||
param(
|
||||
[Parameter(Mandatory = $true, Position = 0)][ScriptBlock]$Command,
|
||||
[string]$Label = "install step"
|
||||
)
|
||||
# Sanitize overrides to a default of 3 (a typo must not disable retries; =1 disables).
|
||||
# TryParse with bounds avoids an Int32 overflow throw. Bounds: 1..100 retries, 0..3600s.
|
||||
$maxAttempts = 3
|
||||
$parsedAttempts = 0
|
||||
if ([int]::TryParse($env:UNSLOTH_INSTALL_RETRIES, [ref]$parsedAttempts) -and $parsedAttempts -ge 1 -and $parsedAttempts -le 100) {
|
||||
$maxAttempts = $parsedAttempts
|
||||
}
|
||||
$delay = 3
|
||||
$parsedDelay = 0
|
||||
if ([int]::TryParse($env:UNSLOTH_INSTALL_RETRY_DELAY, [ref]$parsedDelay) -and $parsedDelay -ge 0 -and $parsedDelay -le 3600) {
|
||||
$delay = $parsedDelay
|
||||
}
|
||||
$attempt = 1
|
||||
while ($true) {
|
||||
$code = Invoke-InstallCommand $Command
|
||||
if ($code -eq 0) { return 0 }
|
||||
if ($attempt -ge $maxAttempts) { return $code }
|
||||
substep ("retrying ""$Label"" after transient failure (attempt $($attempt + 1)/$maxAttempts, waiting ${delay}s)...") "Yellow"
|
||||
Start-Sleep -Seconds $delay
|
||||
$attempt++
|
||||
$delay = $delay * 2
|
||||
}
|
||||
}
|
||||
|
||||
function New-StudioShortcuts {
|
||||
param(
|
||||
[Parameter(Mandatory = $true)][string]$UnslothExePath
|
||||
|
|
@ -1181,7 +1212,7 @@ shell.Run cmd, 0, False
|
|||
|
||||
# ── Install uv ──
|
||||
Write-TauriLog "STEP" "Installing uv package manager"
|
||||
$UvMinVersion = "0.7.22"
|
||||
$UvMinVersion = "0.8.16"
|
||||
function Test-UvVersionOk {
|
||||
$cmd = Get-Command uv -ErrorAction SilentlyContinue
|
||||
if (-not $cmd) { return $false }
|
||||
|
|
@ -1252,6 +1283,15 @@ shell.Run cmd, 0, False
|
|||
$env:UV_COMPILE_BYTECODE_TIMEOUT = "180"
|
||||
}
|
||||
|
||||
# uv >= 0.8.16 retries HTTP/2 streaming body errors; raise retries and read
|
||||
# timeout for large wheel downloads. User-provided values are preserved.
|
||||
if (-not $env:UV_HTTP_RETRIES) {
|
||||
$env:UV_HTTP_RETRIES = "5"
|
||||
}
|
||||
if (-not $env:UV_HTTP_TIMEOUT) {
|
||||
$env:UV_HTTP_TIMEOUT = "180"
|
||||
}
|
||||
|
||||
# ── Create venv (migrate old layout if possible, otherwise fresh) ──
|
||||
# Pass the resolved executable path to uv so it does not re-resolve
|
||||
# a version string back to a conda interpreter.
|
||||
|
|
@ -1930,21 +1970,21 @@ shell.Run cmd, 0, False
|
|||
if ($SkipTorch) {
|
||||
# No-torch: install unsloth + unsloth-zoo with --no-deps, then
|
||||
# runtime deps (typer, safetensors, transformers, etc.) with --no-deps.
|
||||
$baseInstallExit = Invoke-InstallCommand { uv pip install --python $VenvPython --no-deps --reinstall-package unsloth --reinstall-package unsloth-zoo "unsloth>=2026.6.7" unsloth-zoo }
|
||||
$baseInstallExit = Invoke-InstallCommandRetry -Label "install unsloth (migrated no-torch)" { uv pip install --python $VenvPython --no-deps --reinstall-package unsloth --reinstall-package unsloth-zoo "unsloth>=2026.6.7" unsloth-zoo }
|
||||
if ($baseInstallExit -eq 0) {
|
||||
# Resolve pydantic WITH deps so pip pins pydantic-core
|
||||
# to the matching version (no-torch-runtime.txt below
|
||||
# is --no-deps). All transitive deps are torch-free.
|
||||
$baseInstallExit = Invoke-InstallCommand { uv pip install --python $VenvPython pydantic }
|
||||
$baseInstallExit = Invoke-InstallCommandRetry -Label "install pydantic" { uv pip install --python $VenvPython pydantic }
|
||||
}
|
||||
if ($baseInstallExit -eq 0) {
|
||||
$NoTorchReq = Find-NoTorchRuntimeFile
|
||||
if ($NoTorchReq) {
|
||||
$baseInstallExit = Invoke-InstallCommand { uv pip install --python $VenvPython --no-deps -r $NoTorchReq }
|
||||
$baseInstallExit = Invoke-InstallCommandRetry -Label "install no-torch runtime deps" { uv pip install --python $VenvPython --no-deps -r $NoTorchReq }
|
||||
}
|
||||
}
|
||||
} else {
|
||||
$baseInstallExit = Invoke-InstallCommand { uv pip install --python $VenvPython --reinstall-package unsloth --reinstall-package unsloth-zoo "unsloth>=2026.6.7" unsloth-zoo }
|
||||
$baseInstallExit = Invoke-InstallCommandRetry -Label "install unsloth (migrated)" { uv pip install --python $VenvPython --reinstall-package unsloth --reinstall-package unsloth-zoo "unsloth>=2026.6.7" unsloth-zoo }
|
||||
}
|
||||
if ($baseInstallExit -ne 0) {
|
||||
Write-Host "[ERROR] Failed to install unsloth (exit code $baseInstallExit)" -ForegroundColor Red
|
||||
|
|
@ -1958,7 +1998,7 @@ shell.Run cmd, 0, False
|
|||
return (Exit-InstallFailure "Failed to overlay local repo (exit code $overlayExit)" $overlayExit)
|
||||
}
|
||||
substep "overlaying unsloth-zoo from git main..."
|
||||
$zooOverlayExit = Invoke-InstallCommand { uv pip install --python $VenvPython --no-deps --reinstall-package unsloth-zoo "unsloth-zoo @ git+https://github.com/unslothai/unsloth-zoo" }
|
||||
$zooOverlayExit = Invoke-InstallCommandRetry -Label "overlay unsloth-zoo (git main)" { uv pip install --python $VenvPython --no-deps --reinstall-package unsloth-zoo "unsloth-zoo @ git+https://github.com/unslothai/unsloth-zoo" }
|
||||
if ($zooOverlayExit -ne 0) {
|
||||
Write-Host "[ERROR] Failed to overlay unsloth-zoo (exit code $zooOverlayExit)" -ForegroundColor Red
|
||||
return (Exit-InstallFailure "Failed to overlay unsloth-zoo (exit code $zooOverlayExit)" $zooOverlayExit)
|
||||
|
|
@ -1971,7 +2011,7 @@ shell.Run cmd, 0, False
|
|||
Write-TauriLog "STEP" "Installing PyTorch (AMD ROCm Windows)"
|
||||
substep "installing PyTorch from $ROCmIndexUrl..."
|
||||
$torchSpec = if ($ROCmTorchFloor) { $ROCmTorchFloor } else { "torch" }
|
||||
$torchInstallExit = Invoke-InstallCommand { uv pip install --python $VenvPython --force-reinstall --index-url $ROCmIndexUrl $torchSpec torchvision torchaudio }
|
||||
$torchInstallExit = Invoke-InstallCommandRetry -Label "install PyTorch (AMD ROCm)" { uv pip install --python $VenvPython --force-reinstall --index-url $ROCmIndexUrl $torchSpec torchvision torchaudio }
|
||||
if ($torchInstallExit -ne 0) {
|
||||
Write-Host "[ERROR] Failed to install AMD ROCm PyTorch (exit code $torchInstallExit)" -ForegroundColor Red
|
||||
return (Exit-InstallFailure "Failed to install AMD ROCm PyTorch (exit code $torchInstallExit)" $torchInstallExit)
|
||||
|
|
@ -1979,7 +2019,7 @@ shell.Run cmd, 0, False
|
|||
} else {
|
||||
Write-TauriLog "STEP" "Installing PyTorch"
|
||||
substep "installing PyTorch ($TorchIndexUrl)..."
|
||||
$torchInstallExit = Invoke-InstallCommand { uv pip install --python $VenvPython "torch>=2.4,<2.11.0" torchvision torchaudio --index-url $TorchIndexUrl }
|
||||
$torchInstallExit = Invoke-InstallCommandRetry -Label "install PyTorch" { uv pip install --python $VenvPython "torch>=2.4,<2.11.0" torchvision torchaudio --index-url $TorchIndexUrl }
|
||||
if ($torchInstallExit -ne 0) {
|
||||
Write-Host "[ERROR] Failed to install PyTorch (exit code $torchInstallExit)" -ForegroundColor Red
|
||||
return (Exit-InstallFailure "Failed to install PyTorch (exit code $torchInstallExit)" $torchInstallExit)
|
||||
|
|
@ -1991,21 +2031,21 @@ shell.Run cmd, 0, False
|
|||
if ($SkipTorch) {
|
||||
# No-torch: install unsloth + unsloth-zoo with --no-deps, then
|
||||
# runtime deps (typer, safetensors, transformers, etc.) with --no-deps.
|
||||
$baseInstallExit = Invoke-InstallCommand { uv pip install --python $VenvPython --no-deps --upgrade-package unsloth --upgrade-package unsloth-zoo "unsloth>=2026.6.7" unsloth-zoo }
|
||||
$baseInstallExit = Invoke-InstallCommandRetry -Label "install unsloth (no-torch)" { uv pip install --python $VenvPython --no-deps --upgrade-package unsloth --upgrade-package unsloth-zoo "unsloth>=2026.6.7" unsloth-zoo }
|
||||
if ($baseInstallExit -eq 0) {
|
||||
# Same pydantic-with-deps trick as the migrated branch.
|
||||
$baseInstallExit = Invoke-InstallCommand { uv pip install --python $VenvPython pydantic }
|
||||
$baseInstallExit = Invoke-InstallCommandRetry -Label "install pydantic" { uv pip install --python $VenvPython pydantic }
|
||||
}
|
||||
if ($baseInstallExit -eq 0) {
|
||||
$NoTorchReq = Find-NoTorchRuntimeFile
|
||||
if ($NoTorchReq) {
|
||||
$baseInstallExit = Invoke-InstallCommand { uv pip install --python $VenvPython --no-deps -r $NoTorchReq }
|
||||
$baseInstallExit = Invoke-InstallCommandRetry -Label "install no-torch runtime deps" { uv pip install --python $VenvPython --no-deps -r $NoTorchReq }
|
||||
}
|
||||
}
|
||||
} elseif ($StudioLocalInstall) {
|
||||
$baseInstallExit = Invoke-InstallCommand { uv pip install --python $VenvPython --upgrade-package unsloth "unsloth>=2026.6.7" unsloth-zoo }
|
||||
$baseInstallExit = Invoke-InstallCommandRetry -Label "install unsloth (local)" { uv pip install --python $VenvPython --upgrade-package unsloth "unsloth>=2026.6.7" unsloth-zoo }
|
||||
} else {
|
||||
$baseInstallExit = Invoke-InstallCommand { uv pip install --python $VenvPython --upgrade-package unsloth -- "$PackageName" }
|
||||
$baseInstallExit = Invoke-InstallCommandRetry -Label "install unsloth" { uv pip install --python $VenvPython --upgrade-package unsloth -- "$PackageName" }
|
||||
}
|
||||
if ($baseInstallExit -ne 0) {
|
||||
Write-Host "[ERROR] Failed to install unsloth (exit code $baseInstallExit)" -ForegroundColor Red
|
||||
|
|
@ -2020,7 +2060,7 @@ shell.Run cmd, 0, False
|
|||
return (Exit-InstallFailure "Failed to overlay local repo (exit code $overlayExit)" $overlayExit)
|
||||
}
|
||||
substep "overlaying unsloth-zoo from git main..."
|
||||
$zooOverlayExit = Invoke-InstallCommand { uv pip install --python $VenvPython --no-deps --reinstall-package unsloth-zoo "unsloth-zoo @ git+https://github.com/unslothai/unsloth-zoo" }
|
||||
$zooOverlayExit = Invoke-InstallCommandRetry -Label "overlay unsloth-zoo (git main)" { uv pip install --python $VenvPython --no-deps --reinstall-package unsloth-zoo "unsloth-zoo @ git+https://github.com/unslothai/unsloth-zoo" }
|
||||
if ($zooOverlayExit -ne 0) {
|
||||
Write-Host "[ERROR] Failed to overlay unsloth-zoo (exit code $zooOverlayExit)" -ForegroundColor Red
|
||||
return (Exit-InstallFailure "Failed to overlay unsloth-zoo (exit code $zooOverlayExit)" $zooOverlayExit)
|
||||
|
|
@ -2031,7 +2071,7 @@ shell.Run cmd, 0, False
|
|||
Write-TauriLog "STEP" "Installing unsloth"
|
||||
substep "installing unsloth (this may take a few minutes)..."
|
||||
if ($StudioLocalInstall) {
|
||||
$baseInstallExit = Invoke-InstallCommand { uv pip install --python $VenvPython unsloth-zoo "unsloth>=2026.6.7" --torch-backend=auto }
|
||||
$baseInstallExit = Invoke-InstallCommandRetry -Label "install unsloth (auto torch backend)" { uv pip install --python $VenvPython unsloth-zoo "unsloth>=2026.6.7" --torch-backend=auto }
|
||||
if ($baseInstallExit -ne 0) {
|
||||
Write-Host "[ERROR] Failed to install unsloth (exit code $baseInstallExit)" -ForegroundColor Red
|
||||
return (Exit-InstallFailure "Failed to install unsloth (exit code $baseInstallExit)" $baseInstallExit)
|
||||
|
|
@ -2043,13 +2083,13 @@ shell.Run cmd, 0, False
|
|||
return (Exit-InstallFailure "Failed to overlay local repo (exit code $overlayExit)" $overlayExit)
|
||||
}
|
||||
substep "overlaying unsloth-zoo from git main..."
|
||||
$zooOverlayExit = Invoke-InstallCommand { uv pip install --python $VenvPython --no-deps --reinstall-package unsloth-zoo "unsloth-zoo @ git+https://github.com/unslothai/unsloth-zoo" }
|
||||
$zooOverlayExit = Invoke-InstallCommandRetry -Label "overlay unsloth-zoo (git main)" { uv pip install --python $VenvPython --no-deps --reinstall-package unsloth-zoo "unsloth-zoo @ git+https://github.com/unslothai/unsloth-zoo" }
|
||||
if ($zooOverlayExit -ne 0) {
|
||||
Write-Host "[ERROR] Failed to overlay unsloth-zoo (exit code $zooOverlayExit)" -ForegroundColor Red
|
||||
return (Exit-InstallFailure "Failed to overlay unsloth-zoo (exit code $zooOverlayExit)" $zooOverlayExit)
|
||||
}
|
||||
} else {
|
||||
$baseInstallExit = Invoke-InstallCommand { uv pip install --python $VenvPython --torch-backend=auto -- "$PackageName" }
|
||||
$baseInstallExit = Invoke-InstallCommandRetry -Label "install unsloth (auto torch backend)" { uv pip install --python $VenvPython --torch-backend=auto -- "$PackageName" }
|
||||
if ($baseInstallExit -ne 0) {
|
||||
Write-Host "[ERROR] Failed to install unsloth (exit code $baseInstallExit)" -ForegroundColor Red
|
||||
return (Exit-InstallFailure "Failed to install unsloth (exit code $baseInstallExit)" $baseInstallExit)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue