CI: prove an interrupted install can never masquerade as a healthy one

Nothing in CI had ever interrupted an install, which is how the reported failure
shipped: quit the desktop app mid-install, the app SIGTERMs the installer process
group, and if that lands during 'studio deps' the venv loses structlog. Preflight
then probes 'unsloth -h' and 'studio desktop-capabilities', both of which succeed
because the CLI's own deps are core, so the app reported ManagedReady with
can_auto_repair=false while the backend died on import. A permanent dead end.

This kills the installer at each interesting phase and asserts the result is
either genuinely healthy or explicitly repairable, never silently ready. 13 legs
across macos-14, ubuntu-latest and windows: each of the dependency-pass steps
plus the coarse phases (venv, torch, unsloth, setup).

The kill targets the process GROUP, matching install.rs. Killing only the leader
leaves uv and python children to finish the dependency pass, and the test would
quietly prove nothing. Windows has no process groups, so that leg walks the CIM
parent links instead, which is the same reason the app carries windows_job.rs.

One shared probe for all platforms. The Windows check used to be bespoke inline
PowerShell that only ran -h and desktop-capabilities, so it could not observe
studio_install_ok, verify-install or desktop-runtime-check: it would have
reported FALSE_READY for the very PRs that add them, no matter how well they
worked. The probe boots the backend as ground truth and owns the whole process
tree, since terminating only the parent leaves children holding the port.

install.sh runs with --local, which is load-bearing rather than a convenience:
without it the installer resolves unsloth from PyPI and the venv gets the
PUBLISHED CLI, so no branch-side change is present and every deeper probe reports
'absent' regardless of what the branch does.

Verified: against a tree without the detection, windows kill@studio-deps reports
FALSE_READY, reproducing the user report exactly. With #7492 merged the same leg
reports REPAIRABLE, and all 13 legs pass.
This commit is contained in:
Daniel Han 2026-07-28 12:08:10 +00:00
commit 6d99729d58
4 changed files with 629 additions and 0 deletions

103
.github/scripts/interrupt-install.ps1 vendored Normal file
View file

@ -0,0 +1,103 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
#
# Windows counterpart of interrupt-install.sh: run install.ps1 and kill it partway
# through, reproducing a user quitting the desktop app mid-install.
#
# Windows has no process groups, which is why the app carries windows_job.rs -- a Job
# Object is what makes "kill the installer and everything it spawned" work there. This
# script kills the whole process TREE for the same reason: killing only the leader
# leaves uv/python children to finish the dependency pass, and the interruption would
# prove nothing.
#
# Usage:
# pwsh -File .github/scripts/interrupt-install.ps1 -Marker 'studio deps' `
# -LogPath logs/install.log -InstallArgs '-SkipTorch'
[CmdletBinding()]
param(
[string]$Marker = '',
[string]$LogPath = 'logs/install.log',
[string]$InstallArgs = '',
[int]$KillAtSeconds = 900,
[int]$KillAfterMarkerSeconds = 3
)
$ErrorActionPreference = 'Continue'
New-Item -ItemType Directory -Force -Path (Split-Path -Parent $LogPath) | Out-Null
Set-Content -Path $LogPath -Value '' -Encoding utf8
# Stand in for the desktop app, which creates this before spawning the installer and
# clears it only on a terminal outcome (install.rs). We kill install.ps1 directly
# rather than driving the real app, so without this the marker #7490 relies on is
# absent for a reason that has nothing to do with #7490 -- which is exactly what the
# Windows legs reported. Both locations because the Rust side hardcodes
# ~/.unsloth/studio while CI overrides UNSLOTH_STUDIO_HOME. Never cleared: being
# killed is the whole point.
foreach ($dir in @($env:UNSLOTH_STUDIO_HOME, (Join-Path $HOME '.unsloth\studio'))) {
if ([string]::IsNullOrWhiteSpace($dir)) { continue }
try {
New-Item -ItemType Directory -Force -Path $dir -ErrorAction Stop | Out-Null
Set-Content -Path (Join-Path $dir '.desktop-install-in-progress') -Value '' -ErrorAction Stop
} catch { Write-Host "[interrupt] could not seed install marker in ${dir}: $_" }
}
# Run the installer in its own pwsh so stdout can be redirected to the log while we poll.
$argList = @('-NoProfile', '-NonInteractive', '-File', 'install.ps1')
if ($InstallArgs) { $argList += $InstallArgs.Split(' ') }
$proc = Start-Process -FilePath 'pwsh' -ArgumentList $argList `
-RedirectStandardOutput $LogPath -RedirectStandardError "$LogPath.err" `
-PassThru -NoNewWindow
Write-Host "[interrupt] installer pid=$($proc.Id) marker='$Marker' deadline=${KillAtSeconds}s"
function Stop-Tree([int]$RootId) {
# Depth-first: children before parents, so a parent cannot respawn a child we already
# killed. CIM gives us the parent link Windows does not expose via process groups.
$kids = @(Get-CimInstance Win32_Process -Filter "ParentProcessId=$RootId" -ErrorAction SilentlyContinue)
foreach ($k in $kids) { Stop-Tree ([int]$k.ProcessId) }
try { Stop-Process -Id $RootId -Force -ErrorAction Stop; Write-Host "[interrupt] killed pid=$RootId" }
catch { }
}
$killed = $false
$reason = ''
for ($i = 0; $i -lt $KillAtSeconds; $i++) {
if ($proc.HasExited) { $reason = 'exited-before-marker'; break }
if ($Marker) {
$hit = Select-String -Path $LogPath -Pattern $Marker -SimpleMatch:$false -ErrorAction SilentlyContinue
if ($hit) {
$reason = 'marker-hit'
Start-Sleep -Seconds $KillAfterMarkerSeconds
$killed = $true
break
}
}
Start-Sleep -Seconds 1
}
if (-not $killed -and -not $proc.HasExited) { if (-not $reason) { $reason = 'deadline' }; $killed = $true }
if ($killed) {
Write-Host "[interrupt] killing process tree of $($proc.Id) ($reason)"
Stop-Tree $proc.Id
# Any straggler uv/python that reparented away from the installer.
foreach ($name in 'uv', 'python') {
Get-Process -Name $name -ErrorAction SilentlyContinue |
Where-Object { $_.Path -and $_.Path -like "*$env:UNSLOTH_STUDIO_HOME*" } |
ForEach-Object { try { Stop-Process -Id $_.Id -Force } catch { } }
}
}
try { $proc.WaitForExit(30000) | Out-Null } catch { }
$rc = if ($proc.HasExited) { $proc.ExitCode } else { 'running' }
Write-Host "[interrupt] installer exit=$rc reason=$reason killed=$killed"
Write-Host '[interrupt] last log lines:'
Get-Content $LogPath -Tail 15 -ErrorAction SilentlyContinue
if ($Marker -and -not (Select-String -Path $LogPath -Pattern $Marker -ErrorAction SilentlyContinue)) {
Write-Host "::warning::marker '$Marker' never appeared -- killed at the deadline, not the intended step"
}
@(
"interrupt_reason=$reason"
"interrupt_killed=$killed"
"installer_exit=$rc"
) | Set-Content -Path (Join-Path (Split-Path -Parent $LogPath) 'interrupt.env') -Encoding utf8
exit 0

103
.github/scripts/interrupt-install.sh vendored Executable file
View file

@ -0,0 +1,103 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
#
# Run install.sh and SIGTERM it partway through, reproducing what the desktop app does
# when the user quits mid-install: main.rs cleanup_child_processes() ->
# install::stop_install() -> kill the installer PROCESS GROUP (install.rs:798-807).
#
# Killing only the leader would leave `uv`/`python` children running and finishing the
# dep pass, so the interruption has to target the group -- otherwise the test quietly
# proves nothing.
#
# Usage: bash .github/scripts/interrupt-install.sh "<marker>" "<logfile>" [-- install args]
# <marker> regex to wait for in the install log before killing, e.g. "studio deps"
# or "\[TAURI:STEP\] Installing PyTorch". Use "" to kill after --at-seconds.
# Env:
# KILL_AT_SECONDS hard deadline; kill even if the marker never appears (default 900)
# KILL_GRACE seconds to wait for the group to die before SIGKILL (default 10)
set -uo pipefail
MARKER="${1:-}"
LOG="${2:-logs/install.log}"
shift 2 || true
[ "${1:-}" = "--" ] && shift
KILL_AT_SECONDS="${KILL_AT_SECONDS:-900}"
KILL_GRACE="${KILL_GRACE:-10}"
mkdir -p "$(dirname "$LOG")"
: > "$LOG"
# Stand in for the desktop app, which creates this before spawning the installer and
# clears it only on a terminal outcome (install.rs). We kill the installer directly
# rather than driving the real app, so without this the marker #7490 relies on is
# absent for a reason that has nothing to do with #7490. Written to both locations
# because the Rust side hardcodes ~/.unsloth/studio while CI overrides
# UNSLOTH_STUDIO_HOME. Deliberately never cleared: being killed is the whole point.
for _marker_dir in "${UNSLOTH_STUDIO_HOME:-}" "$HOME/.unsloth/studio"; do
[ -n "$_marker_dir" ] || continue
mkdir -p "$_marker_dir" 2>/dev/null || continue
: > "$_marker_dir/.desktop-install-in-progress" 2>/dev/null || true
done
# Job control puts the child in its own process group, so $! is the pgid leader and
# `kill -- -$!` reaches every descendant -- matching the Rust side.
set -m
bash install.sh "$@" > "$LOG" 2>&1 &
PID=$!
set +m
echo "[interrupt] installer pid/pgid=$PID marker='${MARKER}' deadline=${KILL_AT_SECONDS}s"
killed=false
reason=""
for i in $(seq 1 "$KILL_AT_SECONDS"); do
if ! kill -0 "$PID" 2>/dev/null; then
reason="exited-before-marker"
break
fi
if [ -n "$MARKER" ] && grep -qE "$MARKER" "$LOG" 2>/dev/null; then
reason="marker-hit"
# Let it get a beat into the step, so the kill lands mid-work rather than on the
# boundary where the step has not started touching the venv yet.
sleep "${KILL_AFTER_MARKER_SECONDS:-3}"
killed=true
break
fi
sleep 1
done
if [ "$killed" != "true" ] && kill -0 "$PID" 2>/dev/null; then
reason="${reason:-deadline}"
killed=true
fi
if [ "$killed" = "true" ]; then
echo "[interrupt] SIGTERM to process group -$PID ($reason)"
kill -TERM -- -"$PID" 2>/dev/null || kill -TERM "$PID" 2>/dev/null || true
for _ in $(seq 1 "$KILL_GRACE"); do
kill -0 "$PID" 2>/dev/null || break
sleep 1
done
if kill -0 "$PID" 2>/dev/null; then
echo "[interrupt] group survived SIGTERM; SIGKILL"
kill -KILL -- -"$PID" 2>/dev/null || kill -KILL "$PID" 2>/dev/null || true
fi
fi
wait "$PID" 2>/dev/null
rc=$?
echo "[interrupt] installer exit=$rc reason=$reason killed=$killed"
echo "[interrupt] last log lines:"
tail -15 "$LOG" || true
# Report how far it got, so a leg that never reached the target step is visible as such
# rather than passing for the wrong reason.
if [ -n "$MARKER" ] && ! grep -qE "$MARKER" "$LOG" 2>/dev/null; then
echo "::warning::marker '$MARKER' never appeared -- this leg killed at the deadline, not at the intended step"
fi
{
echo "interrupt_reason=$reason"
echo "interrupt_killed=$killed"
echo "installer_exit=$rc"
} > "$(dirname "$LOG")/interrupt.env"
exit 0

View file

@ -0,0 +1,217 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
"""After an install is interrupted, decide whether the desktop app WOULD report the
resulting venv as healthy -- reproducing the Tauri preflight probes so the regression
is testable without building the app.
One implementation for all three platforms. There were briefly two (a shell probe and
an inline PowerShell one), and they diverged: the PowerShell version only ran `-h` and
`desktop-capabilities`, so it could not see the `studio_install_ok`, `verify-install`
or `desktop-runtime-check` signals that the fix PRs introduce -- it would have
reported those PRs as failing no matter how well they worked. A probe that cannot
observe the fix is worse than no probe, hence a single shared one.
The reported bug: quitting the app during the dependency pass SIGTERMs the installer
(install.rs stop_install). Landing in the "studio deps" step drops
studio/backend/requirements/studio.txt, where structlog is declared. Preflight then
probes `unsloth -h` (preflight/managed.rs:419) and `studio desktop-capabilities`
(managed.rs:318); both SUCCEED because typer/click/rich are core, so the app reports
ManagedReady with can_auto_repair=false and the backend dies on `import structlog`.
Verdicts:
HEALTHY the backend boots -- the interruption did no lasting harm
REPAIRABLE the backend is broken AND something reports it, so the app can repair
FALSE_READY the backend is broken and every probe says ready -> THE BUG
Exit: 0 for HEALTHY/REPAIRABLE/NO_CLI, 1 for FALSE_READY, 2 for a usage error.
"""
from __future__ import annotations
import argparse
import json
import os
import socket
import subprocess
import sys
import time
import urllib.error
import urllib.request
from pathlib import Path
def run(cmd: list[str], timeout: int = 120) -> tuple[int, str]:
try:
p = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
return p.returncode, (p.stdout or "") + (p.stderr or "")
except (subprocess.TimeoutExpired, OSError) as e:
return 127, f"{type(e).__name__}: {e}"
def has_subcommand(bin_path: str, args: list[str]) -> bool:
"""Whether the CLI understands a subcommand at all. Older builds do not have the
newer verify commands, and 'absent' must not be confused with 'reported failure'."""
rc, _ = run([bin_path, *args, "--help"], timeout=60)
return rc == 0
def free_port() -> int:
with socket.socket() as s:
s.bind(("127.0.0.1", 0))
return int(s.getsockname()[1])
def main(argv: list[str]) -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("bin", help="path to the unsloth CLI")
ap.add_argument("--port", type=int, default=0, help="0 picks a free port")
ap.add_argument("--out", default="probe", help="directory for probe artefacts")
ap.add_argument("--boot-timeout", type=int, default=120)
a = ap.parse_args(argv)
binp = a.bin
if not Path(binp).exists():
print(f"::error::unsloth bin not found: {binp}")
return 2
out = Path(a.out)
out.mkdir(parents=True, exist_ok=True)
port = a.port or free_port()
facts: dict[str, object] = {}
def say(k: str, v: object) -> None:
facts[k] = v
print(f"[probe] {k:28} = {v}")
# ── the two probes Tauri preflight actually runs ─────────────────────────
rc, log = run([binp, "-h"], timeout=180)
(out / "cli-h.log").write_text(log, encoding="utf-8", errors="replace")
say("cli_h_ok", rc == 0)
rc, caps_raw = run([binp, "studio", "desktop-capabilities", "--json"], timeout=180)
(out / "desktop-capabilities.json").write_text(caps_raw, encoding="utf-8", errors="replace")
say("capabilities_ok", rc == 0)
# studio_install_ok is added by the install-manifest work; absent on older trees,
# which is different from present-and-false.
install_ok: object = "absent"
try:
# The CLI may print a banner before the JSON, so start at the first brace.
brace = caps_raw.find("{")
if brace >= 0:
v = json.loads(caps_raw[brace:]).get("studio_install_ok")
install_ok = "absent" if v is None else bool(v)
except (json.JSONDecodeError, AttributeError):
pass
say("capabilities.studio_install_ok", install_ok)
# ── the deeper probes the fix PRs add ────────────────────────────────────
for label, args in (("verify_install", ["studio", "verify-install"]),
("desktop_runtime_check", ["studio", "desktop-runtime-check"])):
if not has_subcommand(binp, args):
say(label, "absent")
continue
rc, log = run([binp, *args], timeout=300)
(out / f"{label}.log").write_text(log, encoding="utf-8", errors="replace")
say(label, "ok" if rc == 0 else "failed")
# The in-progress marker #7490 writes before spawning the installer.
home = Path(os.environ.get("UNSLOTH_STUDIO_HOME") or (Path.home() / ".unsloth" / "studio"))
say("install_in_progress_marker", (home / ".desktop-install-in-progress").exists())
# ── ground truth: does the backend actually boot? ────────────────────────
# Own the whole process tree: the CLI spawns uvicorn/python children, and
# terminating only the parent leaves them holding the port, so the next leg's
# probe would hang. Same reason the interrupt driver kills the group.
popen_kw: dict = {}
if os.name == "posix":
popen_kw["start_new_session"] = True
else:
popen_kw["creationflags"] = getattr(subprocess, "CREATE_NEW_PROCESS_GROUP", 0)
proc = subprocess.Popen(
[binp, "studio", "--api-only", "-H", "127.0.0.1", "-p", str(port)],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, **popen_kw,
)
backend_ok = False
deadline = time.time() + a.boot_timeout
while time.time() < deadline:
if proc.poll() is not None:
break
for path in ("/api/health", "/healthz"):
try:
with urllib.request.urlopen(f"http://127.0.0.1:{port}{path}", timeout=2) as r:
if r.status == 200:
backend_ok = True
break
except (urllib.error.URLError, OSError, TimeoutError):
pass
if backend_ok:
break
time.sleep(1)
def reap() -> None:
if os.name == "posix":
import signal
for sig in (signal.SIGTERM, signal.SIGKILL):
try:
os.killpg(os.getpgid(proc.pid), sig)
except (ProcessLookupError, PermissionError, OSError):
pass
try:
proc.wait(timeout=10)
return
except subprocess.TimeoutExpired:
continue
else:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
reap()
try:
blog = proc.communicate(timeout=30)[0] or ""
except subprocess.TimeoutExpired:
proc.kill()
blog = proc.communicate()[0] or ""
(out / "backend.log").write_text(blog, encoding="utf-8", errors="replace")
say("backend_ok", backend_ok)
missing = ""
for line in blog.splitlines():
if "ModuleNotFoundError" in line:
missing = line.strip()
if missing:
say("backend_error", missing)
# ── verdict ──────────────────────────────────────────────────────────────
if backend_ok:
verdict = "HEALTHY"
elif (facts.get("verify_install") == "failed"
or facts.get("desktop_runtime_check") == "failed"
or facts.get("capabilities.studio_install_ok") is False
or facts.get("install_in_progress_marker") is True
or not facts.get("cli_h_ok")
or not facts.get("capabilities_ok")):
verdict = "REPAIRABLE"
else:
verdict = "FALSE_READY"
facts["verdict"] = verdict
(out / "verdict.json").write_text(json.dumps(facts, indent=2), encoding="utf-8")
print(f"[probe] VERDICT = {verdict}")
if verdict == "FALSE_READY":
print("::error::Interrupted install reports READY but the backend cannot boot"
f" ({missing or 'import failure'}). Preflight sees -h ok + desktop-capabilities"
" ok, so the app shows ManagedReady with can_auto_repair=false and the user"
" is stuck.")
return 1
if verdict == "REPAIRABLE":
print("[probe] broken install is detectable -> the desktop app can auto-repair")
return 0
if __name__ == "__main__":
raise SystemExit(main(sys.argv[1:]))