3e7fc344bb
Two of the interrupted-install legs were passing without testing anything.
The re-run assertion skipped verdict=NO_CLI, but a kill at "venv" or "torch"
lands before install.sh ever prints "Installing Unsloth" (:2125, :3667, :3961),
so those legs can only ever produce NO_CLI. Three non-gating-exempt cells
(macos-14 kill@venv, macos-14 kill@torch, ubuntu-latest kill@torch) therefore
asserted nothing beyond a marker appearing in a log. NO_CLI is now included:
a re-run must produce a booting backend regardless of how little the first run
managed to install. Each re-run step grows an existence check first, because
the probe exits without writing verdict.json when the binary is absent and the
json.load would crash rather than report.
The Windows straggler sweep matched nothing at all. UNSLOTH_STUDIO_HOME arrives
as D:\a\r\r/.studio-home, since the workflow joins ${{ github.workspace }} with
a forward slash, while Process.Path is all backslashes, so the literal -like
missed even the venv's own python.exe. uv is never under the studio home in any
case: install.ps1 takes it from winget or astral.sh. Normalise the separators,
match uv by name (the runner is ephemeral and runs no other uv), and skip the
home comparison entirely when the variable is empty, which would otherwise turn
the pattern into "**" and kill every python on the runner.
313 lines
16 KiB
YAML
313 lines
16 KiB
YAML
# SPDX-License-Identifier: AGPL-3.0-only
|
|
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
|
|
|
|
# Proves an INTERRUPTED install can never masquerade as a healthy one.
|
|
#
|
|
# Reported failure: a user quits the desktop app while it is installing. The app kills
|
|
# the installer process group (main.rs cleanup_child_processes -> install.rs:798-807),
|
|
# which lands mid "studio deps" -- the step that installs
|
|
# studio/backend/requirements/studio.txt, where structlog is declared. On relaunch,
|
|
# preflight probes `unsloth -h` and `studio desktop-capabilities --json`; both succeed
|
|
# because the CLI's own deps (typer/click/rich) are core, so the app reports
|
|
# ManagedReady with can_auto_repair=false. The backend then dies on
|
|
# `import structlog` and the user is permanently stuck on "Server stopped
|
|
# unexpectedly".
|
|
#
|
|
# Nothing in CI covered this: no job has ever interrupted an install. This workflow
|
|
# kills the installer at each interesting phase and asserts the result is either
|
|
# genuinely healthy or explicitly repairable -- never silently ready.
|
|
|
|
name: Interrupted install recovery
|
|
|
|
on:
|
|
pull_request:
|
|
paths:
|
|
- 'install.sh'
|
|
- 'install.ps1'
|
|
- 'studio/setup.sh'
|
|
- 'studio/setup.ps1'
|
|
- 'studio/install_python_stack.py'
|
|
- 'studio/src-tauri/src/install.rs'
|
|
- 'studio/src-tauri/src/preflight.rs'
|
|
- 'studio/src-tauri/src/preflight/**'
|
|
- 'unsloth_cli/commands/studio.py'
|
|
# studio_install_ok and verify-install, the two decisions the probe asserts
|
|
# on, are implemented here rather than in commands/studio.py, so a change
|
|
# that made install_state() accept a missing manifest would otherwise merge
|
|
# without a single leg running.
|
|
- 'unsloth_cli/_studio_deps.py'
|
|
- 'studio/install_manifest.py'
|
|
# The requirement files are the phases. studio.txt is where structlog is
|
|
# declared, the package whose absence IS the reported false-ready bug, and
|
|
# the single-env files drive the later steps, so moving a package between
|
|
# them changes what every interrupted state looks like.
|
|
- 'studio/backend/requirements/**'
|
|
# `*` never matches `/`, and it is a literal `-install` that follows, so
|
|
# `interrupt*-install*` matches interrupt-install.sh / .ps1 but NOT the
|
|
# underscored probe. List the probe explicitly rather than rely on a glob.
|
|
- '.github/scripts/interrupt-install.sh'
|
|
- '.github/scripts/interrupt-install.ps1'
|
|
- '.github/scripts/interrupted_install_probe.py'
|
|
- '.github/workflows/interrupted-install-ci.yml'
|
|
workflow_dispatch:
|
|
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
env:
|
|
UNSLOTH_STUDIO_DISABLE_PUBLIC_CHECK: '1'
|
|
|
|
jobs:
|
|
# ── macOS + Linux: kill at each phase ─────────────────────────────────────
|
|
interrupt:
|
|
name: ${{ matrix.os }} kill@${{ matrix.label }}
|
|
runs-on: ${{ matrix.os }}
|
|
timeout-minutes: 60
|
|
continue-on-error: ${{ matrix.experimental }}
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
# The exact reported case: killed during the step that installs structlog.
|
|
- {os: macos-14, label: studio-deps, marker: 'studio deps', experimental: false}
|
|
# Coarse phases, earliest to latest -- each leaves a different partial venv.
|
|
- {os: macos-14, label: venv, marker: '\[TAURI:STEP\] Creating virtual environment', experimental: false}
|
|
- {os: macos-14, label: torch, marker: '\[TAURI:STEP\] Installing PyTorch', experimental: false}
|
|
- {os: macos-14, label: unsloth, marker: '\[TAURI:STEP\] Installing Unsloth', experimental: false}
|
|
- {os: macos-14, label: setup, marker: '\[TAURI:STEP\] Running Unsloth setup', experimental: false}
|
|
# Other dependency-pass steps around the named one.
|
|
- {os: macos-14, label: pip-bootstrap, marker: 'pip bootstrap', experimental: false}
|
|
# No base-packages cell: install.sh --local sets skip_base, so
|
|
# install_python_stack returns before any "base packages" label is
|
|
# printed and the kill can never land. It ran to completion instead,
|
|
# proving nothing.
|
|
- {os: macos-14, label: unsloth-extras, marker: 'unsloth extras', experimental: true}
|
|
- {os: macos-14, label: data-designer, marker: 'data designer deps', experimental: true}
|
|
# Linux: same teardown path, different package manager and process semantics.
|
|
- {os: ubuntu-latest, label: studio-deps, marker: 'studio deps', experimental: false}
|
|
- {os: ubuntu-latest, label: torch, marker: '\[TAURI:STEP\] Installing PyTorch', experimental: false}
|
|
|
|
steps:
|
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- name: Linux system deps
|
|
if: runner.os == 'Linux'
|
|
run: |
|
|
sudo apt-get update -qq
|
|
sudo apt-get install -y -qq --no-install-recommends cmake git build-essential libcurl4-openssl-dev
|
|
|
|
- name: Install, interrupted at "${{ matrix.label }}"
|
|
env:
|
|
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
KILL_AT_SECONDS: '1500'
|
|
run: |
|
|
# --local is load-bearing, not a convenience. Without it install.sh:3996
|
|
# resolves `unsloth>=2026.7.5` from PyPI, so the venv gets the PUBLISHED CLI
|
|
# and none of the branch's unsloth_cli changes are present. Every probe of
|
|
# `studio verify-install` / `studio desktop-runtime-check` then reports
|
|
# "absent" no matter what the branch does, which makes the whole lane
|
|
# incapable of observing the fix it exists to test. --local overlays the
|
|
# checkout editable (install.sh:3990) before `studio setup` runs the dep
|
|
# pass, so a kill at "studio deps" leaves the branch's CLI installed.
|
|
bash .github/scripts/interrupt-install.sh \
|
|
'${{ matrix.marker }}' logs/install.log -- --tauri --local
|
|
|
|
- name: The kill must have landed where it was aimed
|
|
run: |
|
|
. logs/interrupt.env
|
|
echo "reason=$interrupt_reason killed=$interrupt_killed exit=$installer_exit"
|
|
if [ "$interrupt_reason" != "marker-hit" ]; then
|
|
echo "::error::installer never reached '${{ matrix.marker }}' (reason=$interrupt_reason)."
|
|
echo "::error::This leg proves nothing. Without this check it passes via the"
|
|
echo "::error::NO_CLI 'safe' path, which is how a --tauri/UNSLOTH_STUDIO_HOME"
|
|
echo "::error::conflict once made all 11 POSIX legs vacuously green."
|
|
tail -30 logs/install.log || true
|
|
exit 1
|
|
fi
|
|
|
|
- name: What state is the install in?
|
|
id: probe
|
|
run: |
|
|
# --tauri refuses a custom UNSLOTH_STUDIO_HOME, so it installs here.
|
|
BIN="$HOME/.unsloth/studio/unsloth_studio/bin/unsloth"
|
|
[ -x "$BIN" ] || BIN="$HOME/.unsloth/studio/bin/unsloth"
|
|
if [ ! -x "$BIN" ]; then
|
|
# No CLI at all is a SAFE outcome: preflight reports NotInstalled and the
|
|
# app offers a normal install. Nothing to assert beyond that.
|
|
echo "verdict=NO_CLI" >> "$GITHUB_OUTPUT"
|
|
echo "[probe] no unsloth CLI installed -> preflight reports NotInstalled (safe)"
|
|
exit 0
|
|
fi
|
|
rc=0
|
|
python3 .github/scripts/interrupted_install_probe.py "$BIN" --out probe || rc=$?
|
|
v="$(python3 -c "import json;print(json.load(open('probe/verdict.json'))['verdict'])")"
|
|
echo "verdict=$v" >> "$GITHUB_OUTPUT"
|
|
exit "$rc"
|
|
|
|
- name: A re-run must repair, not short-circuit
|
|
# NO_CLI included. A kill at venv or torch lands before "Installing Unsloth"
|
|
# (install.sh:2125 / :3667 / :3961), so those legs always take NO_CLI, and
|
|
# skipping the re-run left three non-experimental legs asserting nothing but
|
|
# that a marker appeared. The bug's second half is that `install.sh` sees a
|
|
# "current" version and no-ops over a broken venv.
|
|
if: always() && steps.probe.outputs.verdict != 'HEALTHY'
|
|
run: |
|
|
set -o pipefail
|
|
rc=0
|
|
bash install.sh --tauri --local < /dev/null 2>&1 | tee logs/repair.log || rc=$?
|
|
echo "repair exit: $rc"
|
|
BIN="$HOME/.unsloth/studio/unsloth_studio/bin/unsloth"
|
|
[ -x "$BIN" ] || BIN="$HOME/.unsloth/studio/bin/unsloth"
|
|
# The probe exits without writing verdict.json when the bin is missing, so
|
|
# check here or the json.load below crashes instead of reporting.
|
|
if [ ! -x "$BIN" ]; then
|
|
echo "::error::after a full re-run there is still no unsloth CLI at $BIN"
|
|
tail -30 logs/repair.log || true
|
|
exit 1
|
|
fi
|
|
python3 .github/scripts/interrupted_install_probe.py "$BIN" --out probe-after || true
|
|
v="$(python3 -c "import json;print(json.load(open('probe-after/verdict.json'))['verdict'])")"
|
|
# A booting backend IS the repair, whatever the log narrated. Judging by
|
|
# log text instead failed a leg whose venv was fine: the only match was
|
|
# the frontend build printing "up to date".
|
|
if [ "$v" = "HEALTHY" ]; then
|
|
echo "re-run repaired the install (verdict=HEALTHY)"
|
|
exit 0
|
|
fi
|
|
echo "::error::after a full re-run the backend still does not boot (verdict=$v)"
|
|
if grep -qiE "(venv|dependenc|python stack)[^|]*(up to date|already current)" logs/repair.log; then
|
|
echo "::error::and the re-run treated the venv as current instead of repairing it"
|
|
fi
|
|
exit 1
|
|
|
|
- name: Upload logs
|
|
if: always()
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
|
with:
|
|
name: interrupted-${{ matrix.os }}-${{ matrix.label }}
|
|
path: |
|
|
logs/
|
|
probe/
|
|
probe-after/
|
|
retention-days: 7
|
|
if-no-files-found: warn
|
|
|
|
# ── Windows: no process groups, so the kill path differs ──────────────────
|
|
interrupt-windows:
|
|
name: windows kill@${{ matrix.label }}
|
|
env:
|
|
# These legs run install.ps1 WITHOUT --tauri (install.ps1:189-215 rejects a custom
|
|
# root under --tauri exactly like install.sh:100-147), so the workspace-scoped
|
|
# root is usable here.
|
|
UNSLOTH_STUDIO_HOME: ${{ github.workspace }}/.studio-home
|
|
runs-on: windows-latest
|
|
timeout-minutes: 60
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
# install.ps1 parses `--no-torch` (install.ps1:121); `-SkipTorch` matches no
|
|
# case in that switch and is silently dropped. The torch leg must NOT skip
|
|
# torch, or its marker never appears.
|
|
- {label: studio-deps, marker: 'studio deps', installArgs: '--no-torch --local'}
|
|
- {label: torch, marker: 'Installing PyTorch', installArgs: '--local'}
|
|
|
|
steps:
|
|
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- name: Install, interrupted at "${{ matrix.label }}"
|
|
shell: pwsh
|
|
run: |
|
|
pwsh -NoProfile -File .github/scripts/interrupt-install.ps1 `
|
|
-Marker '${{ matrix.marker }}' -LogPath logs/install.log `
|
|
-InstallArgs '${{ matrix.installArgs }}' -KillAtSeconds 1500
|
|
|
|
- name: The kill must have landed where it was aimed
|
|
shell: pwsh
|
|
run: |
|
|
$vals = @{}
|
|
foreach ($line in (Get-Content logs/interrupt.env)) {
|
|
$kv = $line -split '=', 2
|
|
if ($kv.Count -eq 2) { $vals[$kv[0]] = $kv[1] }
|
|
}
|
|
Write-Host "reason=$($vals['interrupt_reason']) killed=$($vals['interrupt_killed']) exit=$($vals['installer_exit'])"
|
|
if ($vals['interrupt_reason'] -ne 'marker-hit') {
|
|
Write-Host "::error::installer never reached '${{ matrix.marker }}' (reason=$($vals['interrupt_reason']))."
|
|
Write-Host '::error::This leg proves nothing: without this check it passes via the'
|
|
Write-Host '::error::probe NO_CLI safe path, exactly as the POSIX legs once did.'
|
|
Get-Content logs/install.log -Tail 30 -ErrorAction SilentlyContinue
|
|
exit 1
|
|
}
|
|
|
|
- name: What state is the install in?
|
|
id: probe
|
|
shell: pwsh
|
|
run: |
|
|
$bin = Join-Path $env:UNSLOTH_STUDIO_HOME 'unsloth_studio\Scripts\unsloth.exe'
|
|
if (-not (Test-Path $bin)) {
|
|
"verdict=NO_CLI" | Out-File -FilePath $env:GITHUB_OUTPUT -Append -Encoding utf8
|
|
Write-Host '[probe] no unsloth CLI -> preflight reports NotInstalled (safe)'
|
|
exit 0
|
|
}
|
|
# The SAME probe the other platforms run. This step used to be a bespoke
|
|
# inline version that only checked `-h` and `desktop-capabilities`, so it
|
|
# could not observe studio_install_ok / verify-install /
|
|
# desktop-runtime-check -- it would have failed the very PRs that add them,
|
|
# no matter how well they worked.
|
|
python .github/scripts/interrupted_install_probe.py $bin --out probe
|
|
$rc = $LASTEXITCODE
|
|
$v = (Get-Content probe/verdict.json -Raw | ConvertFrom-Json).verdict
|
|
"verdict=$v" | Out-File -FilePath $env:GITHUB_OUTPUT -Append -Encoding utf8
|
|
exit $rc
|
|
|
|
- name: A re-run must repair, not short-circuit
|
|
# Same assertion the POSIX legs make, NO_CLI included: a leg that left no CLI
|
|
# otherwise asserts nothing, and without this a Windows leg proves only that
|
|
# the break was DETECTED, never that install.ps1's version fast path does not
|
|
# short-circuit over it, which is the half of the bug that strands the user.
|
|
if: always() && steps.probe.outputs.verdict != 'HEALTHY'
|
|
shell: pwsh
|
|
run: |
|
|
pwsh -NoProfile -NonInteractive -File install.ps1 ${{ matrix.installArgs }} *>&1 |
|
|
Tee-Object -FilePath logs/repair.log
|
|
$bin = Join-Path $env:UNSLOTH_STUDIO_HOME 'unsloth_studio\Scripts\unsloth.exe'
|
|
if (-not (Test-Path $bin)) {
|
|
Write-Host "::error::after a full re-run there is still no unsloth CLI at $bin"
|
|
Get-Content logs/repair.log -Tail 30 -ErrorAction SilentlyContinue
|
|
exit 1
|
|
}
|
|
python .github/scripts/interrupted_install_probe.py $bin --out probe-after
|
|
$v = (Get-Content probe-after/verdict.json -Raw | ConvertFrom-Json).verdict
|
|
# A booting backend IS the repair, whatever the log narrated. Judging by
|
|
# log text instead failed a POSIX leg whose venv was fine: the only match
|
|
# was the frontend build printing "up to date".
|
|
if ($v -eq 'HEALTHY') {
|
|
Write-Host 're-run repaired the install (verdict=HEALTHY)'
|
|
exit 0
|
|
}
|
|
Write-Host "::error::after a full re-run the backend still does not boot (verdict=$v)"
|
|
$log = Get-Content logs/repair.log -Raw -ErrorAction SilentlyContinue
|
|
if ($log -match '(?i)(venv|dependenc|python stack)[^|]*(up to date|already current)') {
|
|
Write-Host '::error::and the re-run treated the venv as current instead of repairing it'
|
|
}
|
|
exit 1
|
|
|
|
- name: Upload logs
|
|
if: always()
|
|
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
|
with:
|
|
name: interrupted-windows-${{ matrix.label }}
|
|
path: |
|
|
logs/
|
|
probe/
|
|
probe-after/
|
|
retention-days: 7
|
|
if-no-files-found: warn
|