ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Studio: Stop every running Unsloth server, not just the last one recorded (#7577)

Browse source 
* Stop every running Unsloth server, and refuse to start a second on a taken port

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Check the fallback range, guard PID reuse, and keep writing studio.pid

* Signal each server once when its PID is recorded in more than one file

* Confirm a recorded PID is a Studio server before signalling it

* Pin PID records to process start time and check every listener on a port

* Keep every recorded start time per PID and accept in-process Studio servers

* Match the blocking listener address and stop trusting unverifiable PID records

* Never delete a PID record that cannot be verified

* Detect our own server from our own records instead of a psutil listener scan

* Match a pre-upgrade studio.pid to the blocked port before falling back

* Never signal PID 0 or 1, and verify a per-port record before trusting it

* Stop unverifiable records instead of skipping them, and record every bind address

* Drop the command-line guess, fix Windows liveness, and free the PID record last

* Studio: harden the per-port PID records against the cases that lose a server

Follow-up on the per-port PID files. Each item below is a case where the new
code either lost a server the old code could still stop, or stopped something
that was not ours. All were reproduced against real Studio servers.

studio/backend/run.py

- Write the per-port record and the legacy studio.pid independently. They shared
  one try, so a studio root that could not take a new directory entry left the
  server recorded nowhere at all and unstoppable from the CLI; the old code
  still recorded it in studio.pid, which is an overwrite of an existing path and
  can still succeed. _remove_pid_file now also checks studio.pid when the
  per-port write failed.
- Write the record through a temp file and os.replace. `stop` reads these
  concurrently and treats a truncated read as a corrupt record.
- A failed Windows tasklist probe now means "alive", matching the CLI. Treating
  it as dead pruned a live server's record and let the next launch fall back
  past it, which is the orphan this work exists to fix.
- Guard the unlink in _own_studio_on_port. Pruning is a courtesy and must not
  abort startup.
- Extract _resolve_port so the requested-port abort is reachable from a test.
  Deleting that abort previously left the whole suite green.
- Keep the plain fallback for api-only callers. The desktop app hardcodes 8888
  and documents its reliance on the 8888-8908 range, and it reports a non-zero
  backend exit to the user as "Server stopped unexpectedly". It reads the bound
  port back from TAURI_PORT, as `studio run` does from app.state.server_port, so
  a fallback there is harmless and both servers are still recorded and
  stoppable. The interactive path prints the requested port, so it still aborts.
- isdigit() is not enough to gate int(): a superscript two passes it and the
  ValueError escaped into every caller of _read_pid_record.

unsloth_cli/commands/studio.py

- An untimed record no longer cancels a timed one for the same PID. Every
  current server writes both a timed per-port record and an untimed studio.pid,
  so the start-time check was inert exactly where it mattered, and after a crash
  plus a PID reuse `stop` sent SIGTERM to whatever unrelated process had
  inherited the PID.
- Distinguish an unreadable record from an invalid one. A root-owned record, or
  one caught mid-write, still belongs to a live server, and deleting it stranded
  that server.
- Route every PID-file removal through _unlink_quietly. One undeletable record
  raised PermissionError and left the remaining live servers running.
- Same isdigit()/int() guard as the backend.

Tests

- The requested-port abort, the recorded bind address, and the api-only
  fallback are now covered; all three previously survived deletion.
- tests/studio/test_studio_pid_file_contract.py pins run.py's filename scheme to
  the CLI's glob and keeps studio.pid parseable by an older CLI. It lives under
  tests/studio because unsloth_cli/tests is not run by any workflow.
- test_cli_studio_stop_windows.py now checks _signal_stop as well as stop. The
  kill moved into _signal_stop, so the os.kill(pid, 0) guard passed vacuously.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: let a caller that follows the port keep the fallback, and never take studio.pid from a live server

Two problems with keying the own-server abort on api_only.

`unsloth studio run` is not the bare-banner path: it stores `app = run_server(...)`
and reads `app.state.server_port` back, then uses it for the health wait, the
model load and the printed base URL. Gating on api_only aborted it, so starting
a second model while the first was up stopped working, where before it landed on
the next port and printed the right URL. Replace the proxy with an explicit
abort_if_own_studio, defaulting to the old api_only behaviour so the exec'd
`run.py` path is unchanged, and have `studio run` opt out.

The api_only exemption also reopened the orphan from the other side.
_write_pid_file overwrote studio.pid unconditionally, and a pre-upgrade server
is recorded there and nowhere else, so an exempt launch falling back past one
erased its only record. Take the file over only when it is free, already ours,
or held by a dead PID.

Also resync _pid_is_studio_backend with the CLI copy: an untimed record next to
a timed one carried no information but cancelled the start-time check, which is
what let a reused PID be treated as ours.

Tests: 51 backend, 26 CLI, 9 under tests/studio. Real Studio servers still abort
the bare same-port relaunch, still fall back past a foreign listener, and one
`unsloth studio stop` still stops every server in all five scenarios.

* Studio: hand over the legacy PID pointer, and fail stop on unreadable records

Two follow-ups from review of the previous commit.

Only one backend owns studio.pid at a time. When that server exited it
deleted the file, so an older CLI, which reads nothing else, could no
longer stop a sibling that was still serving. _remove_pid_file now hands
the pointer to a live sibling instead of dropping it.

_pid_file_entries skipped records it could not read, for instance one
written by a server started under sudo. When that was the only record,
stop printed "No running Unsloth server found" and exited 0 while the
server kept serving. Unreadable records are now reported and make stop
exit 1, so a partial stop is never mistaken for a complete one.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <unslothai@gmail.com>
This commit is contained in:
Nilay 2026-07-29 14:26:13 +05:30 • committed by GitHub
commit ceef4123e6
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 1699 additions and 59 deletions
Download patch file Download diff file Expand all files Collapse all files

230
unsloth_cli/commands/studio.py
View file

@ -19,7 +19,7 @@ import urllib.error
import urllib.request
from datetime import datetime, timezone
from pathlib import Path
from typing import List, Literal, Optional
from typing import List, Literal, Optional, Sequence
import typer
from unsloth_cli import _studio_deps
@ -2265,6 +2265,9 @@ def run(
# Headless serving prints its own URL/API-key banner; the Tauri-only
# TAURI_PORT line would corrupt that machine-parseable output.
emit_tauri_port = False,
# We read the bound port back below, so a fallback past another Studio is
# safe here and keeps side-by-side model runs working.
abort_if_own_studio = False,
)
# Forward the frontend validated before the gate (in-venv path).
if resolved_frontend is not None:
@ -2424,6 +2427,7 @@ def run(
# ── unsloth studio stop ───────────────────────────────────────────────
_PID_FILE = STUDIO_HOME / "studio.pid"
PID_FILE_GLOB = "studio-*.pid"
def _pid_alive(pid: int) -> bool:
@ -2453,58 +2457,210 @@ def _pid_alive(pid: int) -> bool:
return True
@studio_app.command()
def stop():
"""Stop a running Unsloth Studio server.
def _parse_pid_record(text: str) -> "tuple[int, float | None] | None":
"""Parse ``pid`` / optional ``create_time`` from PID file contents."""
lines = text.splitlines()
if not lines or not lines[0].strip().isdigit():
return None
try:
# isdigit() is not enough: "²".isdigit() is True but int() rejects it.
pid = int(lines[0].strip())
except ValueError:
return None
# kill(0) signals our whole process group; kill(1) is init. Never either.
if pid < 2:
return None
created = None
if len(lines) > 1:
try:
created = float(lines[1].strip())
except ValueError:
created = None
return pid, created
Reads the PID from ~/.unsloth/studio/studio.pid and sends SIGTERM
(or TerminateProcess on Windows) to shut it down gracefully.
def _read_pid_record(path: Path) -> "tuple[int, float | None] | None":
"""Parse ``pid`` / optional ``create_time`` from a PID file."""
try:
text = path.read_text(encoding = "utf-8")
except (OSError, UnicodeDecodeError):
return None
return _parse_pid_record(text)
def _unlink_quietly(path: Path) -> None:
"""Drop a record without letting one bad file end the loop.
An undeletable record must not stop us reaching the other servers -- that is
the orphan this command exists to prevent.
"""
try:
path.unlink(missing_ok = True)
except OSError as e:
typer.echo(f"Could not remove PID file {path.name}: {e}", err = True)
def _report_unreadable(paths: "list[Path]") -> None:
"""Say which servers we could not reach, since `stop` is about to exit 1."""
names = ", ".join(sorted(p.name for p in paths))
typer.echo(
f"Could not read {len(paths)} PID file(s): {names}. A server recorded "
f"there may still be running; re-run with permission to read "
f"{STUDIO_HOME} to stop it.",
err = True,
)
def _pid_file_entries(
unreadable: "list[Path] | None" = None,
) -> "list[tuple[int, list[float | None], list[Path]]]":
"""(pid, create_times, files) per recorded server, including the legacy studio.pid.
Paths that could not be read are appended to `unreadable` when given, so the
caller can tell "nothing is running" apart from "something is running and we
could not see it".
Grouped by PID: a server writes both its per-port file and studio.pid, and
signalling twice would hit the SIG_DFL the first SIGTERM installs, hard-killing
it mid-shutdown. Every recorded time is kept -- a stale file and a live server
can share a PID, and the stale one must not veto the live one.
"""
by_pid: "dict[int, tuple[list[float | None], list[Path]]]" = {}
try:
paths = sorted(STUDIO_HOME.glob(PID_FILE_GLOB)) + [_PID_FILE]
except OSError:
paths = [_PID_FILE]
seen = set()
for path in paths:
if path in seen or not path.is_file():
continue
seen.add(path)
try:
text = path.read_text(encoding = "utf-8")
except (OSError, UnicodeDecodeError) as e:
# Unreadable is not the same as invalid. A root-owned record, or one
# caught mid-write, still belongs to a live server, and deleting it
# strands that server -- the bug this command exists to fix.
typer.echo(f"Cannot read PID file {path.name}: {e}", err = True)
if unreadable is not None:
unreadable.append(path)
continue
record = _parse_pid_record(text)
if record is None:
typer.echo(f"Ignoring invalid PID file {path.name}")
_unlink_quietly(path)
continue
pid, created = record
created_times, files = by_pid.setdefault(pid, ([], []))
created_times.append(created)
files.append(path)
return [(pid, times, files) for pid, (times, files) in by_pid.items()]
def _pid_is_studio_server(pid: int, created_times: "Sequence[float | None]" = ()) -> bool:
"""False only when a recorded start time proves this PID is a different process.
Any recorded time matching is enough -- a stale record must not veto a live
server that reused the PID. Records with no time at all (a legacy studio.pid,
or a server started without psutil) cannot be checked, so they are trusted:
the old `stop` signalled with no checks at all, and skipping a live server is
the orphan bug this exists to fix.
An untimed record sitting *alongside* a timed one carries no information, so
it must not cancel the timed one either. Every current server writes both a
timed per-port record and an untimed studio.pid, so letting the untimed half
win made this check inert exactly where it matters and let `stop` SIGTERM an
unrelated process that had inherited the PID.
"""
known = [c for c in created_times if c is not None]
if not known:
return True
try:
import psutil
actual = psutil.Process(pid).create_time()
except Exception:
return True
return any(abs(actual - c) < 1.0 for c in known)
def _signal_stop(pid: int) -> "str | None":
"""SIGTERM (or taskkill) the pid. Returns an error string, or None on success."""
import signal as _signal
if not _PID_FILE.is_file():
typer.echo("No running Unsloth server found (no PID file).")
raise typer.Exit(0)
pid_text = _PID_FILE.read_text(encoding = "utf-8").strip()
if not pid_text.isdigit():
typer.echo(f"Invalid PID file contents: {pid_text}")
_PID_FILE.unlink(missing_ok = True)
raise typer.Exit(1)
pid = int(pid_text)
# Check if still alive (os.kill(pid, 0) is invalid on Windows -- see _pid_alive).
if not _pid_alive(pid):
typer.echo(f"Unsloth server (PID {pid}) is not running. Cleaning up stale PID file.")
_PID_FILE.unlink(missing_ok = True)
raise typer.Exit(0)
# Send SIGTERM (graceful shutdown) or TerminateProcess on Windows
if pid < 2:
return f"refusing to signal PID {pid}"
try:
if sys.platform == "win32":
# /T also stops llama-server children, which otherwise keep GPU and port.
subprocess.run(["taskkill", "/PID", str(pid), "/T", "/F"], check = True)
else:
os.kill(pid, _signal.SIGTERM)
typer.echo(f"Sent shutdown signal to Unsloth server (PID {pid}).")
except ProcessLookupError:
typer.echo(f"Unsloth server (PID {pid}) already exited.")
_PID_FILE.unlink(missing_ok = True)
raise typer.Exit(0)
return None
except Exception as e:
typer.echo(f"Failed to stop Unsloth server (PID {pid}): {e}", err = True)
raise typer.Exit(1)
return str(e)
return None
# Wait briefly for the process to exit and clean up.
@studio_app.command()
def stop():
"""Stop every running Unsloth Studio server for this STUDIO_HOME.
The port fallback can leave more than one running, so stop them all.
"""
unreadable: "list[Path]" = []
entries = _pid_file_entries(unreadable)
if not entries:
if unreadable:
# Reporting success here would be a lie: the records we could not
# read are kept, and the servers behind them are still serving.
_report_unreadable(unreadable)
raise typer.Exit(1)
typer.echo("No running Unsloth server found (no PID file).")
raise typer.Exit(0)
signalled, failed = [], []
for pid, created_times, paths in entries:
if not _pid_alive(pid) or not _pid_is_studio_server(pid, created_times):
for path in paths:
_unlink_quietly(path)
continue
error = _signal_stop(pid)
if error is not None:
failed.append((pid, error))
typer.echo(f"Failed to stop Unsloth server (PID {pid}): {error}", err = True)
continue
typer.echo(f"Sent shutdown signal to Unsloth server (PID {pid}).")
signalled.append((pid, paths))
if not signalled and not failed:
if unreadable:
_report_unreadable(unreadable)
raise typer.Exit(1)
typer.echo("No running Unsloth server found (cleaned up stale PID files).")
raise typer.Exit(0)
pending = list(signalled)
for _ in range(10):
if not pending:
break
time.sleep(0.5)
if not _pid_alive(pid):
_PID_FILE.unlink(missing_ok = True)
typer.echo("Unsloth server stopped.")
raise typer.Exit(0)
for entry in list(pending):
pid, paths = entry
if not _pid_alive(pid):
for path in paths:
_unlink_quietly(path)
pending.remove(entry)
typer.echo("Unsloth server is shutting down (may take a few seconds).")
stopped = len(signalled) - len(pending)
if stopped:
typer.echo(f"Unsloth server{'s' if stopped > 1 else ''} stopped ({stopped}).")
for pid, _paths in pending:
typer.echo(f"Unsloth server (PID {pid}) is shutting down (may take a few seconds).")
if unreadable:
_report_unreadable(unreadable)
if failed or unreadable:
raise typer.Exit(1)
# ── unsloth studio setup / update ─────────────────────────────────────

530
unsloth_cli/tests/test_studio_stop.py Normal file
View file

@ -0,0 +1,530 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
"""`unsloth studio stop` must stop every server it started.
With one PID file the second launch overwrote the first entry, so stop killed
the newer server, claimed success, and left the older one serving.
"""
from __future__ import annotations
import sys
from pathlib import Path
from types import SimpleNamespace
import pytest
from typer.testing import CliRunner
_REPO_ROOT = Path(__file__).resolve().parents[2]
if str(_REPO_ROOT) not in sys.path:
sys.path.insert(0, str(_REPO_ROOT))
def _studio():
from unsloth_cli.commands import studio as _studio_mod
return _studio_mod
# Captured before _install stubs it, for the tests that exercise it.
_REAL_IS_STUDIO_SERVER = _studio()._pid_is_studio_server
def _install(
monkeypatch,
tmp_path,
*,
alive,
killed = None,
):
"""Point the CLI at tmp_path and fake process liveness."""
studio_mod = _studio()
monkeypatch.setattr(studio_mod, "STUDIO_HOME", tmp_path)
monkeypatch.setattr(studio_mod, "_PID_FILE", tmp_path / "studio.pid")
monkeypatch.setattr(studio_mod.time, "sleep", lambda _s: None)
live = set(alive)
killed = killed if killed is not None else []
monkeypatch.setattr(studio_mod, "_pid_alive", lambda pid: pid in live)
monkeypatch.setattr(studio_mod, "_pid_is_studio_server", lambda pid, created_times = (): True)
def fake_kill(pid, _sig):
killed.append(pid)
live.discard(pid)
monkeypatch.setattr(studio_mod.os, "kill", fake_kill)
monkeypatch.setattr(sys, "platform", "linux")
return studio_mod, live, killed
def _write_pid(tmp_path, name, pid):
(tmp_path / name).write_text(str(pid), encoding = "utf-8")
def _run_stop(studio_mod):
import typer as _typer
app = _typer.Typer()
app.add_typer(studio_mod.studio_app, name = "studio")
return CliRunner().invoke(app, ["studio", "stop"])
def test_stop_kills_every_recorded_server(monkeypatch, tmp_path):
studio_mod, _live, killed = _install(monkeypatch, tmp_path, alive = {8550, 8600})
_write_pid(tmp_path, "studio-8901-8550.pid", 8550)
_write_pid(tmp_path, "studio-8902-8600.pid", 8600)
result = _run_stop(studio_mod)
assert result.exit_code == 0, result.output
assert sorted(killed) == [8550, 8600]
assert not list(tmp_path.glob("studio-*.pid"))
def test_stop_does_not_leave_the_older_instance_running(monkeypatch, tmp_path):
# The reported symptom: stop claimed success while instance A kept serving.
studio_mod, live, _killed = _install(monkeypatch, tmp_path, alive = {8550, 8600})
_write_pid(tmp_path, "studio-8901-8550.pid", 8550)
_write_pid(tmp_path, "studio-8902-8600.pid", 8600)
result = _run_stop(studio_mod)
assert result.exit_code == 0, result.output
assert live == set()
def test_stop_signals_each_server_once(monkeypatch, tmp_path):
# A server writes its per-port file AND studio.pid. It stays alive while it
# shuts down gracefully, so a second SIGTERM would hit the SIG_DFL the first
# one installs and hard-kill it mid-cleanup.
studio_mod = _studio()
monkeypatch.setattr(studio_mod, "STUDIO_HOME", tmp_path)
monkeypatch.setattr(studio_mod, "_PID_FILE", tmp_path / "studio.pid")
monkeypatch.setattr(studio_mod.time, "sleep", lambda _s: None)
monkeypatch.setattr(studio_mod, "_pid_alive", lambda pid: True)
monkeypatch.setattr(studio_mod, "_pid_is_studio_server", lambda pid, created_times = (): True)
killed = []
monkeypatch.setattr(studio_mod.os, "kill", lambda pid, _sig: killed.append(pid))
monkeypatch.setattr(sys, "platform", "linux")
_write_pid(tmp_path, "studio-8901-8550.pid", 8550)
_write_pid(tmp_path, "studio.pid", 8550)
result = _run_stop(studio_mod)
assert result.exit_code == 0, result.output
assert killed == [8550]
assert result.output.lower().count("sent shutdown signal") == 1
def test_stop_removes_every_stale_file_for_one_pid(monkeypatch, tmp_path):
studio_mod, _live, killed = _install(monkeypatch, tmp_path, alive = set())
_write_pid(tmp_path, "studio-8901-8550.pid", 8550)
_write_pid(tmp_path, "studio.pid", 8550)
result = _run_stop(studio_mod)
assert result.exit_code == 0, result.output
assert killed == []
assert not list(tmp_path.glob("*.pid"))
def test_stop_does_not_signal_a_reused_pid(monkeypatch, tmp_path):
# Crash leaves a per-port file behind, the OS hands that PID to something
# else: stop must drop the record, not SIGTERM an unrelated process.
studio_mod, _live, killed = _install(monkeypatch, tmp_path, alive = {8550})
monkeypatch.setattr(studio_mod, "_pid_is_studio_server", lambda pid, created_times = (): False)
_write_pid(tmp_path, "studio-8901-8550.pid", 8550)
result = _run_stop(studio_mod)
assert result.exit_code == 0, result.output
assert killed == []
assert not (tmp_path / "studio-8901-8550.pid").exists()
def test_stop_signals_a_live_server_whose_pid_has_a_stale_record(monkeypatch, tmp_path):
# Crash leaves studio-8888-8550.pid, the OS reuses 8550 for a new server on
# another port. The stale timestamp must not veto the live one.
studio_mod, _live, killed = _install(monkeypatch, tmp_path, alive = {8550})
monkeypatch.setattr(studio_mod, "_pid_is_studio_server", _REAL_IS_STUDIO_SERVER)
class _FakeProcess:
def __init__(self, pid):
self.pid = pid
def create_time(self):
return 999.0
monkeypatch.setitem(sys.modules, "psutil", SimpleNamespace(Process = _FakeProcess))
(tmp_path / "studio-8888-8550.pid").write_text("8550\n111.5", encoding = "utf-8")
(tmp_path / "studio-9000-8550.pid").write_text("8550\n999.0", encoding = "utf-8")
result = _run_stop(studio_mod)
assert result.exit_code == 0, result.output
assert killed == [8550]
assert not list(tmp_path.glob("studio-*.pid"))
def test_a_bare_run_py_command_line_is_not_rejected(monkeypatch):
# `cd studio/backend && python run.py --port 8901` has no "studio" or "unsloth"
# in argv. Guessing from the command line deleted its record without stopping it.
studio_mod = _studio()
class _FakeProcess:
def __init__(self, pid):
self.pid = pid
def cmdline(self):
return ["python", "run.py", "--port", "8901"]
def create_time(self):
return 111.5
monkeypatch.setitem(sys.modules, "psutil", SimpleNamespace(Process = _FakeProcess))
assert studio_mod._pid_is_studio_server(8550) is True
def test_an_untimed_record_is_trusted(monkeypatch):
# A legacy `python run.py --port 8901` has no telltale argv, and the in-venv
# path runs in-process. Guessing from the command line rejected real servers.
studio_mod = _studio()
assert studio_mod._pid_is_studio_server(8550) is True
assert studio_mod._pid_is_studio_server(8550, [None]) is True
def test_an_unverifiable_record_is_still_stopped(monkeypatch):
# psutil is not a base CLI dependency, so the CLI meets timestamped records it
# cannot check. The old `stop` signalled with no checks at all -- skipping one
# would leave a live server running, the orphan bug this exists to fix.
studio_mod = _studio()
monkeypatch.setitem(sys.modules, "psutil", None)
assert studio_mod._pid_is_studio_server(8550, [111.5]) is True
assert studio_mod._pid_is_studio_server(8550, [None]) is True
def test_stop_signals_a_timestamped_record_without_psutil(monkeypatch, tmp_path):
# Multiple servers on different ports: only the newest is also in studio.pid,
# so the earlier ones are timestamp-only and must still be stopped.
studio_mod, _live, killed = _install(monkeypatch, tmp_path, alive = {8550})
monkeypatch.setattr(studio_mod, "_pid_is_studio_server", _REAL_IS_STUDIO_SERVER)
monkeypatch.setitem(sys.modules, "psutil", None)
(tmp_path / "studio-8901-8550.pid").write_text("8550\n111.5", encoding = "utf-8")
result = _run_stop(studio_mod)
assert result.exit_code == 0, result.output
assert killed == [8550]
assert not (tmp_path / "studio-8901-8550.pid").exists()
def test_the_untimed_legacy_record_does_not_cancel_a_timed_one(monkeypatch):
# Every current server writes BOTH a timed per-port record and an untimed
# studio.pid, so letting the untimed half win made this check inert exactly
# where it matters: after a crash and a PID reuse, `stop` SIGTERMed whatever
# unrelated process had inherited the PID. An untimed record carries no
# information, so it must not overrule a start time that says "not ours".
studio_mod = _studio()
class _FakeProcess:
def __init__(self, pid):
self.pid = pid
def create_time(self):
return 999.0
monkeypatch.setitem(sys.modules, "psutil", SimpleNamespace(Process = _FakeProcess))
assert studio_mod._pid_is_studio_server(8550, [111.5, None]) is False
assert studio_mod._pid_is_studio_server(8550, [111.5]) is False
# A matching time still wins over a stale sibling record.
assert studio_mod._pid_is_studio_server(8550, [111.5, 999.0]) is True
assert studio_mod._pid_is_studio_server(8550, [None, None]) is True
def test_stop_does_not_signal_a_reused_pid_recorded_in_both_files(monkeypatch, tmp_path):
# End to end for the case above: a crashed server left studio-8901-8550.pid
# and studio.pid, and 8550 now belongs to something else entirely.
studio_mod, _live, killed = _install(monkeypatch, tmp_path, alive = {8550})
monkeypatch.setattr(studio_mod, "_pid_is_studio_server", _REAL_IS_STUDIO_SERVER)
class _FakeProcess:
def __init__(self, pid):
self.pid = pid
def create_time(self):
return 999.0
monkeypatch.setitem(sys.modules, "psutil", SimpleNamespace(Process = _FakeProcess))
(tmp_path / "studio-8901-8550.pid").write_text("8550\n111.5\n127.0.0.1", encoding = "utf-8")
(tmp_path / "studio.pid").write_text("8550", encoding = "utf-8")
result = _run_stop(studio_mod)
assert result.exit_code == 0, result.output
assert killed == []
assert not list(tmp_path.glob("*.pid"))
def test_pid_identity_check_trusts_the_record_without_psutil(monkeypatch):
# No psutil: fall back to trusting the record rather than never stopping.
studio_mod = _studio()
monkeypatch.setitem(sys.modules, "psutil", None)
assert studio_mod._pid_is_studio_server(8550) is True
def test_pid_identity_check_uses_the_recorded_start_time(monkeypatch):
studio_mod = _studio()
class _FakeProcess:
def __init__(self, pid):
self.pid = pid
def create_time(self):
return 111.5
monkeypatch.setitem(sys.modules, "psutil", SimpleNamespace(Process = _FakeProcess))
assert studio_mod._pid_is_studio_server(8550, [111.5]) is True
assert studio_mod._pid_is_studio_server(8550, [999.0]) is False
def test_stop_drops_a_record_whose_start_time_no_longer_matches(monkeypatch, tmp_path):
# The PID was reused: same number, different process.
studio_mod, _live, killed = _install(monkeypatch, tmp_path, alive = {8550})
monkeypatch.setattr(studio_mod, "_pid_is_studio_server", _REAL_IS_STUDIO_SERVER)
class _FakeProcess:
def __init__(self, pid):
self.pid = pid
def create_time(self):
return 999.0
monkeypatch.setitem(sys.modules, "psutil", SimpleNamespace(Process = _FakeProcess))
(tmp_path / "studio-8901-8550.pid").write_text("8550\n111.5", encoding = "utf-8")
result = _run_stop(studio_mod)
assert result.exit_code == 0, result.output
assert killed == []
assert not (tmp_path / "studio-8901-8550.pid").exists()
# Dropped for the start-time mismatch, not because the record looked corrupt.
assert "invalid pid file" not in result.output.lower()
def test_stop_reads_the_legacy_single_pid_file(monkeypatch, tmp_path):
studio_mod, _live, killed = _install(monkeypatch, tmp_path, alive = {4242})
_write_pid(tmp_path, "studio.pid", 4242)
result = _run_stop(studio_mod)
assert result.exit_code == 0, result.output
assert killed == [4242]
assert not (tmp_path / "studio.pid").exists()
def test_stop_reports_nothing_running_without_pid_files(monkeypatch, tmp_path):
studio_mod, _live, _killed = _install(monkeypatch, tmp_path, alive = set())
result = _run_stop(studio_mod)
assert result.exit_code == 0, result.output
assert "no running unsloth server" in result.output.lower()
def test_stop_cleans_stale_pid_files_without_claiming_a_stop(monkeypatch, tmp_path):
studio_mod, _live, killed = _install(monkeypatch, tmp_path, alive = set())
_write_pid(tmp_path, "studio-8901-8550.pid", 8550)
result = _run_stop(studio_mod)
assert result.exit_code == 0, result.output
assert killed == []
assert not (tmp_path / "studio-8901-8550.pid").exists()
assert "stopped" not in result.output.lower()
def test_stop_does_not_claim_a_stop_while_a_server_is_still_alive(monkeypatch, tmp_path):
# SIGTERM delivered but it never exits: don't claim a stop, keep the file.
studio_mod = _studio()
monkeypatch.setattr(studio_mod, "STUDIO_HOME", tmp_path)
monkeypatch.setattr(studio_mod, "_PID_FILE", tmp_path / "studio.pid")
monkeypatch.setattr(studio_mod.time, "sleep", lambda _s: None)
monkeypatch.setattr(studio_mod, "_pid_alive", lambda pid: True)
monkeypatch.setattr(studio_mod, "_pid_is_studio_server", lambda pid, created_times = (): True)
monkeypatch.setattr(studio_mod.os, "kill", lambda pid, sig: None)
monkeypatch.setattr(sys, "platform", "linux")
_write_pid(tmp_path, "studio-8901-8550.pid", 8550)
result = _run_stop(studio_mod)
assert result.exit_code == 0, result.output
assert "shutting down" in result.output.lower()
assert "stopped" not in result.output.lower()
assert (tmp_path / "studio-8901-8550.pid").exists()
def test_stop_continues_after_one_server_fails_to_stop(monkeypatch, tmp_path):
studio_mod = _studio()
monkeypatch.setattr(studio_mod, "STUDIO_HOME", tmp_path)
monkeypatch.setattr(studio_mod, "_PID_FILE", tmp_path / "studio.pid")
monkeypatch.setattr(studio_mod.time, "sleep", lambda _s: None)
live = {8550, 8600}
monkeypatch.setattr(studio_mod, "_pid_alive", lambda pid: pid in live)
monkeypatch.setattr(studio_mod, "_pid_is_studio_server", lambda pid, created_times = (): True)
def fake_kill(pid, _sig):
if pid == 8550:
raise PermissionError("not permitted")
live.discard(pid)
monkeypatch.setattr(studio_mod.os, "kill", fake_kill)
monkeypatch.setattr(sys, "platform", "linux")
_write_pid(tmp_path, "studio-8901-8550.pid", 8550)
_write_pid(tmp_path, "studio-8902-8600.pid", 8600)
result = _run_stop(studio_mod)
combined = (result.output or "") + (getattr(result, "stderr", "") or "")
assert result.exit_code == 1, combined
assert 8600 not in live
assert "8550" in combined
def test_stop_never_signals_pid_zero_or_init(monkeypatch, tmp_path):
# os.kill(0, SIGTERM) hits our whole process group -- the shell and its jobs.
studio_mod, _live, killed = _install(monkeypatch, tmp_path, alive = {0, 1})
_write_pid(tmp_path, "studio-8901-0.pid", 0)
_write_pid(tmp_path, "studio-8902-1.pid", 1)
result = _run_stop(studio_mod)
assert result.exit_code == 0, result.output
assert killed == []
assert not list(tmp_path.glob("*.pid"))
def test_signal_stop_refuses_pid_zero_or_init(monkeypatch, tmp_path):
studio_mod, _live, killed = _install(monkeypatch, tmp_path, alive = {0, 1})
assert studio_mod._signal_stop(0) is not None
assert studio_mod._signal_stop(1) is not None
assert killed == []
def test_stop_discards_a_corrupt_pid_file(monkeypatch, tmp_path):
studio_mod, _live, _killed = _install(monkeypatch, tmp_path, alive = set())
(tmp_path / "studio-8901-8550.pid").write_text("not-a-pid", encoding = "utf-8")
result = _run_stop(studio_mod)
assert result.exit_code == 0, result.output
assert not (tmp_path / "studio-8901-8550.pid").exists()
def test_stop_keeps_a_record_it_cannot_read(monkeypatch, tmp_path):
# A root-owned record, or one caught mid-write, still belongs to a live
# server. Deleting it is `stop` manufacturing the orphan it exists to fix.
studio_mod, _live, killed = _install(monkeypatch, tmp_path, alive = {8550})
path = tmp_path / "studio-8901-8550.pid"
path.write_text("8550", encoding = "utf-8")
real_read_text = Path.read_text
def deny(self, *args, **kwargs):
if self == path:
raise PermissionError(13, "Permission denied")
return real_read_text(self, *args, **kwargs)
monkeypatch.setattr(Path, "read_text", deny)
result = _run_stop(studio_mod)
assert path.exists(), "an unreadable record must not be deleted"
assert "cannot read" in (result.output + (result.stderr or "")).lower()
def test_stop_does_not_claim_success_when_the_only_record_is_unreadable(monkeypatch, tmp_path):
# A server started under sudo leaves a record we cannot read. Printing "no
# running server" and exiting 0 tells the user the opposite of the truth.
studio_mod, _live, killed = _install(monkeypatch, tmp_path, alive = {8550})
path = tmp_path / "studio-8901-8550.pid"
path.write_text("8550", encoding = "utf-8")
real_read_text = Path.read_text
def deny(self, *args, **kwargs):
if self == path:
raise PermissionError(13, "Permission denied")
return real_read_text(self, *args, **kwargs)
monkeypatch.setattr(Path, "read_text", deny)
result = _run_stop(studio_mod)
assert result.exit_code == 1, "an unreachable server is not a successful stop"
output = result.output + (result.stderr or "")
assert "no running unsloth server" not in output.lower()
assert killed == []
def test_stop_reports_failure_when_one_record_is_unreadable_but_another_stops(
monkeypatch, tmp_path
):
# Stopping the servers we can see is still a partial result, and exiting 0
# would hide the one we could not.
studio_mod, _live, killed = _install(monkeypatch, tmp_path, alive = {8550, 8600})
_write_pid(tmp_path, "studio-8901-8550.pid", 8550)
hidden = tmp_path / "studio-8902-8600.pid"
hidden.write_text("8600", encoding = "utf-8")
real_read_text = Path.read_text
def deny(self, *args, **kwargs):
if self == hidden:
raise PermissionError(13, "Permission denied")
return real_read_text(self, *args, **kwargs)
monkeypatch.setattr(Path, "read_text", deny)
result = _run_stop(studio_mod)
assert killed == [8550], "the readable server must still be stopped"
assert result.exit_code == 1
assert hidden.exists()
def test_stop_reaches_every_server_when_one_record_cannot_be_removed(monkeypatch, tmp_path):
# One undeletable stale record must not end the loop before the live servers.
studio_mod, _live, killed = _install(monkeypatch, tmp_path, alive = {8600})
_write_pid(tmp_path, "studio-8901-8550.pid", 8550) # dead -> stop prunes it
_write_pid(tmp_path, "studio-8902-8600.pid", 8600) # live -> stop signals it
real_unlink = Path.unlink
def deny(self, *args, **kwargs):
if self.name == "studio-8901-8550.pid":
raise PermissionError(13, "Permission denied")
return real_unlink(self, *args, **kwargs)
monkeypatch.setattr(Path, "unlink", deny)
result = _run_stop(studio_mod)
assert killed == [8600], "the live server must still be signalled"
assert result.exit_code == 0, result.output
def test_a_record_whose_pid_is_not_ascii_digits_is_discarded(monkeypatch, tmp_path):
# A superscript two passes isdigit() but int() rejects it, so that gate alone
# let a ValueError escape _read_pid_record and abort the whole command.
studio_mod, _live, _killed = _install(monkeypatch, tmp_path, alive = set())
(tmp_path / "studio-8901-1.pid").write_text("²", encoding = "utf-8")
result = _run_stop(studio_mod)
assert result.exit_code == 0, result.output
assert not (tmp_path / "studio-8901-1.pid").exists()