ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Studio: Stop every running Unsloth server, not just the last one recorded (#7577)

Browse source 
* Stop every running Unsloth server, and refuse to start a second on a taken port

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Check the fallback range, guard PID reuse, and keep writing studio.pid

* Signal each server once when its PID is recorded in more than one file

* Confirm a recorded PID is a Studio server before signalling it

* Pin PID records to process start time and check every listener on a port

* Keep every recorded start time per PID and accept in-process Studio servers

* Match the blocking listener address and stop trusting unverifiable PID records

* Never delete a PID record that cannot be verified

* Detect our own server from our own records instead of a psutil listener scan

* Match a pre-upgrade studio.pid to the blocked port before falling back

* Never signal PID 0 or 1, and verify a per-port record before trusting it

* Stop unverifiable records instead of skipping them, and record every bind address

* Drop the command-line guess, fix Windows liveness, and free the PID record last

* Studio: harden the per-port PID records against the cases that lose a server

Follow-up on the per-port PID files. Each item below is a case where the new
code either lost a server the old code could still stop, or stopped something
that was not ours. All were reproduced against real Studio servers.

studio/backend/run.py

- Write the per-port record and the legacy studio.pid independently. They shared
  one try, so a studio root that could not take a new directory entry left the
  server recorded nowhere at all and unstoppable from the CLI; the old code
  still recorded it in studio.pid, which is an overwrite of an existing path and
  can still succeed. _remove_pid_file now also checks studio.pid when the
  per-port write failed.
- Write the record through a temp file and os.replace. `stop` reads these
  concurrently and treats a truncated read as a corrupt record.
- A failed Windows tasklist probe now means "alive", matching the CLI. Treating
  it as dead pruned a live server's record and let the next launch fall back
  past it, which is the orphan this work exists to fix.
- Guard the unlink in _own_studio_on_port. Pruning is a courtesy and must not
  abort startup.
- Extract _resolve_port so the requested-port abort is reachable from a test.
  Deleting that abort previously left the whole suite green.
- Keep the plain fallback for api-only callers. The desktop app hardcodes 8888
  and documents its reliance on the 8888-8908 range, and it reports a non-zero
  backend exit to the user as "Server stopped unexpectedly". It reads the bound
  port back from TAURI_PORT, as `studio run` does from app.state.server_port, so
  a fallback there is harmless and both servers are still recorded and
  stoppable. The interactive path prints the requested port, so it still aborts.
- isdigit() is not enough to gate int(): a superscript two passes it and the
  ValueError escaped into every caller of _read_pid_record.

unsloth_cli/commands/studio.py

- An untimed record no longer cancels a timed one for the same PID. Every
  current server writes both a timed per-port record and an untimed studio.pid,
  so the start-time check was inert exactly where it mattered, and after a crash
  plus a PID reuse `stop` sent SIGTERM to whatever unrelated process had
  inherited the PID.
- Distinguish an unreadable record from an invalid one. A root-owned record, or
  one caught mid-write, still belongs to a live server, and deleting it stranded
  that server.
- Route every PID-file removal through _unlink_quietly. One undeletable record
  raised PermissionError and left the remaining live servers running.
- Same isdigit()/int() guard as the backend.

Tests

- The requested-port abort, the recorded bind address, and the api-only
  fallback are now covered; all three previously survived deletion.
- tests/studio/test_studio_pid_file_contract.py pins run.py's filename scheme to
  the CLI's glob and keeps studio.pid parseable by an older CLI. It lives under
  tests/studio because unsloth_cli/tests is not run by any workflow.
- test_cli_studio_stop_windows.py now checks _signal_stop as well as stop. The
  kill moved into _signal_stop, so the os.kill(pid, 0) guard passed vacuously.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: let a caller that follows the port keep the fallback, and never take studio.pid from a live server

Two problems with keying the own-server abort on api_only.

`unsloth studio run` is not the bare-banner path: it stores `app = run_server(...)`
and reads `app.state.server_port` back, then uses it for the health wait, the
model load and the printed base URL. Gating on api_only aborted it, so starting
a second model while the first was up stopped working, where before it landed on
the next port and printed the right URL. Replace the proxy with an explicit
abort_if_own_studio, defaulting to the old api_only behaviour so the exec'd
`run.py` path is unchanged, and have `studio run` opt out.

The api_only exemption also reopened the orphan from the other side.
_write_pid_file overwrote studio.pid unconditionally, and a pre-upgrade server
is recorded there and nowhere else, so an exempt launch falling back past one
erased its only record. Take the file over only when it is free, already ours,
or held by a dead PID.

Also resync _pid_is_studio_backend with the CLI copy: an untimed record next to
a timed one carried no information but cancelled the start-time check, which is
what let a reused PID be treated as ours.

Tests: 51 backend, 26 CLI, 9 under tests/studio. Real Studio servers still abort
the bare same-port relaunch, still fall back past a foreign listener, and one
`unsloth studio stop` still stops every server in all five scenarios.

* Studio: hand over the legacy PID pointer, and fail stop on unreadable records

Two follow-ups from review of the previous commit.

Only one backend owns studio.pid at a time. When that server exited it
deleted the file, so an older CLI, which reads nothing else, could no
longer stop a sibling that was still serving. _remove_pid_file now hands
the pointer to a live sibling instead of dropping it.

_pid_file_entries skipped records it could not read, for instance one
written by a server started under sudo. When that was the only record,
stop printed "No running Unsloth server found" and exited 0 while the
server kept serving. Unreadable records are now reported and make stop
exit 1, so a partial stop is never mistaken for a complete one.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <unslothai@gmail.com>
This commit is contained in:
Nilay 2026-07-29 14:26:13 +05:30 • committed by GitHub
commit ceef4123e6
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 1699 additions and 59 deletions
Download patch file Download diff file Expand all files Collapse all files

16
tests/studio/test_cli_studio_stop_windows.py
View file

@ -44,9 +44,12 @@ def _load_pid_alive(platform: str, fake_run = None):
# ── AST: stop() must not use the broken bare liveness probe ──────────────────
def test_stop_does_not_use_bare_oskill_liveness_probe():
"""stop() must not call os.kill(pid, 0) -- it crashes on Windows."""
stop_src = _func_source("stop")
# `stop` delegates signalling to `_signal_stop`, so guarding only `stop` would
# let os.kill(pid, 0) come back one function along and still pass.
@pytest.mark.parametrize("func", ["stop", "_signal_stop"])
def test_stop_does_not_use_bare_oskill_liveness_probe(func):
"""The signalling path must not call os.kill(pid, 0) -- WinError 87 on Windows."""
stop_src = _func_source(func)
tree = ast.parse(stop_src)
for call in ast.walk(tree):
if not isinstance(call, ast.Call):
@ -62,14 +65,17 @@ def test_stop_does_not_use_bare_oskill_liveness_probe():
sig = call.args[1]
if isinstance(sig, ast.Constant) and sig.value == 0:
raise AssertionError(
"stop() still uses os.kill(pid, 0); it raises WinError 87 on "
"Windows. Use the cross-platform _pid_alive() helper instead."
f"{func}() still uses os.kill(pid, 0); it raises WinError 87 "
"on Windows. Use the cross-platform _pid_alive() helper."
)
def test_pid_alive_helper_is_defined_and_used_by_stop():
assert "def _pid_alive(" in _SOURCE, "_pid_alive helper missing"
assert "_pid_alive(pid)" in _func_source("stop"), "stop() must use _pid_alive"
# The kill itself moved into _signal_stop; keep both ends of the path pinned.
assert "def _signal_stop(" in _SOURCE, "_signal_stop helper missing"
assert "taskkill" in _func_source("_signal_stop")
# The helper must special-case Windows via tasklist (os.kill(pid,0) is invalid there).
helper = _func_source("_pid_alive")
assert 'sys.platform == "win32"' in helper

73
tests/studio/test_studio_pid_file_contract.py Normal file
View file

@ -0,0 +1,73 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved.
"""run.py writes the Studio PID files; `unsloth studio stop` globs for them.
Nothing else ties the writer's filename to the reader's glob, and each side's own
tests hardcode the names they expect, so a rename on either side alone leaves both
suites green while `stop` silently finds nothing. `unsloth_cli/tests/` also runs
in no workflow, so this lives here, where the repo CPU job discovers it.
AST + exec of the writer, so no backend dependency stack is imported.
"""
import ast
import os
import sys
from pathlib import Path
_ROOT = Path(__file__).resolve().parents[2]
if str(_ROOT) not in sys.path:
sys.path.insert(0, str(_ROOT))
_RUN_SRC = (_ROOT / "studio" / "backend" / "run.py").read_text(encoding = "utf-8")
def _func_source(source: str, name: str) -> str:
for node in ast.walk(ast.parse(source)):
if isinstance(node, ast.FunctionDef) and node.name == name:
return ast.get_source_segment(source, node)
raise AssertionError(f"function {name!r} not found")
def _backend_pid_path(root: Path, port: int) -> Path:
"""The path run.py's own _pid_file_for_port builds, without importing run.py."""
ns = {"os": os, "Path": Path, "_studio_root": lambda: root}
exec(_func_source(_RUN_SRC, "_pid_file_for_port"), ns)
return ns["_pid_file_for_port"](port)
def test_stop_finds_a_pid_file_named_the_way_the_backend_writes_it(tmp_path, monkeypatch):
from unsloth_cli.commands import studio as cli
path = _backend_pid_path(tmp_path, 8901)
# The same three-line body _write_pid_file emits (create_time is blank when
# psutil is unavailable, and the CLI must tolerate that).
path.write_text(f"{os.getpid()}\n\n127.0.0.1", encoding = "utf-8")
monkeypatch.setattr(cli, "STUDIO_HOME", tmp_path)
monkeypatch.setattr(cli, "_PID_FILE", tmp_path / "studio.pid")
assert [pid for pid, _times, _files in cli._pid_file_entries()] == [os.getpid()]
def test_the_legacy_file_stays_a_bare_pid_an_older_cli_can_parse(tmp_path, monkeypatch):
# An older `unsloth studio stop` reads studio.pid and requires str.isdigit(),
# so the compatibility file must never gain the extra metadata lines.
ns = {
"os": os,
"Path": Path,
"_studio_root": lambda: tmp_path,
"_PID_FILE": tmp_path / "studio.pid",
"_pid_file_for_port": lambda port: _backend_pid_path(tmp_path, port),
"_process_create_time": lambda pid: None,
"_bind_addresses": lambda host, port: {host},
# _write_pid_file consults these before taking over studio.pid.
"_read_pid_record": lambda path: None,
"_pid_alive": lambda pid: False,
"_OWN_PID_FILE": None,
}
exec(_func_source(_RUN_SRC, "_write_pid_file"), ns)
ns["_write_pid_file"](8901, "127.0.0.1")
assert (tmp_path / "studio.pid").read_text(encoding = "utf-8").strip().isdigit()