ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Studio: Stop every running Unsloth server, not just the last one recorded (#7577)

Browse source 
* Stop every running Unsloth server, and refuse to start a second on a taken port

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Check the fallback range, guard PID reuse, and keep writing studio.pid

* Signal each server once when its PID is recorded in more than one file

* Confirm a recorded PID is a Studio server before signalling it

* Pin PID records to process start time and check every listener on a port

* Keep every recorded start time per PID and accept in-process Studio servers

* Match the blocking listener address and stop trusting unverifiable PID records

* Never delete a PID record that cannot be verified

* Detect our own server from our own records instead of a psutil listener scan

* Match a pre-upgrade studio.pid to the blocked port before falling back

* Never signal PID 0 or 1, and verify a per-port record before trusting it

* Stop unverifiable records instead of skipping them, and record every bind address

* Drop the command-line guess, fix Windows liveness, and free the PID record last

* Studio: harden the per-port PID records against the cases that lose a server

Follow-up on the per-port PID files. Each item below is a case where the new
code either lost a server the old code could still stop, or stopped something
that was not ours. All were reproduced against real Studio servers.

studio/backend/run.py

- Write the per-port record and the legacy studio.pid independently. They shared
  one try, so a studio root that could not take a new directory entry left the
  server recorded nowhere at all and unstoppable from the CLI; the old code
  still recorded it in studio.pid, which is an overwrite of an existing path and
  can still succeed. _remove_pid_file now also checks studio.pid when the
  per-port write failed.
- Write the record through a temp file and os.replace. `stop` reads these
  concurrently and treats a truncated read as a corrupt record.
- A failed Windows tasklist probe now means "alive", matching the CLI. Treating
  it as dead pruned a live server's record and let the next launch fall back
  past it, which is the orphan this work exists to fix.
- Guard the unlink in _own_studio_on_port. Pruning is a courtesy and must not
  abort startup.
- Extract _resolve_port so the requested-port abort is reachable from a test.
  Deleting that abort previously left the whole suite green.
- Keep the plain fallback for api-only callers. The desktop app hardcodes 8888
  and documents its reliance on the 8888-8908 range, and it reports a non-zero
  backend exit to the user as "Server stopped unexpectedly". It reads the bound
  port back from TAURI_PORT, as `studio run` does from app.state.server_port, so
  a fallback there is harmless and both servers are still recorded and
  stoppable. The interactive path prints the requested port, so it still aborts.
- isdigit() is not enough to gate int(): a superscript two passes it and the
  ValueError escaped into every caller of _read_pid_record.

unsloth_cli/commands/studio.py

- An untimed record no longer cancels a timed one for the same PID. Every
  current server writes both a timed per-port record and an untimed studio.pid,
  so the start-time check was inert exactly where it mattered, and after a crash
  plus a PID reuse `stop` sent SIGTERM to whatever unrelated process had
  inherited the PID.
- Distinguish an unreadable record from an invalid one. A root-owned record, or
  one caught mid-write, still belongs to a live server, and deleting it stranded
  that server.
- Route every PID-file removal through _unlink_quietly. One undeletable record
  raised PermissionError and left the remaining live servers running.
- Same isdigit()/int() guard as the backend.

Tests

- The requested-port abort, the recorded bind address, and the api-only
  fallback are now covered; all three previously survived deletion.
- tests/studio/test_studio_pid_file_contract.py pins run.py's filename scheme to
  the CLI's glob and keeps studio.pid parseable by an older CLI. It lives under
  tests/studio because unsloth_cli/tests is not run by any workflow.
- test_cli_studio_stop_windows.py now checks _signal_stop as well as stop. The
  kill moved into _signal_stop, so the os.kill(pid, 0) guard passed vacuously.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: let a caller that follows the port keep the fallback, and never take studio.pid from a live server

Two problems with keying the own-server abort on api_only.

`unsloth studio run` is not the bare-banner path: it stores `app = run_server(...)`
and reads `app.state.server_port` back, then uses it for the health wait, the
model load and the printed base URL. Gating on api_only aborted it, so starting
a second model while the first was up stopped working, where before it landed on
the next port and printed the right URL. Replace the proxy with an explicit
abort_if_own_studio, defaulting to the old api_only behaviour so the exec'd
`run.py` path is unchanged, and have `studio run` opt out.

The api_only exemption also reopened the orphan from the other side.
_write_pid_file overwrote studio.pid unconditionally, and a pre-upgrade server
is recorded there and nowhere else, so an exempt launch falling back past one
erased its only record. Take the file over only when it is free, already ours,
or held by a dead PID.

Also resync _pid_is_studio_backend with the CLI copy: an untimed record next to
a timed one carried no information but cancelled the start-time check, which is
what let a reused PID be treated as ours.

Tests: 51 backend, 26 CLI, 9 under tests/studio. Real Studio servers still abort
the bare same-port relaunch, still fall back past a foreign listener, and one
`unsloth studio stop` still stops every server in all five scenarios.

* Studio: hand over the legacy PID pointer, and fail stop on unreadable records

Two follow-ups from review of the previous commit.

Only one backend owns studio.pid at a time. When that server exited it
deleted the file, so an older CLI, which reads nothing else, could no
longer stop a sibling that was still serving. _remove_pid_file now hands
the pointer to a live sibling instead of dropping it.

_pid_file_entries skipped records it could not read, for instance one
written by a server started under sudo. When that was the only record,
stop printed "No running Unsloth server found" and exited 0 while the
server kept serving. Unreadable records are now reported and make stop
exit 1, so a partial stop is never mistaken for a complete one.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <unslothai@gmail.com>
This commit is contained in:
Nilay 2026-07-29 14:26:13 +05:30 • committed by GitHub
commit ceef4123e6
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 1699 additions and 59 deletions
Download patch file Download diff file Expand all files Collapse all files

347
studio/backend/run.py
View file

@ -10,7 +10,7 @@ import os
import sys
import time
from pathlib import Path
from typing import Optional, Tuple
from typing import NoReturn, Optional, Sequence, Tuple
def _fix_torch_cuda_ld_path():
@ -689,6 +689,33 @@ def _get_pid_on_port(port: int) -> "tuple[int, str] | None":
return None
def _bind_addresses(host: str, port: int) -> "set[str]":
"""Every address *host* resolves to. `localhost` is both 127.0.0.1 and ::1, and
recording only the first lets a later launch on the other one miss us."""
import socket
try:
infos = socket.getaddrinfo(host, port, socket.AF_UNSPEC, socket.SOCK_STREAM)
except OSError:
return {host}
return {info[4][0] for info in infos} or {host}
def _addresses_collide(recorded: "str | None", host: str, port: int) -> bool:
"""Would a server bound to *recorded* block a bind to *host*?
*recorded* may list several addresses. Unknown or wildcard on either side
collides: refusing with a clear message beats silently starting a duplicate.
"""
wildcards = ("0.0.0.0", "::", "")
if not recorded or host in wildcards:
return True
listed = {a.strip() for a in recorded.split(",") if a.strip()}
if not listed or listed & set(wildcards):
return True
return bool(listed & _bind_addresses(host, port))
def _is_port_free(host: str, port: int) -> bool:
"""Check if a port is available for binding.
@ -733,18 +760,213 @@ def _find_free_port(
host: str,
start: int,
max_attempts: int = 20,
avoid_own_studio: bool = False,
) -> int:
"""Find a free port from `start`, trying up to max_attempts ports."""
"""Find a free port from `start`, trying up to max_attempts ports.
``avoid_own_studio`` aborts rather than skipping past one of our own servers
in the fallback range, which would start a duplicate on a later port.
"""
for offset in range(max_attempts):
candidate = start + offset
if _is_port_free(host, candidate):
return candidate
if avoid_own_studio:
own = _own_studio_on_port(candidate, host)
if own is not None:
_abort_already_running(own, candidate)
raise RuntimeError(f"Could not find a free port in range {start}-{start + max_attempts - 1}")
from utils.paths.storage_roots import studio_root as _studio_root
# Legacy single-instance file; still read so `stop` finds an older build's server.
_PID_FILE = _studio_root() / "studio.pid"
PID_FILE_GLOB = "studio-*.pid"
def _pid_file_for_port(port: int) -> Path:
# PID in the name: 127.0.0.1 and ::1 can share a port, and one file per port
# would let the second bind overwrite the first.
return _studio_root() / f"studio-{port}-{os.getpid()}.pid"
def _pid_alive(pid: int) -> bool:
try:
import psutil
return psutil.pid_exists(pid)
except ImportError:
pass
if sys.platform == "win32":
# os.kill(pid, 0) raises OSError for every pid on Windows, so tasklist is
# the only usable probe here.
import subprocess
try:
out = subprocess.run(
["tasklist", "/FI", f"PID eq {int(pid)}", "/NH", "/FO", "CSV"],
capture_output = True,
text = True,
timeout = 10,
).stdout
except Exception:
# Unconfirmed means keep, matching the CLI's _pid_alive. Pruning a
# live server's record is what lets the next launch fall back past it
# and strand it, which is the bug this file exists to fix. A stale
# record instead costs one clear "already running" message.
return True
return f'"{int(pid)}"' in out
try:
os.kill(pid, 0)
except ProcessLookupError:
return False
except OSError:
return True
return True
def _process_create_time(pid: int) -> "float | None":
try:
import psutil
return psutil.Process(pid).create_time()
except Exception:
return None
def _read_pid_record(path: Path) -> "tuple[int, float | None, str | None] | None":
"""Parse ``pid`` / optional ``create_time`` / optional bind address."""
try:
lines = path.read_text(encoding = "utf-8").splitlines()
except (OSError, UnicodeDecodeError):
return None
if not lines or not lines[0].strip().isdigit():
return None
try:
# isdigit() is not enough: a superscript two passes it but int() rejects it.
pid = int(lines[0].strip())
except ValueError:
return None
# kill(0) signals our whole process group; kill(1) is init. Never either.
if pid < 2:
return None
created = None
if len(lines) > 1:
try:
created = float(lines[1].strip())
except ValueError:
created = None
address = lines[2].strip() if len(lines) > 2 and lines[2].strip() else None
return pid, created, address
def _pid_is_studio_backend(pid: int, created_times: "Sequence[float | None]" = ()) -> bool:
"""False only when a recorded start time proves this PID is a different process.
Any recorded time matching is enough -- a stale record must not veto a live
server that reused the PID. Untimed records cannot be checked at all, so they
are trusted: a legacy `python run.py` has no telltale argv, and guessing from
the command line rejected real servers.
"""
known = [c for c in created_times if c is not None]
if not known:
return True
actual = _process_create_time(pid)
if actual is None:
return True
return any(abs(actual - c) < 1.0 for c in known)
def _own_studio_on_port(port: int, host: str) -> "int | None":
"""PID of one of our own servers already bound to *port* for *host*.
Reads our own records rather than enumerating listeners: psutil is optional,
and without it a listener scan finds nothing and we silently start a duplicate.
"""
try:
paths = list(_studio_root().glob(f"studio-{port}-*.pid"))
except OSError:
return None
for path in paths:
record = _read_pid_record(path)
if record is None:
continue
pid, created, address = record
if not _pid_alive(pid):
# Pruning is a courtesy; an undeletable record must not abort startup.
try:
path.unlink(missing_ok = True)
except OSError:
pass
continue
if not _addresses_collide(address, host, port):
continue
if _pid_is_studio_backend(pid, [created]):
return pid
return _legacy_studio_on_port(port)
def _legacy_studio_on_port(port: int) -> "int | None":
"""A pre-upgrade server recorded only its PID, so match it to the listener.
Falling back past one leaves it running while `_write_pid_file` overwrites the
only record of it. When the listener is unknowable, assume it is ours.
"""
record = _read_pid_record(_PID_FILE)
if record is None:
return None
pid, created, _address = record
if not _pid_alive(pid):
return None
# A current build writes a per-port file too, so its port is already known --
# and this port's records were just checked. Only count a record that still
# matches the live process: a stale one may just share a reused PID.
for other in _per_port_records():
if other and other[0] == pid and _pid_is_studio_backend(pid, [other[1]]):
return None
blocker = _get_pid_on_port(port)
if blocker is not None and blocker[0] != pid:
return None
if not _pid_is_studio_backend(pid, [created]):
return None
return pid
def _per_port_records() -> "list[tuple[int, float | None, str | None] | None]":
try:
return [_read_pid_record(p) for p in _studio_root().glob(PID_FILE_GLOB)]
except OSError:
return []
def _resolve_port(
host: str,
port: int,
avoid_own_studio: bool = True,
) -> int:
"""The requested port, or the next free one.
With ``avoid_own_studio`` this aborts rather than falling back past one of our
own servers, on *port* itself or anywhere in the fallback range: skipping one
is what strands it. Callers that read the bound port back pass False and keep
the plain fallback.
"""
if _is_port_free(host, port):
return port
if avoid_own_studio:
own = _own_studio_on_port(port, host)
if own is not None:
_abort_already_running(own, port)
return _find_free_port(host, port + 1, avoid_own_studio = avoid_own_studio)
def _abort_already_running(pid: int, port: int) -> "NoReturn":
print(
f"Error: Unsloth Studio is already running on port {port} (PID {pid}). Run "
"`unsloth studio stop` first, or start this one on a different --port.",
file = sys.stderr,
flush = True,
)
sys.exit(1)
# Direct backend launches bypass the CLI's env re-export; do it here for
# real custom roots so unsloth-zoo's import-time LLAMA_CPP_DEFAULT_DIR
@ -770,25 +992,101 @@ if _STUDIO_ROOT_RESOLVED != _LEGACY_STUDIO_ROOT:
os.environ.setdefault("UNSLOTH_IS_PRESENT", "1")
def _write_pid_file():
"""Write the current process PID to the studio PID file."""
_OWN_PID_FILE: "Path | None" = None
def _write_pid_file(port: int, host: str = ""):
"""Record this PID under its own port so `stop` can find every server."""
global _OWN_PID_FILE
path = _pid_file_for_port(port)
try:
_PID_FILE.parent.mkdir(parents = True, exist_ok = True)
_PID_FILE.write_text(str(os.getpid()), encoding = "utf-8")
path.parent.mkdir(parents = True, exist_ok = True)
except OSError:
pass
try:
# Start time pins the record to this process; the bind address tells a
# later launch whether this server would actually block it.
created = _process_create_time(os.getpid())
address = ",".join(sorted(_bind_addresses(host, port))) if host else ""
body = f"{os.getpid()}\n{'' if created is None else repr(created)}\n{address}"
# Write-then-rename: `stop` reads these concurrently, and a reader that
# catches the truncate window sees a corrupt record and deletes it.
tmp = path.with_name(path.name + ".tmp")
try:
tmp.write_text(body, encoding = "utf-8")
os.replace(tmp, path)
finally:
# A failed replace would otherwise leave the scratch file behind. It
# does not end in .pid, so no glob picks it up either way.
tmp.unlink(missing_ok = True)
except OSError:
pass
else:
_OWN_PID_FILE = path
# An older CLI's `stop` only reads this one, and expects a bare PID. Written
# independently of the per-port record: if that one failed, this is the only
# thing keeping the server stoppable at all.
try:
# Never take it from a server that is still running. A pre-upgrade server
# is recorded here and nowhere else, so overwriting its entry is exactly
# what strands it -- the orphan this file exists to prevent.
prior = _read_pid_record(_PID_FILE) if _PID_FILE.is_file() else None
if prior is None or prior[0] == os.getpid() or not _pid_alive(prior[0]):
_PID_FILE.write_text(str(os.getpid()), encoding = "utf-8")
except OSError:
pass
def _remove_pid_file():
"""Remove the PID file if it belongs to this process."""
def _legacy_heir() -> "int | None":
"""Another live server's PID, to hand the legacy studio.pid over to.
Only one server owns studio.pid at a time, so its exit would otherwise drop
the single record an older CLI can read, stranding any sibling that is still
serving.
"""
try:
if _PID_FILE.is_file():
stored = _PID_FILE.read_text(encoding = "utf-8").strip()
if stored == str(os.getpid()):
paths = sorted(_studio_root().glob(PID_FILE_GLOB))
except OSError:
return None
for path in paths:
if _OWN_PID_FILE is not None and path == _OWN_PID_FILE:
continue
record = _read_pid_record(path)
if record is None or record[0] == os.getpid():
continue
if _pid_alive(record[0]) and _pid_is_studio_backend(record[0], [record[1]]):
return record[0]
return None
def _remove_pid_file():
"""Remove the PID files that belong to this process.
_PID_FILE is checked even when the per-port record was never written, since
_write_pid_file writes the two independently.
"""
# Nothing here may raise: _graceful_shutdown calls this at the end, and an
# unreadable or undeletable record must not abandon the rest of the exit
# path. _read_pid_record already swallows OSError/UnicodeDecodeError.
if _OWN_PID_FILE is not None:
try:
record = _read_pid_record(_OWN_PID_FILE) if _OWN_PID_FILE.is_file() else None
if record is not None and record[0] == os.getpid():
_OWN_PID_FILE.unlink(missing_ok = True)
except OSError:
pass
try:
record = _read_pid_record(_PID_FILE) if _PID_FILE.is_file() else None
if record is not None and record[0] == os.getpid():
# Hand the pointer to a live sibling rather than deleting it. An
# older CLI reads only this file, so dropping it while another
# server is still up leaves that server unstoppable.
heir = _legacy_heir()
if heir is None:
_PID_FILE.unlink(missing_ok = True)
# Runs first in _graceful_shutdown: a corrupt PID file raising here would
# abandon the children the rest of that function exists to kill.
except (OSError, UnicodeDecodeError):
else:
_PID_FILE.write_text(str(heir), encoding = "utf-8")
except OSError:
pass
@ -798,7 +1096,6 @@ def _graceful_shutdown(server = None):
Called from signal handlers to clean up children before exit. Critical on
Windows where atexit handlers are unreliable after Ctrl+C.
"""
_remove_pid_file()
logger.info("Graceful shutdown initiated -- cleaning up subprocesses...")
# 1. Shut down uvicorn (releases the listening socket).
@ -851,6 +1148,9 @@ def _graceful_shutdown(server = None):
except Exception as e:
logger.warning("Error in process-lifetime sweep: %s", e)
# Last: while cleanup runs the server is still alive, and dropping the record
# early leaves a retried `stop` or a new launch unable to find it.
_remove_pid_file()
logger.info("All subprocesses cleaned up")
@ -1400,6 +1700,7 @@ def run_server(
enable_tools: "Optional[bool]" = None,
password: "Optional[str]" = None,
emit_tauri_port: bool = True,
abort_if_own_studio: "Optional[bool]" = None,
):
"""
Start the FastAPI server.
@ -1533,10 +1834,16 @@ def run_server(
)
# Auto-find a free port if the requested one is in use.
if not _is_port_free(host, port):
original_port = port
blocker = _get_pid_on_port(port)
port = _find_free_port(host, port + 1)
original_port = port
# Refusing rather than falling back is for callers that cannot follow us to
# the new port. `studio run` reads app.state.server_port back and the desktop
# app reads TAURI_PORT, so both should keep the plain fallback; only the
# bare launch, which has nothing but the banner, benefits from the refusal.
if abort_if_own_studio is None:
abort_if_own_studio = not api_only
port = _resolve_port(host, port, avoid_own_studio = abort_if_own_studio)
if port != original_port:
blocker = _get_pid_on_port(original_port)
if not silent:
print("")
print("=" * 50)
@ -1734,7 +2041,7 @@ def run_server(
(time.perf_counter() - boot_started) * 1000,
)
_write_pid_file()
_write_pid_file(port, host)
import atexit
atexit.register(_remove_pid_file)

568
studio/backend/tests/test_studio_pid_files.py Normal file
View file

@ -0,0 +1,568 @@
# SPDX-License-Identifier: AGPL-3.0-only
# Copyright 2026-present the Unsloth AI Inc. team. All rights reserved. See /studio/LICENSE.AGPL-3.0
"""Per-port PID files, so `unsloth studio stop` can find every server.
Imports run.py directly, so run under the Unsloth venv.
"""
from __future__ import annotations
import os
import sys
from pathlib import Path
from types import SimpleNamespace
import pytest
_BACKEND = Path(__file__).resolve().parents[1]
if str(_BACKEND) not in sys.path:
sys.path.insert(0, str(_BACKEND))
import run # noqa: E402
# Captured before the autouse fixture stubs them, for the tests that exercise them.
_REAL_IS_STUDIO_BACKEND = run._pid_is_studio_backend
_REAL_PID_ALIVE = run._pid_alive
@pytest.fixture(autouse = True)
def isolated_root(tmp_path, monkeypatch):
monkeypatch.setattr(run, "_studio_root", lambda: tmp_path)
monkeypatch.setattr(run, "_PID_FILE", tmp_path / "studio.pid")
monkeypatch.setattr(run, "_OWN_PID_FILE", None)
monkeypatch.setattr(run, "_pid_alive", lambda pid: True)
monkeypatch.setattr(run, "_pid_is_studio_backend", lambda pid, created_times = (): True)
yield
def _files(tmp_path):
return sorted(p.name for p in tmp_path.glob("studio-*.pid"))
def _pid_of(path):
return path.read_text(encoding = "utf-8").splitlines()[0]
def test_write_pid_file_records_port_and_pid(tmp_path):
run._write_pid_file(8901)
assert _files(tmp_path) == [f"studio-8901-{os.getpid()}.pid"]
assert _pid_of(tmp_path / f"studio-8901-{os.getpid()}.pid") == str(os.getpid())
def test_write_pid_file_records_the_start_time(tmp_path):
# Pins the record to this process, so a reused PID isn't mistaken for it.
run._write_pid_file(8901)
record = run._read_pid_record(tmp_path / f"studio-8901-{os.getpid()}.pid")
assert record[0] == os.getpid()
assert record[1] == pytest.approx(run._process_create_time(os.getpid()))
def test_write_pid_file_keeps_the_legacy_file_a_bare_pid(tmp_path):
# An older CLI's `stop` reads studio.pid and expects only digits.
run._write_pid_file(8901)
assert (tmp_path / "studio.pid").read_text(encoding = "utf-8") == str(os.getpid())
def test_second_port_does_not_clobber_the_first(tmp_path):
(tmp_path / "studio-8901-8550.pid").write_text("8550", encoding = "utf-8")
run._write_pid_file(8902)
assert _pid_of(tmp_path / "studio-8901-8550.pid") == "8550"
assert (tmp_path / f"studio-8902-{os.getpid()}.pid").exists()
def test_same_port_on_two_binds_does_not_clobber(tmp_path):
# 127.0.0.1:8888 and ::1:8888 can both listen; one file per port would lose one.
(tmp_path / "studio-8888-8550.pid").write_text("8550", encoding = "utf-8")
run._write_pid_file(8888)
assert len(_files(tmp_path)) == 2
def test_remove_pid_file_only_removes_our_own(tmp_path, monkeypatch):
run._write_pid_file(8901)
(tmp_path / "studio-8902-8600.pid").write_text("8600", encoding = "utf-8")
# Nothing to hand the legacy pointer to, so it goes away with us.
monkeypatch.setattr(run, "_pid_alive", lambda pid: pid == os.getpid())
run._remove_pid_file()
assert _files(tmp_path) == ["studio-8902-8600.pid"]
assert not (tmp_path / "studio.pid").exists()
def test_the_legacy_pointer_moves_to_a_live_sibling(tmp_path):
# Only one server owns studio.pid. Deleting it on our way out would leave an
# older CLI, which reads nothing else, unable to stop the sibling still up.
run._write_pid_file(8901)
(tmp_path / "studio-8902-8600.pid").write_text("8600", encoding = "utf-8")
run._remove_pid_file()
assert (tmp_path / "studio.pid").read_text(encoding = "utf-8").strip() == "8600"
def test_the_legacy_pointer_is_not_handed_to_a_dead_sibling(tmp_path, monkeypatch):
run._write_pid_file(8901)
(tmp_path / "studio-8902-8600.pid").write_text("8600", encoding = "utf-8")
monkeypatch.setattr(run, "_pid_is_studio_backend", lambda pid, created_times = (): False)
run._remove_pid_file()
assert not (tmp_path / "studio.pid").exists()
def test_remove_pid_file_leaves_a_reused_entry_alone(tmp_path):
run._write_pid_file(8901)
own = tmp_path / f"studio-8901-{os.getpid()}.pid"
own.write_text("999999", encoding = "utf-8")
run._remove_pid_file()
assert own.read_text(encoding = "utf-8") == "999999"
def test_windows_liveness_does_not_call_every_pid_alive(monkeypatch):
# os.kill(pid, 0) raises OSError for every pid on Windows, so without the
# tasklist fallback a stale record would block its port forever.
import subprocess
monkeypatch.setattr(run, "_pid_alive", _REAL_PID_ALIVE)
monkeypatch.setitem(sys.modules, "psutil", None)
monkeypatch.setattr(sys, "platform", "win32")
monkeypatch.setattr(
subprocess, "run", lambda *a, **k: SimpleNamespace(stdout = '"python.exe","8550",...')
)
assert run._pid_alive(8550) is True
assert run._pid_alive(9999) is False
def test_windows_liveness_keeps_the_record_when_tasklist_fails(monkeypatch):
# Unconfirmed must mean keep, matching the CLI's _pid_alive. Pruning a live
# server's record lets the next launch fall back past it and strand it, which
# is the bug this file exists to fix; a stale record costs one clear abort.
import subprocess
def _boom(*a, **k):
raise OSError("tasklist missing")
monkeypatch.setattr(run, "_pid_alive", _REAL_PID_ALIVE)
monkeypatch.setitem(sys.modules, "psutil", None)
monkeypatch.setattr(sys, "platform", "win32")
monkeypatch.setattr(subprocess, "run", _boom)
assert run._pid_alive(8550) is True
def test_read_pid_record_parses_pid_time_and_address(tmp_path):
(tmp_path / "r.pid").write_text("8550\n111.5\n127.0.0.1", encoding = "utf-8")
assert run._read_pid_record(tmp_path / "r.pid") == (8550, 111.5, "127.0.0.1")
def test_read_pid_record_tolerates_a_bare_pid(tmp_path):
(tmp_path / "r.pid").write_text("8550", encoding = "utf-8")
assert run._read_pid_record(tmp_path / "r.pid") == (8550, None, None)
def test_read_pid_record_rejects_pid_zero_and_init(tmp_path):
# kill(0) signals our whole process group.
(tmp_path / "zero.pid").write_text("0", encoding = "utf-8")
(tmp_path / "init.pid").write_text("1", encoding = "utf-8")
assert run._read_pid_record(tmp_path / "zero.pid") is None
assert run._read_pid_record(tmp_path / "init.pid") is None
def test_read_pid_record_rejects_a_corrupt_file(tmp_path):
(tmp_path / "r.pid").write_text("not-a-pid", encoding = "utf-8")
assert run._read_pid_record(tmp_path / "r.pid") is None
def test_graceful_shutdown_drops_the_record_last(monkeypatch):
# Cleanup can take seconds while the server is still alive. Dropping the record
# first leaves a retried `stop` or a new launch unable to find it.
order = []
monkeypatch.setattr(run, "_remove_pid_file", lambda: order.append("remove_record"))
class _Server:
def __setattr__(self, name, value):
order.append("release_socket")
run._graceful_shutdown(_Server())
assert order == ["release_socket", "remove_record"]
def test_own_studio_on_port_is_found_without_psutil(tmp_path, monkeypatch):
# psutil is optional; a listener scan finds nothing without it, so detection
# must come from our own records or we silently start a duplicate.
monkeypatch.setitem(sys.modules, "psutil", None)
(tmp_path / "studio-8901-8550.pid").write_text("8550\n\n127.0.0.1", encoding = "utf-8")
assert run._own_studio_on_port(8901, "127.0.0.1") == 8550
def test_no_record_for_the_port_means_no_own_studio(tmp_path):
# jupyter-lab on 8888 must keep the fallback, not abort the launch.
(tmp_path / "studio-8901-8550.pid").write_text("8550", encoding = "utf-8")
assert run._own_studio_on_port(8888, "127.0.0.1") is None
def test_own_studio_on_port_prunes_a_dead_record(tmp_path, monkeypatch):
monkeypatch.setattr(run, "_pid_alive", lambda pid: False)
(tmp_path / "studio-8901-8550.pid").write_text("8550", encoding = "utf-8")
assert run._own_studio_on_port(8901, "127.0.0.1") is None
assert not (tmp_path / "studio-8901-8550.pid").exists()
def test_a_reused_pid_is_not_treated_as_our_studio(tmp_path, monkeypatch):
# Stale record + the OS handing that PID to something else must not abort.
monkeypatch.setattr(run, "_pid_is_studio_backend", lambda pid, created_times = (): False)
(tmp_path / "studio-8901-8550.pid").write_text("8550", encoding = "utf-8")
assert run._own_studio_on_port(8901, "127.0.0.1") is None
def test_an_unverifiable_record_still_blocks_a_duplicate(tmp_path, monkeypatch):
# Can't tell: refusing with a clear message beats a silent second instance.
monkeypatch.setattr(run, "_pid_is_studio_backend", lambda pid, created_times = (): True)
(tmp_path / "studio-8901-8550.pid").write_text("8550", encoding = "utf-8")
assert run._own_studio_on_port(8901, "127.0.0.1") == 8550
def test_start_time_mismatch_rejects_a_reused_pid(monkeypatch):
monkeypatch.setattr(run, "_pid_is_studio_backend", _REAL_IS_STUDIO_BACKEND)
monkeypatch.setattr(run, "_process_create_time", lambda pid: 999.0)
assert run._pid_is_studio_backend(8550, [111.5]) is False
assert run._pid_is_studio_backend(8550, [999.0]) is True
def test_a_stale_record_does_not_veto_a_live_server_sharing_the_pid(monkeypatch):
# Crash leaves studio-8888-1234.pid, the OS reuses 1234 for a new server on
# another port. Keeping only the first timestamp would reject the live one.
monkeypatch.setattr(run, "_pid_is_studio_backend", _REAL_IS_STUDIO_BACKEND)
monkeypatch.setattr(run, "_process_create_time", lambda pid: 999.0)
assert run._pid_is_studio_backend(1234, [111.5, 999.0]) is True
assert run._pid_is_studio_backend(1234, [111.5, 222.5]) is False
def test_a_stale_record_on_another_port_does_not_hide_a_live_server(tmp_path, monkeypatch):
# 1234 was reused: the stale 8888 record must not stop us seeing 9000.
monkeypatch.setattr(run, "_pid_is_studio_backend", _REAL_IS_STUDIO_BACKEND)
monkeypatch.setattr(run, "_process_create_time", lambda pid: 999.0)
(tmp_path / "studio-8888-1234.pid").write_text("1234\n111.5\n", encoding = "utf-8")
(tmp_path / "studio-9000-1234.pid").write_text("1234\n999.0\n", encoding = "utf-8")
assert run._own_studio_on_port(8888, "127.0.0.1") is None
assert run._own_studio_on_port(9000, "127.0.0.1") == 1234
def test_a_start_time_is_the_only_thing_that_disproves_a_record(monkeypatch):
monkeypatch.setattr(run, "_pid_is_studio_backend", _REAL_IS_STUDIO_BACKEND)
monkeypatch.setattr(run, "_process_create_time", lambda pid: 999.0)
assert run._pid_is_studio_backend(8550, [999.0]) is True
assert run._pid_is_studio_backend(8550, [111.5]) is False
def test_a_bare_run_py_command_line_is_not_rejected(monkeypatch):
# `cd studio/backend && python run.py --port 8901` has no "studio" or "unsloth"
# in argv. Guessing from the command line called that "not ours".
monkeypatch.setattr(run, "_pid_is_studio_backend", _REAL_IS_STUDIO_BACKEND)
class _FakeProcess:
def __init__(self, pid):
self.pid = pid
def cmdline(self):
return ["python", "run.py", "--port", "8901"]
def create_time(self):
return 111.5
monkeypatch.setitem(sys.modules, "psutil", SimpleNamespace(Process = _FakeProcess))
assert run._pid_is_studio_backend(8550) is True
def test_an_untimed_legacy_record_is_trusted(monkeypatch):
# `python run.py --port 8901` has no telltale argv, so guessing from the
# command line rejected real servers. Only a start time can disprove one.
monkeypatch.setattr(run, "_pid_is_studio_backend", _REAL_IS_STUDIO_BACKEND)
monkeypatch.setattr(run, "_process_create_time", lambda pid: 999.0)
assert run._pid_is_studio_backend(8550) is True
assert run._pid_is_studio_backend(8550, [None]) is True
def test_the_untimed_legacy_record_does_not_cancel_a_timed_one(monkeypatch):
# Mirrors _pid_is_studio_server in the CLI. An untimed record carries no
# information, so it must not overrule a start time that says "not ours" --
# every current server writes one of each, which made the check inert.
monkeypatch.setattr(run, "_pid_is_studio_backend", _REAL_IS_STUDIO_BACKEND)
monkeypatch.setattr(run, "_process_create_time", lambda pid: 999.0)
assert run._pid_is_studio_backend(8550, [111.5, None]) is False
assert run._pid_is_studio_backend(8550, [111.5, 999.0]) is True
def test_a_legacy_server_on_the_port_is_recognised(tmp_path, monkeypatch):
# Pre-upgrade servers wrote only studio.pid. Falling back past one strands it
# and then overwrites its record.
monkeypatch.setattr(run, "_get_pid_on_port", lambda p: (8550, "python"))
(tmp_path / "studio.pid").write_text("8550", encoding = "utf-8")
assert run._own_studio_on_port(8901, "127.0.0.1") == 8550
def test_a_legacy_record_for_a_different_listener_falls_back(tmp_path, monkeypatch):
# jupyter holds the port; the legacy server is elsewhere. Keep falling back.
monkeypatch.setattr(run, "_get_pid_on_port", lambda p: (117, "jupyter-lab"))
(tmp_path / "studio.pid").write_text("8550", encoding = "utf-8")
assert run._own_studio_on_port(8901, "127.0.0.1") is None
def test_an_unknowable_listener_treats_the_legacy_record_as_ours(tmp_path, monkeypatch):
# No psutil: _get_pid_on_port can't say. Refusing beats a silent duplicate.
monkeypatch.setattr(run, "_get_pid_on_port", lambda p: None)
(tmp_path / "studio.pid").write_text("8550", encoding = "utf-8")
assert run._own_studio_on_port(8901, "127.0.0.1") == 8550
def test_a_dead_legacy_record_falls_back(tmp_path, monkeypatch):
monkeypatch.setattr(run, "_pid_alive", lambda pid: False)
monkeypatch.setattr(run, "_get_pid_on_port", lambda p: None)
(tmp_path / "studio.pid").write_text("8550", encoding = "utf-8")
assert run._own_studio_on_port(8901, "127.0.0.1") is None
def test_a_stale_per_port_record_does_not_mask_a_legacy_server(tmp_path, monkeypatch):
# Crashed current build left studio-8901-8550.pid; 8550 was then reused by a
# pre-upgrade server recorded only in studio.pid. The stale record must not
# count as "port already known" and send us falling back past the live one.
monkeypatch.setattr(run, "_pid_is_studio_backend", _REAL_IS_STUDIO_BACKEND)
monkeypatch.setattr(run, "_process_create_time", lambda pid: 999.0)
monkeypatch.setattr(run, "_get_pid_on_port", lambda p: (8550, "python"))
(tmp_path / "studio-8901-8550.pid").write_text("8550\n111.5\n127.0.0.1", encoding = "utf-8")
(tmp_path / "studio.pid").write_text("8550", encoding = "utf-8")
assert run._own_studio_on_port(8901, "127.0.0.1") == 8550
def test_a_current_server_elsewhere_does_not_block_a_foreign_port(tmp_path, monkeypatch):
# Current builds write studio.pid too. Without psutil the legacy check can't
# see the listener, so it must not claim our 8901 server holds jupyter's 8888.
monkeypatch.setattr(run, "_get_pid_on_port", lambda p: None)
(tmp_path / "studio-8901-5000.pid").write_text("5000\n\n127.0.0.1", encoding = "utf-8")
(tmp_path / "studio.pid").write_text("5000", encoding = "utf-8")
assert run._own_studio_on_port(8888, "127.0.0.1") is None
def test_a_per_port_record_is_preferred_over_the_legacy_one(tmp_path, monkeypatch):
monkeypatch.setattr(run, "_get_pid_on_port", lambda p: (8550, "python"))
(tmp_path / "studio-8901-8600.pid").write_text("8600\n\n127.0.0.1", encoding = "utf-8")
(tmp_path / "studio.pid").write_text("8550", encoding = "utf-8")
assert run._own_studio_on_port(8901, "127.0.0.1") == 8600
def test_our_studio_on_another_bind_address_does_not_abort(tmp_path):
# Our server holds ::1:8889; binding 127.0.0.1:8889 is not a conflict with us,
# so fall through to the next port instead of refusing.
(tmp_path / "studio-8889-8550.pid").write_text("8550\n\n::1", encoding = "utf-8")
assert run._own_studio_on_port(8889, "127.0.0.1") is None
assert run._own_studio_on_port(8889, "::1") == 8550
def test_address_matching(tmp_path):
assert run._addresses_collide("0.0.0.0", "127.0.0.1", 8889) is True
assert run._addresses_collide("127.0.0.1", "0.0.0.0", 8889) is True
assert run._addresses_collide("127.0.0.1", "127.0.0.1", 8889) is True
assert run._addresses_collide("::1", "127.0.0.1", 8889) is False
# An unrecorded address is unknown, so assume a conflict.
assert run._addresses_collide(None, "127.0.0.1", 8889) is True
def test_a_hostname_resolves_the_same_way_the_bind_does(tmp_path):
# `localhost` and the address _is_port_free actually binds must agree, or a
# recorded server is missed and a duplicate starts.
recorded = ",".join(sorted(run._bind_addresses("localhost", 8889)))
assert run._addresses_collide(recorded, "localhost", 8889) is True
def test_a_hostname_records_every_address_it_resolves_to(tmp_path):
# `localhost` binds 127.0.0.1 AND ::1. Recording only the first lets a later
# launch on the other literal miss us and start a duplicate.
addrs = run._bind_addresses("localhost", 8889)
recorded = ",".join(sorted(addrs))
for literal in addrs:
assert run._addresses_collide(recorded, literal, 8889) is True
def test_a_multi_address_record_matches_either_literal(tmp_path):
recorded = "127.0.0.1,::1"
assert run._addresses_collide(recorded, "127.0.0.1", 8889) is True
assert run._addresses_collide(recorded, "::1", 8889) is True
assert run._addresses_collide("127.0.0.1", "::1", 8889) is False
def test_fallback_aborts_on_our_own_server_further_up_the_range(tmp_path, monkeypatch):
# jupyter holds 8888, our server holds 8889: skipping to 8890 is the duplicate.
(tmp_path / "studio-8889-8550.pid").write_text("8550\n\n127.0.0.1", encoding = "utf-8")
monkeypatch.setattr(run, "_is_port_free", lambda host, p: p >= 8890)
with pytest.raises(SystemExit) as excinfo:
run._find_free_port("127.0.0.1", 8889, avoid_own_studio = True)
assert excinfo.value.code == 1
def test_fallback_still_skips_foreign_processes(tmp_path, monkeypatch):
# No record for 8889, so the blocker is not ours: keep falling back.
monkeypatch.setattr(run, "_is_port_free", lambda host, p: p >= 8890)
assert run._find_free_port("127.0.0.1", 8889, avoid_own_studio = True) == 8890
def test_the_requested_port_is_kept_when_it_is_free(monkeypatch):
monkeypatch.setattr(run, "_is_port_free", lambda host, p: True)
assert run._resolve_port("127.0.0.1", 8888) == 8888
def test_our_own_server_on_the_requested_port_aborts_rather_than_falling_back(
tmp_path, monkeypatch
):
# The reported bug: 8888 is ours, so falling back to 8889 is the duplicate
# that leaves 8888 serving with nothing recording it.
monkeypatch.setattr(run, "_is_port_free", lambda host, p: p != 8888)
(tmp_path / "studio-8888-8550.pid").write_text("8550\n\n127.0.0.1", encoding = "utf-8")
with pytest.raises(SystemExit) as excinfo:
run._resolve_port("127.0.0.1", 8888)
assert excinfo.value.code == 1
def test_a_foreign_process_on_the_requested_port_still_falls_back(monkeypatch):
# jupyter-lab on 8888 must not stop Unsloth starting on 8889.
monkeypatch.setattr(run, "_is_port_free", lambda host, p: p != 8888)
assert run._resolve_port("127.0.0.1", 8888) == 8889
def test_a_caller_that_reads_the_port_back_keeps_the_plain_fallback(tmp_path, monkeypatch):
# api-only callers (the desktop app via TAURI_PORT, `studio run` via
# app.state.server_port) follow us to the new port, so aborting there only
# turns a working launch into a crash the desktop app reports as "stopped
# unexpectedly". Both servers are still recorded, so `stop` finds them.
monkeypatch.setattr(run, "_is_port_free", lambda host, p: p != 8888)
(tmp_path / "studio-8888-8550.pid").write_text("8550\n\n127.0.0.1", encoding = "utf-8")
assert run._resolve_port("127.0.0.1", 8888, avoid_own_studio = False) == 8889
def test_the_recorded_address_is_every_address_the_bind_resolves_to(tmp_path):
# The only test that runs the writer with a real host. Recording `host`
# verbatim, or dropping the line, passes every other test here and silently
# stops matching a launch that spells the same interface differently.
run._write_pid_file(8901, "localhost")
record = run._read_pid_record(tmp_path / f"studio-8901-{os.getpid()}.pid")
assert record[2] is not None, "no bind address recorded"
assert set(record[2].split(",")) == run._bind_addresses("localhost", 8901)
def test_a_server_started_on_a_hostname_is_found_again_by_ip(tmp_path):
run._write_pid_file(8901, "localhost")
for literal in run._bind_addresses("localhost", 8901):
assert run._own_studio_on_port(8901, literal) == os.getpid()
def test_bind_addresses_keeps_every_family_a_hostname_resolves_to(monkeypatch):
# Independent oracle: the sibling test derives its expectation from this
# function's own output, so dropping a family would pass it.
import socket
monkeypatch.setattr(
socket,
"getaddrinfo",
lambda *a, **k: [
(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("127.0.0.1", 8889)),
(socket.AF_INET6, socket.SOCK_STREAM, 6, "", ("::1", 8889, 0, 0)),
],
)
assert run._bind_addresses("localhost", 8889) == {"127.0.0.1", "::1"}
def test_the_legacy_file_is_written_even_when_the_per_port_record_fails(tmp_path, monkeypatch):
# A studio root that cannot take a new entry used to leave the server
# recorded nowhere at all, so the CLI could not stop it. studio.pid is an
# overwrite of an existing path, so it can still succeed and must be tried.
blocked = tmp_path / "not-a-directory"
blocked.write_text("", encoding = "utf-8")
monkeypatch.setattr(
run, "_pid_file_for_port", lambda port: blocked / f"studio-{port}-{os.getpid()}.pid"
)
run._write_pid_file(8901, "127.0.0.1")
assert (tmp_path / "studio.pid").read_text(encoding = "utf-8") == str(os.getpid())
assert run._OWN_PID_FILE is None
def test_a_record_whose_pid_is_not_ascii_digits_is_discarded(tmp_path):
# A superscript two passes isdigit() but int() rejects it, so that gate alone
# let a ValueError escape into every caller of _read_pid_record.
(tmp_path / "r.pid").write_text("²", encoding = "utf-8")
assert run._read_pid_record(tmp_path / "r.pid") is None
def test_the_legacy_file_is_not_taken_from_a_live_server(tmp_path):
# A pre-upgrade server is recorded in studio.pid and nowhere else, so a
# second launch overwriting it is exactly what strands it. That is the
# orphan this file exists to prevent, reached from the other direction.
(tmp_path / "studio.pid").write_text("8550", encoding = "utf-8")
run._write_pid_file(8902, "127.0.0.1")
assert (tmp_path / "studio.pid").read_text(encoding = "utf-8") == "8550"
assert (tmp_path / f"studio-8902-{os.getpid()}.pid").exists()
def test_the_legacy_file_is_taken_over_from_a_dead_server(tmp_path, monkeypatch):
# A stale record must not keep the pointer forever, or an older CLI could
# never stop anything again.
monkeypatch.setattr(run, "_pid_alive", lambda pid: False)
(tmp_path / "studio.pid").write_text("8550", encoding = "utf-8")
run._write_pid_file(8902, "127.0.0.1")
assert (tmp_path / "studio.pid").read_text(encoding = "utf-8") == str(os.getpid())