Commit graph

3 commits

Author SHA1 Message Date
Daniel Han
8e0d082c92
Reap Studio child processes when the parent dies abnormally (#6425)
* Reap Studio child processes when the parent dies abnormally

Standalone `unsloth studio` launches orphaned cloudflared and llama-server when
the parent exited without running the cooperative shutdown path (terminal-window
close, Task Manager End Task, SIGKILL): the children reparented to init and kept
running, leaving an authenticated Cloudflare tunnel up for days.

Add utils/process_lifetime.py: a parent-owned Windows Job Object
(JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE, children auto-inherit) plus Linux
PR_SET_PDEATHSIG, behind a best-effort helper that mirrors the desktop app's
windows_job.rs. initialize_parent_lifetime() runs at the top of run_server;
long-lived spawns (cloudflared, llama-server, RAG embedder, llama.cpp updater)
get the PDEATHSIG preexec, multiprocessing workers are adopted into the job, and
_graceful_shutdown plus atexit gain a terminate_all() backstop sweep. The
cooperative shutdown path is otherwise unchanged.

Verified on Linux: killing the parent now reaps cloudflared and llama-server
within ~2s instead of orphaning them.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* test: add real Windows kill-on-job-close integration test

Spawn a parent that installs the job and a child that inherits it, terminate
the parent, and assert the child is reaped. Skipped off Windows. Also make the
liveness probe Windows-safe (os.kill(pid, 0) terminates on Windows).

* Fix Win64 handle truncation in the Job Object calls

Set explicit argtypes so the 64-bit job/process handles are not marshaled as
c_int (which truncated them on Win64, failing AssignProcessToJobObject). Assert
install success in the Windows integration test.

* Bind multiprocessing workers to parent death; harden the sweep

Review follow-ups:
- Multiprocessing workers (inference/export/training/data-recipe/Xet) cannot be
  given a preexec_fn by the parent, so adopt_pid alone left them orphanable on a
  Linux SIGKILL. They now bind themselves with PR_SET_PDEATHSIG at startup via
  bind_current_process_to_parent_lifetime(), wired into the shared
  run_without_native_path_secret entrypoint and the Xet child entry.
- Wire the previously-missed data-recipe worker through adopt_pid.
- terminate_all now honors its timeout: SIGTERM, wait, then SIGKILL the
  survivors, so cooperative children can exit cleanly.
- Track adopted pids with a /proc starttime identity and add forget_pid, so the
  shutdown sweep never signals a recycled pid.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: Michael Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-06-18 05:51:22 -07:00
oobabooga
01d152b06f
Studio: only advertise a Cloudflare tunnel once it actually serves (#6264) 2026-06-12 14:36:35 -03:00
Daniel Han
1a99980b46
Studio: auto Cloudflare tunnel for 0.0.0.0 launches (#6204)
* Studio: auto Cloudflare tunnel for 0.0.0.0 launches

Binding Studio to 0.0.0.0 for remote access often leaves the raw
http://<ip>:<port> URL unreachable (https-vs-http, blocked high ports,
closed cloud security groups). On a wildcard bind, auto-start a free
cloudflared quick tunnel and show its https://*.trycloudflare.com URL in
the startup banner:

  Secure link access via Cloudflare: https://<random>.trycloudflare.com

- new studio/backend/cloudflare_tunnel.py: find or download+cache the
  cloudflared binary (per-OS/arch GitHub release, safe .tgz extract),
  start the tunnel, parse the URL, tear it down. Stdlib only; best-effort
  and non-fatal throughout (a missing binary or offline box never blocks
  or slows startup).
- run_server starts the tunnel for 0.0.0.0 only (skips loopback, api-only
  and Colab), prints the line in the banner, and _graceful_shutdown stops
  the child so it never orphans.
- --cloudflare/--no-cloudflare flag (default on) on `unsloth studio` and
  `unsloth studio run`, forwarded through the re-exec into run_server.
- tests for the helper, the CLI flag forwarding, and the run.py defaults.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio cloudflare: send a User-Agent on the cloudflared download

GitHub's CDN can 403 the default Python-urllib User-Agent on release asset
downloads. Set an explicit UA and pin it with a test.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio cloudflare: address review (opt-out for subcommands, tunnel teardown)

- reject --no-cloudflare placed before a subcommand (it would not reach the
  subcommand), mirroring the --parallel guard
- register the tunnel before waiting for its URL so a shutdown during the wait
  stops cloudflared instead of orphaning it
- tear the server + children down if `unsloth studio run` startup aborts
  (health timeout, model-load error, Ctrl+C) before the wait loop

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-06-11 07:10:08 -07:00