unsloth/studio/backend/utils
Daniel Han 9fc21b3977
Studio: make lifespan shutdown resilient to a dead default executor (#6307)
* Studio: make lifespan shutdown resilient to a dead default executor

On an abrupt shutdown (closing the Windows console window, or interpreter
teardown racing uvicorn's graceful stop) the event loop's default thread-pool
executor can already be shut down by the time the FastAPI lifespan shutdown
runs. The first post-yield statement was an unguarded
`await asyncio.to_thread(terminate_hub_downloads)`, so executor.submit raised
`RuntimeError: cannot schedule new futures after shutdown`. That raise
propagated up through every nested merged_lifespan __aexit__, aborted the rest
of the cleanup (DEVICE reset, compiled-cache clear), and surfaced as
"Application shutdown failed. Exiting."

Extract the post-yield cleanup into utils/lifespan_shutdown.run_lifespan_shutdown
and guard each step independently. On the to_thread RuntimeError, fall back to
running the (already best-effort, quick) terminate inline on the loop thread so
shutdown still completes cleanly. The helper is dependency-injected and free of
the heavy backend import graph, so it is unit-tested in isolation.

Add tests/test_lifespan_shutdown.py (4 cases: dead-executor survival, normal
path, terminate error, clear error). Validated on windows-latest and
ubuntu-latest runners on Python 3.13.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Studio: only retry terminate inline when scheduling fails, not when the body raises

Address review feedback on the shutdown helper: the previous
`except RuntimeError` after `asyncio.to_thread(terminate_downloads)` could
not tell a dead-executor scheduling failure from a RuntimeError raised by
terminate_downloads itself, so a body-side RuntimeError on a healthy executor
ran the cleanup a second time inline.

Schedule via loop.run_in_executor and await separately: a dead default executor
raises synchronously at submit time (inline fallback), while a body exception
only surfaces when the future is awaited (logged, never retried). Add a
regression test that a body RuntimeError runs terminate exactly once.

* Studio: run terminate cleanup with a copied context (parity with asyncio.to_thread)

Simulation across the executor-state x exception x DEVICE matrix surfaced the one
behavioural difference from the original implementation: asyncio.to_thread copies
the caller's contextvars into the worker thread, while a bare
run_in_executor(None, fn) does not. Restore exact parity by scheduling
ctx.run(terminate_downloads) from a contextvars.copy_context(), so the refactor
is a behavioural no-op apart from the intended dead-executor recovery. Add a
regression test asserting the copied context is visible to terminate_downloads.

* Studio: tighten comments in lifespan_shutdown helper and tests

Condense the verbose docstrings/comments to the non-obvious rationale and drop
the self-evident ones. Verified comment-only with comment_tools.py check
--strip-docstrings (code signature unchanged); tests and sims still green.

* Studio: address review nits on lifespan shutdown helper

Annotate hw_module as types.ModuleType, reword the schedule/await comment
(inline fallback runs, it does not retry), and add a test for the public
loop.shutdown_default_executor() path (the 'Executor shutdown has been called'
RuntimeError that real uvicorn shutdown takes, distinct from the submit-time
'cannot schedule new futures after shutdown').

---------

Co-authored-by: Daniel Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2026-06-15 22:51:46 -07:00
..
datasets Studio: training survives a non-writable HF datasets cache (#6148) 2026-06-10 08:22:47 -07:00
hardware Guard Apple GPU power against negative counter-reset readings (#6235) 2026-06-12 01:56:05 -07:00
inference Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
models Studio: keep llama-server discovery from crashing on an access-denied candidate (#6268) 2026-06-12 11:20:07 -07:00
paths Studio: fix training output dir escaping outputs root for models on another drive (#6293) 2026-06-13 04:06:17 -07:00
.gitkeep root studio folder 2026-02-02 09:13:49 +00:00
__init__.py Final cleanup 2026-03-12 18:28:04 +00:00
_studio_release_build.py Add Studio web update banner and release version display (#5308) 2026-05-11 18:24:01 +04:00
api_errors.py Studio: improve OpenAI- and Anthropic-compatible API spec compliance (#6010) 2026-06-09 17:13:25 +02:00
cache_cleanup.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
cpu_threads.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
downsample.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
helper_precache_settings.py Studio: make Helper LLM startup pre-cache opt in (#6113) 2026-06-09 15:28:34 +02:00
host_policy.py Studio: enable stdio MCP servers on a loopback bind (#6295) 2026-06-15 03:02:32 +01:00
lifespan_shutdown.py Studio: make lifespan shutdown resilient to a dead default executor (#6307) 2026-06-15 22:51:46 -07:00
llama_cpp_freshness.py Studio: drop the on-disk freshness cache after a llama.cpp update (#6234) 2026-06-12 02:43:55 -07:00
llama_cpp_update.py Studio: show llama.cpp version and GPU specs in the About panel (#6261) 2026-06-15 15:25:08 +03:00
native_path_leases.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
studio_version.py Studio: llama.cpp update banner redesign, About tab license info, UI polish (#6196) 2026-06-11 09:27:34 -07:00
subprocess_compat.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
transformers_version.py Studio: fix Gemma-4-12B-it not loading (#6054) 2026-06-10 08:39:07 -07:00
update_status.py Studio: make code comments and docstrings more succinct (#6029) 2026-06-08 23:07:28 -07:00
upload_limits.py Formatting: ruff line-length 100, kwarg-spacing passes, drop blank after short local imports (#6079) 2026-06-08 04:24:13 -07:00
utils.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00
wheel_utils.py Reduce and tighten code comments and docstrings repo-wide (#6095) 2026-06-08 23:09:51 -07:00