09852ba18b* Studio: don't time out the live progress stream during pre-first-step prep The live progress SSE counts every 1s poll without a step update toward a 30-minute stall timeout, after which it emits an error event and ends the stream. But that counter also runs during the pre-first-step phase (model load + tokenizing the dataset), which is never reset because no step has happened yet. On a large dataset that prep can take well over 30 minutes, so the live view is torn down with an error while the run is perfectly healthy and still preparing -- the run then trains on in the background with the UI showing nothing, exactly the "no progress for hours" decoupling. Apply the stall timeout only once the stream has actually seen a live step. Before the first step the run is preparing and may legitimately emit no step for a long time; heartbeats still flow so the client stays connected and the worker's liveness still ends the loop when training finishes. A genuine post-step stall still times out. Extracted the threshold to a module constant so it can be tuned/tested. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Studio: seed seen_live_step from the resume point on reconnect Review follow-up: seen_live_step reset to False on every SSE request, so a client reconnecting past the first step (Last-Event-ID set, or the run already has step history) only receives heartbeats and never flips it true. A worker that hangs after step N would then never trip the stall timeout for that reconnected client. Initialize it from resume_from_step / existing step history so reconnects keep the post-step timeout behavior, while a genuine pre-first-step run still stays exempt. Added a reconnect regression test. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Tighten prep-phase progress timeout comments Condense the verbose explanatory comments and docstring on the prep-phase stall timeout exemption to shorter, clearer forms. Comment/whitespace only; verified no code changed via AST diff. No behaviour change. --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| assets | ||
| auth | ||
| core | ||
| hub | ||
| loggers | ||
| models | ||
| plugins | ||
| requirements | ||
| routes | ||
| state | ||
| storage | ||
| tests | ||
| utils | ||
| __init__.py | ||
| _platform_compat.py | ||
| cloudflare_tunnel.py | ||
| colab.py | ||
| main.py | ||
| run.py | ||
| startup_banner.py | ||