Kill the group, and stop the probe blocking on a full pipe

The escalation was gated on the leader still being alive, so a leader that exits
promptly on SIGTERM while a uv or python descendant ignores it skipped the
SIGKILL entirely, and wait reaped only the leader. Proven with a descendant that
traps TERM: pre-fix its heartbeat keeps ticking while the probe would be running,
post-fix it stops. Signal the group unconditionally and drain it after the reap,
since an unreaped leader is still a member of its own group.

The probe started the backend on stdout=PIPE and read nothing until after the
poll loop, so a backend logging more than the pipe buffer during import blocked
before binding. Measured 65536 bytes here; a child emitting 200 KB never reaches
its bind line, which would make backend_ok false for a healthy install. Write
straight to the artefact file.

Also trigger on studio/backend/requirements/**, where structlog is declared.
This commit is contained in:
danielhanchen 2026-07-28 19:24:53 +00:00
commit 80fc65dfe3
3 changed files with 36 additions and 11 deletions

View file

@ -86,14 +86,31 @@ if [ "$killed" = "true" ]; then
kill -0 "$PID" 2>/dev/null || break
sleep 1
done
if kill -0 "$PID" 2>/dev/null; then
echo "[interrupt] group survived SIGTERM; SIGKILL"
kill -KILL -- -"$PID" 2>/dev/null || kill -KILL "$PID" 2>/dev/null || true
fi
# Unconditional, and to the GROUP. The leader can exit on SIGTERM while a uv or
# python descendant ignores it or is mid-shutdown; `kill -0 "$PID"` then reported
# the leader gone, this escalation was skipped, and `wait` reaped only the leader,
# leaving that descendant free to finish the dependency pass while the probe ran.
# Signalling an already-empty group is a no-op.
echo "[interrupt] SIGKILL to process group -$PID"
kill -KILL -- -"$PID" 2>/dev/null || kill -KILL "$PID" 2>/dev/null || true
fi
wait "$PID" 2>/dev/null
rc=$?
# Only after the reap: an unreaped leader is still a member of its own group, so
# polling the group before `wait` would report it alive forever. Do not let the
# probe start while an installer process is still running.
if [ "$killed" = "true" ]; then
for _ in $(seq 1 "$KILL_GRACE"); do
kill -0 -- -"$PID" 2>/dev/null || break
kill -KILL -- -"$PID" 2>/dev/null || true
sleep 1
done
if kill -0 -- -"$PID" 2>/dev/null; then
echo "::warning::processes from installer group -$PID outlived SIGKILL"
fi
fi
echo "[interrupt] installer exit=$rc reason=$reason killed=$killed"
echo "[interrupt] last log lines:"
tail -15 "$LOG" || true

View file

@ -135,9 +135,16 @@ def main(argv: list[str]) -> int:
popen_kw["start_new_session"] = True
else:
popen_kw["creationflags"] = getattr(subprocess, "CREATE_NEW_PROCESS_GROUP", 0)
# Straight to the artefact file, never a PIPE: nothing reads that pipe until after
# the polling loop, so a backend whose imports emit more than the OS pipe buffer
# (64 KiB on Linux and macOS, a single page by default on Windows) blocks on write
# BEFORE it binds the port. backend_ok, which this whole verdict pivots on, would
# then be false for a perfectly good install.
blog_path = out / "backend.log"
blog_fh = blog_path.open("w", encoding = "utf-8", errors = "replace")
proc = subprocess.Popen(
[binp, "studio", "--api-only", "-H", "127.0.0.1", "-p", str(port)],
stdout = subprocess.PIPE,
stdout = blog_fh,
stderr = subprocess.STDOUT,
text = True,
**popen_kw,
@ -180,12 +187,8 @@ def main(argv: list[str]) -> int:
proc.kill()
reap()
try:
blog = proc.communicate(timeout = 30)[0] or ""
except subprocess.TimeoutExpired:
proc.kill()
blog = proc.communicate()[0] or ""
(out / "backend.log").write_text(blog, encoding = "utf-8", errors = "replace")
blog_fh.close()
blog = blog_path.read_text(encoding = "utf-8", errors = "replace")
say("backend_ok", backend_ok)
missing = ""