Kill the group, and stop the probe blocking on a full pipe
The escalation was gated on the leader still being alive, so a leader that exits promptly on SIGTERM while a uv or python descendant ignores it skipped the SIGKILL entirely, and wait reaped only the leader. Proven with a descendant that traps TERM: pre-fix its heartbeat keeps ticking while the probe would be running, post-fix it stops. Signal the group unconditionally and drain it after the reap, since an unreaped leader is still a member of its own group. The probe started the backend on stdout=PIPE and read nothing until after the poll loop, so a backend logging more than the pipe buffer during import blocked before binding. Measured 65536 bytes here; a child emitting 200 KB never reaches its bind line, which would make backend_ok false for a healthy install. Write straight to the artefact file. Also trigger on studio/backend/requirements/**, where structlog is declared.
This commit is contained in:
parent
3100e90453
commit
80fc65dfe3
3 changed files with 36 additions and 11 deletions
25
.github/scripts/interrupt-install.sh
vendored
25
.github/scripts/interrupt-install.sh
vendored
|
|
@ -86,14 +86,31 @@ if [ "$killed" = "true" ]; then
|
|||
kill -0 "$PID" 2>/dev/null || break
|
||||
sleep 1
|
||||
done
|
||||
if kill -0 "$PID" 2>/dev/null; then
|
||||
echo "[interrupt] group survived SIGTERM; SIGKILL"
|
||||
kill -KILL -- -"$PID" 2>/dev/null || kill -KILL "$PID" 2>/dev/null || true
|
||||
fi
|
||||
# Unconditional, and to the GROUP. The leader can exit on SIGTERM while a uv or
|
||||
# python descendant ignores it or is mid-shutdown; `kill -0 "$PID"` then reported
|
||||
# the leader gone, this escalation was skipped, and `wait` reaped only the leader,
|
||||
# leaving that descendant free to finish the dependency pass while the probe ran.
|
||||
# Signalling an already-empty group is a no-op.
|
||||
echo "[interrupt] SIGKILL to process group -$PID"
|
||||
kill -KILL -- -"$PID" 2>/dev/null || kill -KILL "$PID" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
wait "$PID" 2>/dev/null
|
||||
rc=$?
|
||||
|
||||
# Only after the reap: an unreaped leader is still a member of its own group, so
|
||||
# polling the group before `wait` would report it alive forever. Do not let the
|
||||
# probe start while an installer process is still running.
|
||||
if [ "$killed" = "true" ]; then
|
||||
for _ in $(seq 1 "$KILL_GRACE"); do
|
||||
kill -0 -- -"$PID" 2>/dev/null || break
|
||||
kill -KILL -- -"$PID" 2>/dev/null || true
|
||||
sleep 1
|
||||
done
|
||||
if kill -0 -- -"$PID" 2>/dev/null; then
|
||||
echo "::warning::processes from installer group -$PID outlived SIGKILL"
|
||||
fi
|
||||
fi
|
||||
echo "[interrupt] installer exit=$rc reason=$reason killed=$killed"
|
||||
echo "[interrupt] last log lines:"
|
||||
tail -15 "$LOG" || true
|
||||
|
|
|
|||
17
.github/scripts/interrupted_install_probe.py
vendored
17
.github/scripts/interrupted_install_probe.py
vendored
|
|
@ -135,9 +135,16 @@ def main(argv: list[str]) -> int:
|
|||
popen_kw["start_new_session"] = True
|
||||
else:
|
||||
popen_kw["creationflags"] = getattr(subprocess, "CREATE_NEW_PROCESS_GROUP", 0)
|
||||
# Straight to the artefact file, never a PIPE: nothing reads that pipe until after
|
||||
# the polling loop, so a backend whose imports emit more than the OS pipe buffer
|
||||
# (64 KiB on Linux and macOS, a single page by default on Windows) blocks on write
|
||||
# BEFORE it binds the port. backend_ok, which this whole verdict pivots on, would
|
||||
# then be false for a perfectly good install.
|
||||
blog_path = out / "backend.log"
|
||||
blog_fh = blog_path.open("w", encoding = "utf-8", errors = "replace")
|
||||
proc = subprocess.Popen(
|
||||
[binp, "studio", "--api-only", "-H", "127.0.0.1", "-p", str(port)],
|
||||
stdout = subprocess.PIPE,
|
||||
stdout = blog_fh,
|
||||
stderr = subprocess.STDOUT,
|
||||
text = True,
|
||||
**popen_kw,
|
||||
|
|
@ -180,12 +187,8 @@ def main(argv: list[str]) -> int:
|
|||
proc.kill()
|
||||
|
||||
reap()
|
||||
try:
|
||||
blog = proc.communicate(timeout = 30)[0] or ""
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
blog = proc.communicate()[0] or ""
|
||||
(out / "backend.log").write_text(blog, encoding = "utf-8", errors = "replace")
|
||||
blog_fh.close()
|
||||
blog = blog_path.read_text(encoding = "utf-8", errors = "replace")
|
||||
say("backend_ok", backend_ok)
|
||||
|
||||
missing = ""
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue