* studio: settle GPU VRAM after killing llama-server before the next reload
The NVIDIA driver reclaims a dead process's CUDA allocations
asynchronously after the kernel reaps the PID -- typically tens to
hundreds of milliseconds. Sampling `_get_gpu_free_memory` in that
window reads artificially low, which propagates into `_select_gpus`
/ `_fit_context_to_vram` and flips the layer-split toward `--fit on`
with more CPU-offloaded layers than steady-state would have required.
On a tight VRAM card the resulting mmap thrash + OOM matches the
Apply-reload kill path that bare-shell launches with the same flags
never hit (continues the lineage of #5161 / #5401 / #5427).
Adds `LlamaCppBackend._wait_for_vram_settle`: bounded poll of
`_get_gpu_free_memory` that returns as soon as two consecutive
samples agree per-GPU within `max(256 MiB, 2% of larger sample)`,
or `max_wait` (default 2 s) wall-clock elapses with probe time
included in the bound. Records `_last_kill_monotonic` inside
`_kill_process`'s `finally` block so the wait engages on both
in-process `load_model -> _kill_process -> load` and the frontend
chat-settings Apply path (`/unload` then `/load`). The call site
runs OUTSIDE the broad `self._lock` so concurrent `/unload`,
`/cancel`, `/status` are not blocked during the wait.
Short-circuits at zero cost on cold start (no kill recorded), stale
kill (older than 15 s, driver has already settled), CPU-only host
(probe returns empty), and probe exceptions (nvidia-smi gone away).
11 new unit tests in `test_llama_cpp_wait_for_vram_settle.py` cover:
cold-start zero cost, stale-kill skip, slow-probe deadline bound,
GPU index-set change, per-GPU stability with one draining card, the
2 % adaptive tolerance, _kill_process timestamp recording on real
kill vs no-op, and an `inspect.getsource` contract that pins the
call site to outside the Phase 3 lock and uses `_last_kill_monotonic`
so a future refactor can't silently regress any of these properties.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Michael Han <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2e1d0e2f19