_estimate_mtp_overhead_bytes is also reached for the separate-drafter spec modes
(draft-simple / draft-eagle3) through _user_draft_via_extras. Those modes load a
small distinct drafter with its own KV -- already counted in the draft KV +
weights -- and keep no duplicated full target context; only MTP runs a second
context over the target model's own KV geometry (llama.cpp ctx_tgt). Charging the
~main-KV-sized f16 copy there over-reserved by tens of GiB on an MLA model and
needlessly shrank the advertised context, the same under-advertising #6312 set
out to fix.
Thread mtp_keeps_target_ctx through _estimate_mtp_overhead_bytes (True for MTP,
False for separate-drafter modes) and derive _engaged_is_mtp at the fit call site
so the target copy is added only when the engaged mode is actually MTP. MLA + MTP
(GLM-5.2 / DeepSeek / Kimi) is unchanged, so the GLM-5.2 OOM fix is preserved;
non-MLA and the draft-simple / draft-eagle3 paths no longer pay the copy.
test_mtp_mla_target_ctx.py adds a case asserting the separate-drafter reserve
collapses to the draft KV (no target copy) while the default MTP path keeps it.
* Studio: reserve the duplicated MTP target KV context for MLA models
GLM-5.2 UD-IQ1_S advertised its native 1,048,576-token context, loaded, then
crashed cublasCreate on the first generation with "CUDA error: the resource
allocation failed" on a 2x B200 box. The model loaded fine; the decode OOMed.
Cause: when MTP speculative decoding is engaged, llama.cpp keeps a second full
copy of the target model's KV context for draft verification (ctx_tgt=yes in the
spec log), at f16. On an MLA model that copy is ~the main KV again -- for GLM-5.2
at 1M ctx llama.cpp sized it at ~97.5 GiB -- but the auto-fit reserve only
counted the tiny embedded draft head (~2 GiB), 46x too low. So weights (~202 GiB)
+ main KV (~83 GiB) + a 2 GiB reserve looked like it fit in 2x182 GiB, when the
real footprint with the ~97 GiB MTP copy is ~382 GiB and overruns the cards.
Disabling speculative decoding removed the copy and the same context ran fine.
_estimate_mtp_overhead_bytes now adds the duplicated target context (the main KV
re-estimated at f16) for MLA models, so auto-fit backs the context off (or selects
more GPUs) instead of advertising one that OOMs. It is gated strictly on MLA
(kv_lora_rank present), which is exactly the family that keeps the extra copy
(GLM-5.x, DeepSeek, Kimi-K2); non-MLA MTP (Qwen, Gemma) is byte-for-byte
unchanged. The reserve stays deterministic from GGUF dims, matching #6312.
test_mtp_mla_target_ctx.py covers it: the MLA reserve includes the f16 target
copy and dominates the draft head, the copy is f16 regardless of the main cache
type and scales with context, non-MLA embedded heads keep overhead == draft KV,
and _fit_context_to_vram on the GLM-5.2 / 2x B200 budget now returns a context
below the requested 1M where the old draft-only reserve kept the full 1M.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Studio: stub structlog/loggers in MTP-MLA test so it is import-order-independent
The new test imports core.inference.llama_cpp, which pulls in orchestrator ->
structlog. In the lightweight test env structlog is absent, so when this file is
collected before test_mtp_vram_budget.py (it sorts first) or run directly,
collection aborted with ModuleNotFoundError. Install the same loggers/structlog
(+ conditional httpx) stubs the sibling MTP tests use before the import, matching
the established per-file convention.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
