ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Studio: surface the llama.cpp update affordance when MTP is disabled (#6192)

Browse source 
* Studio: surface the llama.cpp update affordance when MTP is disabled

When a model asks for MTP (auto on an MTP model, or forced mtp / mtp+ngram)
but it gets disabled, the load already degrades gracefully and serves without
speculative decoding. Until now the UI gave no hint why, or that an update
would fix it.

Record why MTP was dropped on the backend (spec_fallback_reason): the probe
found no mtp token (binary_no_mtp), the spawn aborted with an outdated-arch /
context-build error such as a prebuilt that predates the Gemma drafter
(binary_outdated), or the current build could not run it, e.g. a CUDA kernel
limit (runtime_error). Expose it in the inference status. In the chat
Speculative Decoding section, show a short note and, for the two update-fixable
reasons, an inline Update llama.cpp button that reuses the existing update flow.
A runtime_error gets the note without an update push, since a newer build may
not fix it.

Backend tests cover the reason being set / cleared. Frontend typechecks.

* Address review: tighten the update hint to genuinely outdated binaries

Reserve binary_outdated (which surfaces the Update llama.cpp affordance) for an
unknown-architecture abort, which proves the prebuilt predates the model;
classify the generic memory/context build failures as runtime_error, where an
update may not help. Frontend: only append the "Update llama.cpp to enable it"
sentence when an update is actually available, so the text never points at an
action the UI is not offering.
This commit is contained in:
Daniel Han 2026-06-11 06:10:17 -07:00 • committed by GitHub
commit a5d6e6928d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 157 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

24
studio/backend/core/inference/llama_cpp.py
View file

@ -676,6 +676,11 @@ class LlamaCppBackend:
# Separate MTP drafter launched with the current model; reload-dedup
# key so a drafter that appears next to the weights forces a reload.
self._mtp_draft_path: Optional[str] = None
# Why MTP was disabled on the last load that asked for it (auto on an
# MTP model, or forced mtp / mtp+ngram), else None. Drives the "update
# llama.cpp" hint in the UI. "binary_no_mtp" / "binary_outdated" ->
# a newer prebuilt would help; "runtime_error" -> it may not.
self._spec_fallback_reason: Optional[str] = None
self._hf_variant: Optional[str] = None
self._is_vision: bool = False
self._healthy = False
@ -794,6 +799,11 @@ class LlamaCppBackend:
def mtp_draft_path(self) -> Optional[str]:
return self._mtp_draft_path
@property
def spec_fallback_reason(self) -> Optional[str]:
"""Why MTP was disabled on the last MTP-requesting load, else None."""
return self._spec_fallback_reason
@property
def extra_args(self) -> Optional[List[str]]:
"""Extra llama-server flags from the last load (a copy). None =
@ -3594,8 +3604,13 @@ class LlamaCppBackend:
# failing (unknown arch / draft or context build); an
# unrelated crash (e.g. OOM) gets a neutral message.
_lo = "\n".join(self._stdout_lines).lower()
# Only an unknown architecture proves the prebuilt predates
# this MTP model (an update fixes it). The memory/context
# build failures are generic (VRAM / ctx pressure), where an
# update may not help, so classify those as runtime_error.
_arch_unsupported = "unknown model architecture" in _lo
if (
"unknown model architecture" in _lo
_arch_unsupported
or "failed to measure draft model memory" in _lo
or "failed to measure mtp context memory" in _lo
or "failed to create llama_context" in _lo
@ -3605,10 +3620,14 @@ class LlamaCppBackend:
"speculative decoding -- run `unsloth studio "
"update` for MTP"
)
self._spec_fallback_reason = (
"binary_outdated" if _arch_unsupported else "runtime_error"
)
else:
_retry_reason = (
"retrying without speculative decoding in case MTP is the cause"
)
self._spec_fallback_reason = "runtime_error"
_drafter = (
Path(launch_mtp_draft_path).name
if launch_mtp_draft_path
@ -3804,6 +3823,7 @@ class LlamaCppBackend:
# Reset; emit branches re-set on the resolved emission.
self._spec_draft_n_max = None
self._speculative_type = None
self._spec_fallback_reason = None
# Canonical UI-facing requested mode (legacy values mapped via
# _canonicalize_spec_mode).
@ -3853,6 +3873,7 @@ class LlamaCppBackend:
"run `unsloth studio update`. Loading without "
"speculative decoding."
)
self._spec_fallback_reason = "binary_no_mtp"
return False
draft_n_max = _resolved_draft_n_max()
n_max_flag = caps.get("spec_draft_n_max_flag") or "--spec-draft-n-max"
@ -4118,6 +4139,7 @@ class LlamaCppBackend:
self._gguf_path = None
self._hf_repo = None
self._mtp_draft_path = None
self._spec_fallback_reason = None
self._hf_variant = None
self._is_vision = False
self._is_audio = False