* Studio: surface the llama.cpp update affordance when MTP is disabled
When a model asks for MTP (auto on an MTP model, or forced mtp / mtp+ngram)
but it gets disabled, the load already degrades gracefully and serves without
speculative decoding. Until now the UI gave no hint why, or that an update
would fix it.
Record why MTP was dropped on the backend (spec_fallback_reason): the probe
found no mtp token (binary_no_mtp), the spawn aborted with an outdated-arch /
context-build error such as a prebuilt that predates the Gemma drafter
(binary_outdated), or the current build could not run it, e.g. a CUDA kernel
limit (runtime_error). Expose it in the inference status. In the chat
Speculative Decoding section, show a short note and, for the two update-fixable
reasons, an inline Update llama.cpp button that reuses the existing update flow.
A runtime_error gets the note without an update push, since a newer build may
not fix it.
Backend tests cover the reason being set / cleared. Frontend typechecks.
* Address review: tighten the update hint to genuinely outdated binaries
Reserve binary_outdated (which surfaces the Update llama.cpp affordance) for an
unknown-architecture abort, which proves the prebuilt predates the model;
classify the generic memory/context build failures as runtime_error, where an
update may not help. Frontend: only append the "Update llama.cpp to enable it"
sentence when an update is actually available, so the text never points at an
action the UI is not offering.
* Studio: enable MTP for sub-3B Gemma separate-drafter GGUFs
The sub-3B auto-drop to ngram-mod was tuned for an embedded draft head
(Qwen), whose per-token cost regresses below 3B. Gemma ships the head as a
separate root mtp-*.gguf drafter, a tiny standalone model that is cheap
enough to win below 3B: B200 Q4_K_XL bench, draft-mtp n=2 vs spec-off,
gemma-4-E2B (2B) = 1.21x (accept ~0.65) while ngram-mod is 1.00x.
Exempt a separate drafter from the sub-3B gate everywhere the threshold is
applied: the resolver (_mtp_too_small), the auto-fit VRAM reserve, the
drafter auto-download decision, and the reload-skip mirror via a
has_separate_drafter flag on _auto_mode_drops_mtp. Embedded sub-3B heads
(Qwen) still drop to ngram-mod. A drafter the binary cannot build (older
prebuilt, or a CUDA kernel limit) still aborts the spawn and the load
retries once without speculative decoding.
Adds the full Qwen3.5 + Gemma-4 (regular and QAT) auto/off/forced resolver
matrix, plus explicit sub-3B exemption tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Always compare the separate drafter in the reload-skip mirror
The sub-3B wrapper around the drafter compare could skip it when the drafter
was deleted out from under a running sub-3B server (detected None, stored set),
leaving a stale launch. The resolved-path compare is cheap and already handles
every case, so drop the _auto_mode_drops_mtp guard (and its now-unused imports)
and always compare when the mode can use a drafter and the user does not own
--spec-type. Addresses review feedback on #6191.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: gracefully disable MTP when the model has no head or drafter
Selecting MTP or MTP+Ngram in Speculative Decoding on a GGUF with no nextn
head and no separate drafter aborted the whole load. llama-server does not
no-op an empty draft-mtp request: it exits with 'failed to measure MTP
context memory: failed to create llama_context', surfaced to the user as a
generic 'llama-server failed to start. Check that the GGUF file is valid
and you have enough memory.'
Build-time fix in _build_speculative_flags: when a forced mtp / mtp+ngram
mode targets a model with no MTP head and no drafter (is_mtp_model is
False), default back instead of emitting draft-mtp. mtp falls back to
--spec-default; mtp+ngram keeps the ngram-mod half, which needs no head.
Real MTP models (embedded head or separate drafter), sub-3B MTP overrides,
and the auto path are unchanged.
Runtime hardening: the existing post-launch MTP retry only fired for
separate-file drafters (--model-draft in spec_flags), so an embedded-head
model that the binary cannot build still hard-failed. Gate the retry on the
spec block requesting MTP, recognise the embedded-head abort strings
('failed to measure MTP context memory', 'failed to create llama_context'),
and make the drafter name None-safe in the warning.
Tests: extend the resolver matrix (forced mtp / mtp+ngram on a non-MTP
model) and add two cases asserting the default-back emission.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Trim and tighten code comments and docstrings across studio/ Python. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.
* studio: add --spec-draft-n-max toggle for MTP speculative decoding
Surface llama-server's --spec-draft-n-max as a first-class
LoadRequest field so users can tune the MTP draft tree size from
the chat settings panel. Default behaviour is unchanged: when the
caller omits spec_draft_n_max, the existing platform defaults still
apply (6 on GPU, 3 on CPU/Mac).
Why this matters: on context-constrained loads the draft KV cache
competes with the target model's KV cache for VRAM. Lowering
spec_draft_n_max reduces that pressure, lets a larger user context
fit, and recovers throughput; raising it pays off when draft
acceptance is high enough to amortise the extra cache.
Backend
- LoadRequest gains an optional spec_draft_n_max: int (1..16).
- LlamaCppBackend.load_model accepts and persists the override on
self._spec_draft_n_max, used in place of the hardcoded 6/3 in the
MTP emit branch.
- LoadResponse and InferenceStatusResponse echo the active value
(None when the platform default is in effect) so the UI can
hydrate the input on refresh.
- _already_in_target_state and _request_matches_loaded_settings
compare spec_draft_n_max alongside speculative_type so a value
change triggers a reload rather than no-op'ing.
- strip_shadowing_flags now strips inherited --spec-* extras when
either speculative_type or spec_draft_n_max is in fields_set, so
an inherited --spec-draft-n-max cannot last-wins-override a fresh
request's first-class field.
Frontend
- LoadModelRequest, LoadModelResponse, InferenceStatusResponse
TypeScript shapes get spec_draft_n_max.
- chat-runtime-store gains specDraftNMax / loadedSpecDraftNMax and
a setter, hydrated from /v1/status and /v1/load.
- chat-settings-sheet renders a "Draft Tokens" numeric input
directly under the Speculative Decoding switch when that switch
is on. Toggling the switch off clears the override; the Reset
button restores the loaded value.
Tests
- Four new regression tests cover _already_in_target_state with
matching / mismatching / non-MTP / unset spec_draft_n_max.
- Existing test_llama_server_args.py and test_llama_cpp_mtp_detection.py
green: 141 passed locally.
* studio: add --spec-draft-p-min and --spec-draft-p-split to spec strip set
llama.cpp server documents --spec-draft-p-min (default 0.75, min draft
acceptance probability) and --spec-draft-p-split (default 0.10). Both
are first-class spec-decoding knobs that should travel with the rest
of the --spec-* family when an Apply re-sets speculative_type, so an
inherited override doesn't leak across a fresh load.
* studio/tests: skip MTP capability-probe tests on Windows
The four probe_server_capabilities tests use a bash stub written to
tmp_path/llama-server, which Windows' subprocess can't execute
directly (no shebang resolution, .bat / .cmd would be needed). Mark
them skipif sys.platform == 'win32' so the rest of the MTP plumbing
suite stays green on Windows CI. Unix coverage is unchanged.
* studio: lower MTP GPU default --spec-draft-n-max from 6 to 2
Bench on B200 / Qwen3.6-27B-MTP-GGUF UD-Q4_K_XL across five prompt
types (essay, code, story, math, science) with greedy temp=0:
prompt OFF n=1 n=2 n=3 n=6
essay 79.1 93.4 93.8 84.7 64.6
code 79.1 104.4 116.6 113.5 103.0
story 79.1 99.2 105.7 101.8 88.9
math 79.1 100.8 110.8 111.8 98.2
science 79.1 100.1 110.8 110.8 102.9
The previous hardcoded GPU default of 6 was 17% SLOWER than spec-off
on the essay prompt (64.6 vs 79.1 t/s) and 11-50% slower than n=2 on
the rest. n=2 wins on 4/5 prompts with a 1.18x-1.47x speedup vs OFF;
n=3 wins on the math prompt by a hair. n=6 collapses once acceptance
rate drops past n=3 -- wasted draft decode dominates the per-step
budget.
Matches the dataset README ("n_max=2 is the sweet spot for 36 of 42
quants"). Keeps CPU/Mac default at 3, which empirically tracks the
narrower ngram+MTP chained budget on those platforms.
Users who want the old behaviour can pass spec_draft_n_max in
LoadRequest (the toggle this PR also adds) or --spec-draft-n-max via
llama_extra_args.
* studio: skip MTP auto-promote on sub-2B models, backfill chat usage
Two MTP-visibility fixes uncovered while bisecting llama.cpp post-#22673
on Qwen3.6-27B-MTP-GGUF UD-Q4_K_XL on B200.
Size gate. Direct llama-server bench (no Studio measurement loop) at
n_predict=192 across 9 prompts shows MTP regresses vs spec-off on
sub-2B dense models because draft cost exceeds savings:
Qwen3.5-0.8B Q4_K_XL GPU: 452.0 OFF -> 283.4 t/s n=2 (0.63x)
CPU: 84.5 OFF -> 64.9 t/s n=3 (0.77x)
Qwen3.5-4B Q4_K_XL GPU: 241.0 OFF -> 258.2 t/s n=2 (1.07x)
Qwen3.5-9B Q4_K_XL GPU: 201.6 OFF -> 228.9 t/s n=2 (1.14x)
Qwen3.5-27B Q4_K_XL GPU: 78.8 OFF -> 113.6 t/s n=2 (1.44x)
Qwen3.6-27B Q4_K_XL GPU: 78.8 OFF -> 113.6 t/s n=2 (1.44x)
Qwen3.6-35B-A3B Q4 GPU: 192.3 OFF -> 223.2 t/s n=2 (1.16x)
The 2B inflection is sharp. Skip auto-promote to draft-mtp when the
identifier reports <2.0B params; users can still force via --spec-type
or the Speculative Decoding toggle. Mirror the gate in the
reload-skip check so a sub-2B reload-with-default does not bounce a
spec-off backend.
Chat-completions usage. llama-server's final SSE chunk emits both an
OpenAI-style usage block and a custom timings block. timings.predicted_n
is always populated, but usage.completion_tokens is zero on some
server builds. The Studio chat UI computes generation t/s from
meta.usage.completion_tokens / totalStreamTime, so a zero
completion_tokens makes the UI fall back to wall-clock time
(including SSE / proxy / template overhead) which dilutes MTP gains and
makes ON look the same as OFF.
Add _backfill_usage_from_timings: if usage.completion_tokens is missing
or zero AND timings has predicted_n/prompt_n, synthesize a complete
usage dict. Apply at the streaming metadata yield in
generate_chat_completion and at the three accumulator/yield sites in
generate_chat_completion_with_tools so per-iteration counts are not
silently lost across tool calls.
Tests cover both the gate (sub-2B skips, 2B+ promotes) and the
backfill (zero usage filled, real usage preserved, empty timings
passthrough).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: probe + emit legacy ngram-mod flags for pre-rename llama-server
llama.cpp upstream renamed the ngram-mod tuning knobs:
--draft-max -> --spec-ngram-mod-n-max (and --spec-draft-n-max)
--draft-min -> --spec-ngram-mod-n-min (and --spec-draft-n-min)
--spec-ngram-size-n -> --spec-ngram-mod-n-match
The new names are real flags on post-rename builds and stub removal
entries on the same builds (with description "argument has been
removed"). Pre-rename builds only carry the legacy names as real
flags. Studio was emitting the new names unconditionally, so a user
running a pre-rename llama-server (e.g. an older prebuilt or a
hand-installed binary) would see "unknown argument" errors when the
ngram-mod path engages, or silent drop of the ngram knobs.
Extend `probe_server_capabilities` to parse the help text into
per-flag description blocks and tell real flags apart from removal
stubs by the "argument has been removed" marker. Add three new probe
fields: `ngram_mod_flavor` ("new" / "legacy" / None),
`supports_ngram_mod`, and `spec_draft_n_max_flag` (the actual n_max
flag the binary accepts). Cached by (path, mtime) the same way as
`mtp_token`.
Add `_build_ngram_mod_flags(caps, ...)` that picks the right flag
set, returning [] when neither is usable so callers can drop ngram
chaining entirely on minimal binaries.
Wire both call sites to use the probe-driven flag set:
- CPU/Mac MTP comma-chain (--spec-type ngram-mod,draft-mtp) emits
legacy or new knobs as appropriate. If neither set is available,
degrade to MTP-only (warn but still engage spec).
- Standalone --spec-type ngram-mod branch uses the same helper.
Tests cover post-rename detection, legacy detection, removal-stub
discrimination, minimal-binary case, and all three branches of
`_build_ngram_mod_flags` plus custom n_match/n_min/n_max values.
Verified against three real binaries (Studio bundled 726704a, my
build of 45b455e HEAD, and the MTP merge baseline 2555826) all
correctly reporting ngram_mod_flavor=new.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: sub-3B MTP falls back to ngram-mod, not off
Earlier sub-2B gate disabled speculative decoding entirely for tiny
dense MTP models because the MTP draft head's per-token cost exceeds
the acceptance savings at that scale. The "fully off" fallback was
conservative -- ngram-mod has near-zero idle cost on diverse content
and consistently outperforms both off and draft-mtp at sub-3B.
Clean-methodology bench (each of 9 distinct prompts run once after
two unrelated warmup prompts so the ngram-mod hash pool is
realistically populated but never holds the exact deterministic
output we're about to measure):
Q4_K_XL on B200:
0.8B OFF=451 draft-mtp n=2=263 (0.58x) ngram-only=498 (1.10x)
2B OFF=377 draft-mtp n=2=308 (0.82x) ngram-only=369 (1.00x)
4B OFF=240 draft-mtp n=2=260 (1.08x) -- 4B+ wins with MTP
Q4_K_XL on x86 48 cores:
0.8B OFF= 80 chained n=2= 69 (0.86x) ngram-only= 95 (1.19x)
2B OFF= 62 chained n=2= 51 (0.83x) ngram-only= 63 (1.01x)
4B OFF= 31 chained n=2= 41 (1.33x)
Change:
- Raise the MTP-skip threshold from 2.0B to 3.0B (2B falls below it).
- When skipping the MTP head, fall back to --spec-type ngram-mod via
the probe-driven _build_ngram_mod_flags helper. Works on both
post-rename and pre-rename llama-server builds.
- If the binary advertises neither ngram-mod flavor, fall back to
spec-off (older binaries that don't support ngram-mod at all).
- Mirror the same fallback in _already_in_target_state so a sub-3B
reload-with-default does not bounce a ngram-mod backend.
Tests updated: monkeypatch probe_server_capabilities so the gate
behavior is deterministic regardless of which llama-server happens
to be on the host. +1 new test for the "binary has no ngram-mod
support" branch; renamed prior 2B/0.8B tests to reflect new semantics.
This generalizes the size gate to be probe-driven instead of a hard
"disable spec" branch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: 5-mode Speculative Decoding dropdown (Auto / MTP / Ngram / MTP+Ngram / Off)
Replace the Chat Settings Speculative Decoding on/off Switch with a 5-option
Select. Auto preserves today's platform-aware resolver (MTP on MTP GGUFs,
ngram-mod fallback for sub-3B, --spec-default for non-MTP). The other 3 modes
force the user's choice on BOTH GPU and CPU: MTP emits draft-mtp only (no
ngram chain on CPU), Ngram emits ngram-mod only, MTP+Ngram emits the
ngram-mod,draft-mtp chain on both platforms. Off is the existing fully-off
state, kept so the Switch's "disable" capability isn't lost.
Backend
- New module-level _canonicalize_spec_mode(value) maps any accepted input
(canonical, legacy "default" / "draft-mtp" / "ngram-mod" / "ngram-simple",
or comma-chained "ngram-mod,draft-mtp") onto one of auto / mtp / ngram /
mtp+ngram / off / ngram-simple / None. Lets external callers and old
persisted UI state round-trip without breaking.
- LlamaCppBackend grows a _requested_spec_mode field + requested_spec_mode
property storing the canonical UI mode the user requested. Status
responses round-trip this instead of the resolved internal flag, so the
dropdown restores the picked value after reload / refresh (Auto on a 27B
MTP GGUF resolves to draft-mtp internally but the dropdown stays on
"Auto").
- The resolver block in load_model is extracted into a unit-testable
_build_speculative_flags method. Forced MTP / MTP+Ngram on a sub-3B or
non-MTP GGUF logs a warning and engages anyway (user override > the
Auto-path sub-3B fallback).
- _already_in_target_state and routes/inference._request_matches_loaded_settings
now compare canonical-requested mode, dropping the old auto-promotion
mirror. spec_draft_n_max still gates on the resolved spec so Auto + a
changed n_max still bounces a reload.
Frontend
- chat-settings-sheet.tsx: Switch swapped for Select modeled on the KV
Cache Dtype Select. Items: Auto / MTP / Ngram / MTP+Ngram / Off. Draft
Tokens input only visible when speculativeType is "mtp" or "mtp+ngram".
- chat-runtime-store.ts: initial value flips from "default" to "auto".
- use-chat-model-runtime.ts normalizeSpeculativeType mirrors the backend
canonicaliser so persisted "default" / "draft-mtp" / "ngram-mod" / chain
values hydrate to the right dropdown option.
- types/api.ts: docs the canonical wire vocabulary.
Tests
- 53 new assertions in test_llama_cpp_mtp_detection.py: full
_canonicalize_spec_mode table, a 23-row resolver matrix across
(requested mode) x (GPU/CPU) x (model size class), plus n_max override,
user-extra-args precedence, requested-mode round-trip, and graceful
degrade on an outdated llama-server without an MTP token.
- 165 existing backend tests still green. 218 total in the MTP /
server-args / reload-inheritance suite.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: reset Speculative Decoding to Auto on model switch
When the user switches from model A to a different model B, clear the
runtime store's speculativeType + specDraftNMax (and their loaded*
shadows). The new load request then carries null, the backend
canonicalises that to "auto", and its platform-aware resolver runs
fresh for the new model.
Without this, a non-MTP model loaded with "Off" carried the Off choice
into a subsequent MTP load, suppressing MTP auto-promotion (and the
sub-3B ngram-mod fallback) until the user manually opened settings and
flipped the dropdown back to Auto. The clean-sweep deep probe caught
it as anomaly A-1.
The reset only fires when currentCheckpoint != modelId, so a
same-model reapply or forceReload still honours the user's current
spec choice. End-to-end probe on Qwen3.5-4B-GGUF (non-MTP, Off) ->
Qwen3.5-0.8B-MTP confirms: dropdown shows Auto, /api/inference/status
returns speculative_type=auto, studio.log shows the Auto sub-3B
fallback emitted --spec-type ngram-mod.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: engage draft-mtp on vision MTP GGUFs
The draft-mtp auto-promotion in LlamaCppBackend.load_model was gated on
not effective_is_vision, and the spec-emit branch repeated the same
guard. Every Unsloth -MTP GGUF repo ships an mmproj projector, so
effective_is_vision was always True for those repos and the MTP speedup
silently never engaged out of the box.
llama.cpp #22673 explicitly states MTP is compatible with vision input.
The bundled b9204 server happily loads both: a manual run with
--mmproj ... --spec-type draft-mtp --spec-draft-n-max 6 logs
"loaded multimodal model" followed by
"adding speculative implementation 'draft-mtp'".
Drop the vision gate from both sites and rewrite the matching short
circuit in _already_in_target_state so reload checks reach the auto
promotion path on vision MTP loads. Add three regression tests covering
vision MTP match (auto and default), and non MTP vision repo unaffected.
Verified on a B200 with unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL:
base decode 179.7 t/s vs MTP decode 253.8 t/s, draft acceptance 0.57,
1.41x speedup on a 255 token completion. mmproj still loads and image
input remains available.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: prefer Qwen3.5 -MTP GGUF variants in default model lists
With the vision gate dropped in the previous commit, draft-mtp now
auto-engages on -MTP GGUF repos out of the box. Swap the four Qwen3.5
recommended entries in DEFAULT_MODELS_GGUF and DEFAULT_MODELS_STANDARD
to their -MTP-GGUF counterparts so new users get the speedup by default:
unsloth/Qwen3.5-4B-GGUF -> unsloth/Qwen3.5-4B-MTP-GGUF
unsloth/Qwen3.5-9B-GGUF -> unsloth/Qwen3.5-9B-MTP-GGUF
unsloth/Qwen3.5-35B-A3B-GGUF -> unsloth/Qwen3.5-35B-A3B-MTP-GGUF
unsloth/Qwen3.5-0.8B-GGUF -> unsloth/Qwen3.5-0.8B-MTP-GGUF
All four HF repos exist (HEAD 200) and ship the same UD-Q4_K_XL quant
layout as the non-MTP variants. Non-Qwen3.5 entries are untouched.
* bump version to 2026.5.4
Picks up the studio MTP vision-gate fix and the Qwen3.5 -MTP default
swap in this PR.
* studio: prefer Qwen3.6-35B-A3B-MTP-GGUF in default model lists
Same rationale as the previous Qwen3.5 swap. The Qwen3.6 MTP variant
exists at unsloth/Qwen3.6-35B-A3B-MTP-GGUF (HF HEAD 200) and now
auto-engages draft-mtp out of the box with the gate fix.
* studio: drop --spec-draft-n-max from 6 to 3 for draft-mtp
n=6 is too greedy: on Qwen3.6 the draft has to guess 6 tokens ahead
and acceptance crashes to ~0.45, leaving only ~14% throughput gain.
PR ggml-org/llama.cpp#22673's author benched n=3 at ~0.72 acceptance
and 2 to 3x speedup on the same Qwen3.6 family, and the README sample
command uses n=2 or n=3. Match that.
CPU/Mac branch already uses n=3, so this aligns both paths.
* studio: set --spec-draft-n-max back to 6 for draft-mtp on GPU
Reverts the n=3 tuning. n=6 is the original default; user-side comparisons
hold the larger draft window steady so the toggle (next commit) is the
primary on/off lever.
* studio: add Speculative Decoding toggle under Max Tokens
Adds a top-level kill switch (panel-switch under Max Tokens, mirroring
Auto-Healing Tool Calls) that forces the /load request's
speculative_type to "off" when disabled. The backend "off" branch in
LlamaCppBackend.load_model skips both the draft-mtp auto-promotion and
the spec-emit branch, so neither --spec-type draft-mtp nor
--spec-default reaches llama-server.
Wiring:
- chat-runtime-store: new speculativeDecodingEnabled bool, default
true, persisted to localStorage under unsloth_speculative_decoding,
plus a setSpeculativeDecodingEnabled setter.
- chat-settings-sheet: SpeculativeDecodingToggle rendered immediately
beneath the Max Tokens slider for non-external models.
- use-chat-model-runtime: when speculativeDecodingEnabled is false,
override speculative_type to "off" in the loadModel call so the
switch wins over any pre-existing speculativeType state (including
the existing per-model toggle in Model Settings).
Verified end to end on unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL:
toggle ON emits --spec-type draft-mtp --spec-draft-n-max 6; toggle
OFF emits zero --spec-* flags on the same MTP GGUF.
* studio: relocate Speculative Decoding toggle into Model Settings
Move the toggle out from under Max Tokens and back into the Model
Settings section, directly beneath KV Cache Dtype, where the existing
Apply/Reset workflow already drives a reload on dirty. This way flipping
the switch in the UI actually picks up: the section becomes dirty,
Apply re-runs /load with the new speculative_type.
Drop the !currentModelIsMultimodal gate so vision MTP GGUFs can also
disable speculative decoding from the UI.
Switch the toggle's off-value from null to "off" so the backend's "off"
short-circuit fires for MTP models too (null normalises to None which
re-triggers the draft-mtp auto-promotion).
Tooltip now reads "Faster generation with 0% accuracy hit".
Remove the now-redundant speculativeDecodingEnabled bool + setter from
the runtime store and the load-time override in use-chat-model-runtime;
the toggle binds directly to speculativeType.
* studio: restore OOM/TIGHT badge on recommended GGUF rows
The recommended-list row passed vramStatus=null for any GGUF repo
because the existing useRecommendedModelVram hook reads safetensors
totals from HF model info, which GGUF-only repos do not expose. As a
result, an OOM Q-quant repo would render with only a "GGUF" badge and
no visual signal that nothing in it fits.
Add useGgufRecommendedFit: per repo, fetch the variant list via the
existing /api/models/gguf-variants endpoint, take the smallest
variant's size_bytes, and classify with the same 0.7*GPU + 0.7*RAM
thresholds as GgufVariantExpander. Session-scoped cache + in-flight
dedup so a repo is requested at most once.
Wire the result into the three GGUF row sites in pickers.tsx so OOM
and TIGHT badges show on the collapsed cards.
* Revert "studio: restore OOM/TIGHT badge on recommended GGUF rows"
This reverts commit 07793b1240df72b13e51d6dc15f63c4ee8c6cba9.
The new useGgufRecommendedFit hook was treating the symptom. PR #5561
identified the real root cause: useGpuInfo was calling /api/system
with plain fetch instead of authFetch, so the session-auth check
failed silently and gpu.available stayed false everywhere. With no
GPU info, every fit check (variant expander, recommended carousel)
fell back to "no signal" and dropped the OOM/TIGHT badges.
Reverting the over-engineered hook and applying the authFetch fix
in the next commit, which restores the existing badges with one line.
* chore: replace qwen suggested with MTP variant
* fix: restore GPU info auth for GGUF fit badges
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Studio: warn when llama.cpp prebuilt is too old for MTP
Layered on #5527. Adds a one-shot llama-server --help capability probe
so users get a clear signal when their prebuilt is missing MTP support,
plus a graceful fallback if they load an MTP GGUF against an outdated
binary.
What's surfaced:
1. Startup log + stderr line in main.py:lifespan() if MTP isn't
advertised:
WARNING: llama.cpp prebuilt is missing MTP support
(--spec-type mtp / draft-mtp). Run `unsloth studio update` to
refresh it. MTP GGUFs will load without speculative decoding.
2. Load-time graceful fallback in load_model's spec block: skip the
auto-emit and log a clear warning instead of letting llama-server
fail with an unknown-flag error.
3. /api/inference/status now returns llama_cpp_supports_mtp: bool so
the frontend can show a banner / popup.
Probe internals:
- Class-level cache keyed on (binary_path, mtime). One subprocess call
the first time, instant thereafter. Touching the binary (e.g. via
`unsloth studio update`) invalidates the cache automatically because
the mtime changes, so the new build is picked up without restarting
the server.
- Recognises both upstream naming forms: the original draft-mtp from
llama.cpp PR #22673 and the renamed mtp variant in later commits.
- Spec block uses whichever token the binary accepts so we emit the
right value regardless of which release the user has.
Tests:
- 6 new cases in test_llama_cpp_mtp_detection.py covering each probe
variant (draft-mtp, renamed mtp, pre-MTP build, missing binary,
mtime-based cache invalidation).
- Existing 38 MTP detection cases still pass; broader 188-test
regression suite (server args, reload inheritance, gguf metadata,
load progress, context fit, model validation) still green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: auto-enable MTP speculative decoding for MTP GGUFs
Detect Unsloth's MTP (multi-token-prediction) GGUFs and auto-emit the
right --spec-type draft-mtp flags for llama-server (llama.cpp PR
#22673), so users get the speedup without configuration.
Detection prefers the GGUF metadata field <arch>.nextn_predict_layers
(verified on Qwen3.6-27B-MTP-GGUF / qwen35 and Qwen3.6-35B-A3B-MTP-GGUF
/ qwen35moe). Falls back to a -MTP marker in the identifier / filename
so HF-mode loads can detect MTP from the repo name before the GGUF is
downloaded.
Flag presets follow the Unsloth MTP guide:
GPU: --spec-type draft-mtp --spec-draft-n-max 6
CPU/Mac: --spec-type draft-mtp --spec-draft-n-max 3 \
--spec-type ngram-mod --spec-ngram-mod-n-match 24 \
--spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 6
User overrides win: if the caller passes --spec-type / --spec-default
via unsloth run / unsloth studio run pass-through (or HTTP
llama_extra_args), the auto-emit steps aside so llama-server only sees
the user's flag. Scalar tuning knobs like --spec-draft-n-max compose
with the auto preset via llama-server's last-wins parsing.
_already_in_target_state mirrors the same promotion so a repeat /load
with unchanged settings against an MTP backend running draft-mtp
short-circuits cleanly instead of forcing a reload.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Daniel Han