Recognise the Gemma 4 separate-drafter MTP family, auto-download the drafter with retry, fall back to n-gram with a clear reason when it cannot be resolved, and retry the download on reload. Gemma 3n (ships no drafter) and embedded-MTP models (Qwen) are unaffected.
Fixes#6406
* Studio: Auto disables MTP for MLA models (GLM-5.2 et al.); UNSLOTH_MLA_MTP_ENABLED to re-enable
Studio's Auto speculative mode promotes any embedded-MTP model >=3B to
--spec-type draft-mtp. For MLA models (GLM-5.2/DeepSeek/Kimi) that is a
regression: llama.cpp's MLA/DSA MTP path keeps a duplicated full target-KV
context and recomputes the sparse-attention indexer every draft step, so it
runs ~2x slower than no speculation (GLM-5.2 UD-IQ1_S bench: 27 vs 45 tok/s,
flat across draft depth 1..6 and 96-100% acceptance, on both prose and code).
vLLM/SGLang get a speedup from the same model, so this is a llama.cpp
implementation gap, not a model property.
Auto now drops embedded MTP for MLA models and falls back to ngram-mod (or
spec-off when the binary lacks ngram-mod), mirroring the existing sub-3B
fallback. The metadata separator is kv_lora_rank: it is present on MLA models
and absent on non-MLA embedded-MTP models (Qwen3.x-MTP), whose MTP module is
structurally identical but fast, so a "full layer" heuristic cannot tell them
apart. Qwen MTP, separate drafters (Gemma, --model-draft), and non-MTP models
are unchanged.
Explicit overrides still engage the slower MTP route: choosing MTP / MTP+Ngram
in Settings, or passing --spec-type in extra args. UNSLOTH_MLA_MTP_ENABLED=1
re-enables Auto promotion for MLA once the upstream path is optimized.
A new spec_fallback_reason value "mla_mtp_disabled" surfaces this as an
Auto-mode policy downgrade (not a binary/update problem), with a settings
banner that points users at the MTP override. It is deliberately kept out of
the "Update llama.cpp" affordance since updating does not help.
Tests: resolver-matrix rows for MLA->ngram-mod / MLA-no-ngram->off /
non-MLA-Qwen->draft-mtp / MLA-separate-drafter->draft-mtp /
non-MTP-MLA->default / forced mtp|mtp+ngram on MLA->draft-mtp / env flag;
kv_lora_rank metadata fixtures; and reload-skip coverage (Auto ngram-mod is
idempotent, forced mtp bounces a reload).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: reserve MTP draft VRAM in GGUF auto-fit
Auto-fit advertised a context (for example ~110k for the Qwen3.6-27B MTP
GGUF) that fit on paper but OOMed mid-generation or during tool calls once
MTP speculative decoding was active. The MTP draft path's VRAM was reserved
as a flat 5% of total VRAM, which tracks neither of the two real costs: the
MTP head keeps its own attention KV cache that grows with context, and the
speculative verification buffer grows with --spec-draft-n-max. On the
hybrid Mamba/attention Qwen3.6 models the main KV is small, so auto-fit
happily kept a near-native context while the draft path pushed the load
over budget at runtime.
Replace the flat fraction with a byte-accurate, context- and n_max-aware
reserve sized from GGUF dims: draft KV from nextn_predict_layers and the
attention dims at f16 (llama.cpp's MTP draft context uses f16 KV regardless
of the main cache type), plus a verify buffer per embedding-unit per draft
token. The reserve is evaluated per candidate context inside the fit binary
search and added to every pin/fit check, including the tensor-parallel
planner and its even-split decision. Coefficients were calibrated against
llama-server VRAM measurements on the Qwen3.6-27B MTP GGUF (RMS 14 MiB).
The flat fraction remains as a fallback when GGUF dims are unavailable, so
non-MTP loads are unchanged. The budget now also engages when the user wires
MTP through extra args (--spec-type draft-mtp, including chains), reads the
effective draft depth from --spec-draft-n-max or the legacy --draft-max with
extras taking precedence over the first-class field, reserves a separate
drafter's weights when supplied via --model-draft/--spec-draft-model/-md,
and mirrors _build_speculative_flags so it never reserves for MTP the launch
resolver will not emit (needs a head/drafter and a binary that supports
--spec-type mtp).
Adds tests/test_mtp_vram_budget.py.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: total-based VRAM budget + deterministic compute-graph buffer
Build on the byte-accurate MTP reserve with three changes that make the
GGUF auto-fit budget deterministic across architectures and recover usable
context, especially for MTP models on a single tight card.
1. Total-based budget. Cap GPU occupancy at a fraction of TOTAL VRAM rather
than a fraction of FREE VRAM, and raise the fraction from 0.90 to 0.95:
budget = free - (1 - 0.95) * total (per GPU, summed for a pool)
The reserve is now absolute (a fixed slice of the card) instead of
shrinking as the GPU fills, so a partly-used GPU keeps a constant cushion
for compute/CUDA/verify buffers instead of over-promising context and
spilling to CPU at runtime. _get_gpu_memory() reads memory.total alongside
memory.free; _fit_context_to_vram, _select_gpus and the load_model pool
loops thread the totals through. Multi-GPU layer-split pools
sum(free_i - 0.05*total_i); tensor mode reserves per device.
2. Deterministic compute-graph buffer. Replace the flat 5 GB/device tensor
reserve (a magic constant that over-reserved about 8x on a 27B model) with
_estimate_compute_buffer_bytes, sized from GGUF dims and the launch flags:
out = n_vocab * n_ubatch * 4 # vocab-width output buffer
act = 4 * n_embd * n_ubatch * 4 # activation scratch
pipeline_per_device = act + out * (n_parallel - 1)
tensor_per_device = 2*act + out * n_parallel
The buffer is context-independent and scales with --parallel (serving
slots), not with how the model is split across GPUs. It is now reserved in
BOTH multi-GPU paths (layer split folds one buffer into the pooled
footprint; tensor mode reserves it per device). The flat 5 GB stays only as
a fallback when vocab/embedding dims are unavailable. Calibrated against
llama-server measurements (parallel 1/2/4/8 give 36/492/1388/3220 MiB on a
single GPU; about 600 MiB/device tensor); the estimate is a small upper
bound.
3. GGUF parsing. Read vocab size (tokenizer tokens array length) and
feed_forward_length for the compute-buffer estimate.
Effect on the Qwen3.6-27B MTP Q6_K case (MTP on): a single 32 GB card at
about 31 GB free advertises f16 23k to 64k, q8_0 44k to 115k, q4_0 82k to
200k; 2x 24 GB tensor mode recovers the full 262k window for f16 (was about
134k). Validated on hardware: 1x 32 GB f16 at 64768 loads at 29.3 GB / 120
t/s; 2x 23 GB tensor f16 at 262144 loads at 22.2 GB/device / 98 t/s; both
within 0.4% of the estimate. Adds test_compute_buffer.py and updates the
KV/context-fit/MTP-budget tests for the 0.95 constant and the new budget.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten comments in the VRAM auto-fit changes
Condense the docstrings and inline comments added by this PR (internal backend
helpers): drop restated-signature docstrings, fold multi-line block comments to
one or two lines, and remove notes that just repeat the code. No behavior change
(AST-verified comment/docstring-only via comment_tools.py); the backend test
suite is unchanged and green.
* studio: address review findings in the VRAM auto-fit budget
Five fixes from a parallel-reviewer pass on this PR; all confirmed against the
real functions and covered by new tests.
- Tensor mode now honors the total-based VRAM cap. _plan_tensor_parallel took
total_by_idx and budgets each GPU at free - (1-frac)*total, mirroring the
layer-split paths; previously it fit against raw free and could spend the 5%
safety cushion on a partly-used multi-GPU box (reproduced ~3.3 GB over).
- Draft K and V cache types are parsed and accounted independently. A one-sided
override (e.g. --cache-type-k-draft q4_0, V left f16) no longer applies the
small quant to both axes and under-reserves the f16 axis. The embedded-head
formula sizes per axis; the separate-drafter path uses the heavier type so it
never under-reserves.
- The compute-graph buffer honors a user --ubatch / --ubatch-size / -ub override
(parsed and threaded into every _estimate_compute_buffer_bytes call and the
tensor planner); it previously always assumed the 512 default, under-reserving
up to ~8x at --ubatch 4096.
- GPU ranking uses the usable budget (free - (1-frac)*total) instead of raw free
in _select_gpus and both auto-context subset loops, so a more-used large card
no longer outranks a less-used small card that has more usable room.
Adds regression tests for each (tensor total cap, ubatch reserve scaling, split
K/V no-under-reserve, --ubatch parser, usable-ranking GPU selection). Full
targeted backend suite green (321 passed).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: gate tensor-parallel admission on usable VRAM budget
The tensor-parallel GPU admission filters still used raw free VRAM after
the total-based budget landed, an asymmetric fix: a partly-used large card
can clear the per-device compute-buffer reserve on raw free while its usable
budget (free - (1-frac)*total) does not, so the planner admitted it and the
even split could emit a near-zero weight slice for a GPU that should have
been excluded.
- _plan_tensor_parallel: admit GPUs by usable budget, not raw free (move the
_usable helper above the filter).
- load_model: admit the tensor set by _gpu_usable, and downgrade to layer
split when the pooled usable budget cannot hold weights plus per-device
compute buffers (the planner can only floor the context, not stop an
overcommitted launch).
Adds regression tests: planner drops a GPU whose usable budget is below the
reserve, and a source-level check that load_model admits on the usable
budget and carries the pooled-weight downgrade.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: size the MTP reserve for the user's overriding drafter
A user --model-draft passed in extra_args is appended last and wins at the
llama-server launch, but the VRAM budget preferred Studio's auto-detected
drafter (mtp_draft_path or extras), so a larger custom drafter was
under-reserved. Flip the precedence to extras-first, matching the draft-depth
(n_max) resolution two lines above. Adds a source-level regression test.
* studio: account for MTP reserve in tensor gate, restore 2-col GPU probe
Two issues found by re-review of the prior fix:
- The tensor-parallel capacity gate only checked the model weights against the
pooled budget, not the MTP reserve. A separate-drafter MTP load whose weights
fit but weights + drafter do not could still launch overcommitted in tensor
mode. Add the non-shrinkable MTP reserve (drafter weights + floor draft KV, or
the flat 2 GiB fallback when dims are unavailable) to the gate.
- The nvidia-smi probe was switched to a three-column query (index,free,total)
for the total-based budget but required exactly three columns, so a driver or
mock returning the legacy two-column "index,free" was dropped and the probe
fell through to the real GPUs. Accept two columns (total 0) and treat an
unknown total as the legacy free*fraction in _select_gpus.
Tests: tensor gate asserts the MTP term is included; _get_gpu_memory parses both
two- and three-column output; the existing two-column GPU-detection mocks pass
again.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: keep the VRAM cushion in tensor planning when GPU totals are unknown
_plan_tensor_parallel fell back to raw free VRAM when a GPU's total was
unavailable (a two-column nvidia-smi probe reporting total 0), while
_select_gpus and the load_model ranking both fall back to free*fraction. That
let tensor planning spend the 5% cushion the rest of the fit preserves and
over-advertise context in exactly that path. Align the fallback to
free*_CTX_FIT_VRAM_FRACTION. Updates the no-totals planner test expectations
(now free*frac) and adds a regression test that total 0 keeps the cushion.
* studio: honor LLAMA_ARG_* env overrides and HF draft flags in the VRAM budget
The budget parsed llama-server flags only from the request's extra_args, but the
child process inherits Studio's full environment (child_env_without_native_path_secret
copies os.environ), and llama-server honors LLAMA_ARG_* env vars for the same
options. So a service-level override the child acts on was invisible to the fit,
which could then advertise a context/GPU set that OOMs at load.
- _extra_args_n_ubatch: fall back to LLAMA_ARG_UBATCH (drives the compute buffer;
an unseen 4096 vs the 512 default under-reserves ~8x).
- _extra_args_mtp_draft_path: also recognize the HF draft-repo flags
(--spec-draft-hf/-hfd/-hfrd/--hf-repo-draft) and fall back to
LLAMA_ARG_SPEC_DRAFT_MODEL / LLAMA_ARG_SPEC_DRAFT_HF_REPO. An HF repo isn't a
local file so it can't be sized, but recognizing it routes to the flat reserve
instead of mis-sizing Studio's auto/embedded drafter.
- _extra_args_draft_cache_types: fall back to
LLAMA_ARG_SPEC_DRAFT_CACHE_TYPE_K/_V per axis.
CLI extra_args win over env (they are appended last at launch). Each parser takes
an injectable env for deterministic tests. Adds env-fallback and HF-flag tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: review polish - drop non-flag --ubatch, harden GPU probe, document buffer
Non-blocking items from a second review pass; no behavior change in the common path:
- _extra_args_n_ubatch: drop --ubatch; the binary only accepts --ubatch-size/-ub,
so parsing --ubatch implied support it does not have (it would over-reserve for a
launch that fails on the unknown flag).
- _get_gpu_memory: skip a malformed nvidia-smi line instead of letting one bad line
raise and drop the whole NVIDIA probe to the torch fallback.
- _estimate_compute_buffer_bytes: document that the per-slot output-buffer model
assumes a small n_outputs_max (chat decode); it would under-count for
embeddings / --logits-all / reranking, which Studio does not run on this path.
* studio: honor LLAMA_ARG_SPEC_TYPE when deciding the MTP reserve
_extra_args_requests_mtp only checked extra_args, but the child inherits Studio's
env and llama-server honors LLAMA_ARG_SPEC_TYPE. So a service-level
LLAMA_ARG_SPEC_TYPE=draft-mtp would run MTP while the fit skipped the draft
reserve and could advertise a context/GPU set that OOMs at load. Recognize the
env value (CLI still wins). Completes the env-override coverage alongside ubatch,
draft model, and draft cache types. Adds an env regression test.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: reserve VRAM for non-MTP model-based draft modes too
The draft reserve only engaged for MTP. A user passing a non-MTP model-based
draft mode (--spec-type draft-simple / draft-eagle3) with a --model-draft loads
a separate draft model whose weights + KV consume GPU memory, but the fit
reserved nothing and could OOM at load. Engage the existing drafter reserve for
those modes when extras (or LLAMA_ARG_SPEC_TYPE) name a drafter; ngram-* load no
model and are unaffected. Purely additive (reserves where there was none).
Adds parser + gate tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: floor quantized embedded MTP draft KV at f16; fix two test issues
Address PR review feedback (three findings):
1. Quantized embedded MTP draft KV was underpriced. The embedded head is a
single draft layer, so llama.cpp cannot amortize quantized-KV overhead over
many layers the way the main model does: a quantized draft KV (e.g.
--spec-draft-type-k q4_0) actually fits LESS context than f16, not more
(ggml-org/llama.cpp#24102, where a collaborator recommends f16 for the draft
KV). Pricing q4_0 at 0.5625 of an element (~28% of f16) under-reserved, so a
quantized override could advertise a context that shrinks or OOMs at load.
Floor the embedded draft KV bytes-per-element at f16 (quantized types priced
as f16, f32 still its full 4 bytes). The separate multi-layer drafter, where
quantization does amortize, keeps the user's real type.
2. test_load_model_reserves_for_non_mtp_draft_modes asserted an exact one-line
source substring that pre-commit black wrapped across lines, breaking CI.
Strip whitespace before matching so the check survives any line-wrapping.
3. test_compute_buffer.py installed a partial httpx stub via setdefault that, if
collected before test_kv_cache_estimation.py, leaked into sys.modules without
HTTPError/Response and could break the transformers introspection tier by
collection order. Adopt the sister file's pattern: only stub when real httpx
is absent, and include the full symbol set.
Updates the affected draft-KV tests to assert the f16 floor.
* studio: guard httpx stub in test_mtp_vram_budget too
test_mtp_vram_budget.py installed a partial httpx stub via setdefault that, like
test_compute_buffer.py before it, lacked HTTPError/Response and could leak into
sys.modules ahead of tests that need huggingface_hub/transformers, breaking the
introspection tier by collection order. Apply the same guard used by
test_kv_cache_estimation.py: only stub when real httpx is absent, with the full
symbol set.
* studio: per-device layer-split reserve, effective spec-type, drafter weights, KV restore
Address PR review feedback (four findings in the auto-fit budget):
A. Reserve the per-device layer-split overhead. A layer (pipeline) split allocates
a fixed per-device overhead (CUDA context + per-device compute scratch) on every
participating GPU, beyond the slot-scaling compute buffer that is conserved across
the split. Measured ~0.9 GB/device on the Qwen3.6-27B GGUF (b9625), independent of
--parallel: layer-split TOTAL VRAM grew +894 MiB (parallel=8) / +946 MiB
(parallel=1) per extra GPU, ~linear to +2.6 GB at 4 GPUs. The fit folded a single
compute buffer for all subset sizes, so a k-GPU layer split was short by
~(k-1)*0.9 GB and could pin a context that fits the pool on paper but OOMs a device.
Reserve (k-1) * _PIPELINE_PER_DEVICE_OVERHEAD_MIB per subset in the layer-split fit;
k=1 adds nothing, so single-GPU sizing (and the validated benchmark rows) is unchanged.
B. Track the effective --spec-type. _extra_args_requests_mtp returned true on the
first MTP-ish --spec-type and consulted LLAMA_ARG_SPEC_TYPE even when a CLI
--spec-type was present, contrary to llama.cpp (last CLI value wins; a CLI flag
overrides the env). So `--spec-type draft-mtp --spec-type ngram-mod` or a non-MTP
CLI value with a stale MTP env over-reserved a drafter the launch won't load
(shrinking context / selecting extra GPUs). Route both detectors through a new
_effective_spec_type helper.
C. Keep known drafter weights in the fallback reserve. When a separate drafter's KV
metadata can't be sized, _estimate_mtp_overhead_bytes returned None and discarded
the drafter's known weight bytes, falling back to the flat 5% reserve; a drafter
larger than that cushion could launch over budget and OOM. Reserve the known
weights even when KV sizing fails (None only when nothing is known).
D. Restore quantized KV on tensor->layer-split downgrade. The tensor attempt drops a
quantized KV cache (tensor mode aborts on it). When the GPU-count or capacity gate
then downgrades to layer split -- which supports quantized KV -- the dropped type
was lost and the launch used f16, using more VRAM and shrinking context. Remember
the dropped type and restore it on downgrade (the launch re-emits it from the var).
Adds regression tests for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: per-device overhead in GPU pin, skip CPU draft, gate env spec-type
Address PR review feedback (three follow-up findings):
F1. Reserve the per-device layer-split overhead in the pin path too. The earlier
per-device reserve was added to the auto-context fit loops but not to
_select_gpus, which the explicit-ctx and file-size-only paths use to PIN GPUs
with -ngl -1 (no --fit fallback). A 2+ GPU pin within ~1 GiB/extra-GPU of the
budget could OOM a device at load. Add a per_device_overhead_bytes arg to
_select_gpus so a k-GPU pin must hold model + (k-1)*overhead; pass the pipeline
overhead at both pin call sites. Single-GPU pins are unchanged.
F2. Don't charge a CPU-offloaded drafter against the GPU budget. A user passing
--spec-draft-ngl 0 or --spec-draft-device none/cpu keeps the separate draft
model's weights + KV on CPU, but the budget still charged the full drafter GGUF
size, auto-reducing context or downgrading GPU selection. Detect the CPU-offload
flags and drop the separate drafter (and its flat fallback) from the budget; an
embedded head follows the main -ngl and is unaffected.
F3. Consult LLAMA_ARG_SPEC_TYPE only when it can reach the child. llama-server's CLI
args override env, and _build_speculative_flags emits a --spec-type/--spec-default
for every UI mode except "off". So a stale MTP env on a non-MTP model (auto mode)
made the fit reserve MTP that the emitted --spec-default disables, shrinking
context / picking extra GPUs. Gate the env consult on "no user --spec-type and UI
mode off"; the MTP-model auto path still engages via Studio's own detection.
Adds regression tests for each.
* studio: drafter budget precedence and --spec-default in effective spec-type
Two spec-precedence fixes surfaced by an independent multi-reviewer pass:
R3. Size the drafter the launch actually loads. _mtp_draft_for_budget consulted
LLAMA_ARG_SPEC_DRAFT_MODEL (via _extra_args_mtp_draft_path's env fallback)
before Studio's resolved mtp_draft_path, but _build_speculative_flags emits
--model-draft mtp_draft_path, which overrides the env at launch. With a stale
(smaller) env drafter, the budget under-reserved and could OOM. Order the
budget by what actually launches: CLI extras --model-draft (appended last,
wins), then Studio's emitted mtp_draft_path (when MTP engages and the user
doesn't own --spec-type), then the env drafter.
R4. Treat --spec-default as a CLI spec override in _effective_spec_type. It only
recognized --spec-type, so extras=["--spec-default"] with LLAMA_ARG_SPEC_TYPE=
draft-mtp fell through to the env and over-reserved MTP, even though the CLI
--spec-default overrides the env to a non-MTP default. Recognize it as a CLI
spec flag (resolves to "default", non-MTP) that suppresses the env fallback.
Adds regression tests for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: refine MTP draft reserve (parallel slots, last-wins, KV cushion, ranking)
Address PR review feedback (five follow-up findings, all edges of this session's
earlier MTP/auto-fit changes):
G1. Price the separate drafter's KV per --parallel slot. _mtp_draft_kv_bytes called
the drafter's _estimate_kv_cache_bytes with the default n_parallel=1, but the
drafter is served under the main model's slot count; a sliding-window drafter
(Gemma) grows KV per slot and was under-reserved. Thread n_parallel through the
draft KV / overhead estimate and the fit closure.
G2. Honor last-wins for the draft-offload flags. _extra_args_draft_offloaded_to_cpu
returned True on the first CPU value, so --spec-draft-ngl 0 --spec-draft-ngl -1
(final = GPU) wrongly dropped the drafter reserve while the server kept it on
GPU -> OOM. Decide on the final value of each flag only.
G3. Keep the flat cushion when only the drafter weights could be sized. The weights
fallback installs mtp_overhead_fn, which made callers drop the flat MTP reserve,
leaving the still-unsized draft KV with no cushion. Keep the flat fraction on in
that weights-only case, on top of the byte-accurate weights.
G4. Rank auto/cap GPU subsets by the active budget fraction. The ranking used a
hard-coded 0.95 while the fit tests _pin_fraction (lowered by the flat MTP
reserve); on mixed-total GPUs that could order subsets differently and pick a
worse plan. Rank with the same fraction the fit uses.
G5. Keep the embedded-head flat reserve under a draft CPU-offload flag. F2's
not-_draft_on_cpu guard also dropped the reserve for an embedded MTP head, which
is part of the main model and stays on GPU regardless of --spec-draft-ngl. Only
suppress the flat reserve for a CPU-offloaded separate drafter (no embedded head).
Adds regression tests for each.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: keep GPU on non-integer total, keep tensor flat reserve for weights-only
Two review findings:
- _get_gpu_memory dropped a whole GPU when nvidia-smi reported a non-integer
memory.total ("N/A" on some drivers / MIG / vGPU): index, free and total were
parsed in one try/except that skipped the line on any ValueError, so the GPU
vanished from the probe and the load could silently spill to CPU. Parse index
and free (required) first, then total separately, defaulting to 0 (the fit then
uses the free*frac path for that GPU). Adds N/A and bad-free test cases.
- Tensor planning skipped the flat MTP reserve for a weights-only drafter (file
size known, KV unsizable): the capacity gate used the byte floor whenever
mtp_overhead_fn was set, so it reserved only the drafter weights and no draft
KV. Tensor mode has no --fit valve, so that could overcommit and OOM. Keep the
flat reserve (never below the byte floor) in the weights-only case too, mirroring
the layer-split _mtp_kv_unsized handling. Adds a regression test.
(A third suggestion -- fold --batch-size into the compute-buffer reserve -- was
checked on hardware and declined: -b 8192 -ub 512 used identical VRAM to the
default at -c 64000, so the logical batch does not size the graph buffer; the
estimate correctly uses the physical micro-batch.)
* studio: budget the main KV from LLAMA_ARG_CACHE_TYPE env when Studio emits none
The child inherits LLAMA_ARG_CACHE_TYPE_K / LLAMA_ARG_CACHE_TYPE_V, but Studio
emits --cache-type-k/-v only when the param or extras set the type. When neither
does, a heavier env type (f32) reaches the child while the auto-fit budget
assumed the f16 default, under-reserving the main KV and risking OOM at the
advertised context. This is the one main-KV axis that lacked the env-aware
handling the other axes already have (spec-type, draft model, draft cache type,
ubatch).
load_model now adopts the heavier of the two env types when it exceeds f16 (only
f32 does), and the launch re-emits it so child and budget stay byte-consistent.
Quantized env types are <= f16 and remain safely over-reserved by the default,
so they are left untouched (no change). A single value is used because the
budget's KV estimate has one cache_type_kv knob, matching parse_cache_override's
existing key/value collapse.
Adds _env_main_cache_type_for_budget plus regression tests covering f32 adoption,
the K/V heavier-of collapse, quantized/unknown no-ops, and the load_model source
precedence.
* studio: budget tensor parallel when LLAMA_ARG_SPLIT_MODE env selects it
Studio emits --split-mode tensor only on its tensor branch; the default
layer-split path emits nothing and resolve_tensor_parallel consults only extras.
The child inherits LLAMA_ARG_SPLIT_MODE, so a tensor env on a layer-split plan
silently runs the child tensor-parallel (heavier per-device compute buffer)
while the budget reserved only the layer-split per-device overhead, under-
reserving on multi-GPU.
load_model now flips the plan to tensor when extras do not set a split mode and
the env selects tensor, so Studio plans, reserves, and emits tensor consistently.
The flip is one-directional (guarded on not tensor_parallel and no extras
split-mode) so an existing tensor plan is never downgraded and extras keep
precedence. Other env modes (layer/row/none) are not a runtime-heavier surprise
and are left untouched.
Adds _env_split_mode_is_tensor plus unit and load_model source-level tests.
* studio: reconcile inherited llama.cpp env with the budgeted launch decision
Addresses a review pass over the VRAM auto-fit work. The budget now sizes the
right amount, but the child process inherits LLAMA_ARG_* env (see
child_env_without_native_path_secret), and a few axes could still run the child
in a mode Studio neither chose nor budgeted.
Mixed known/unknown GPU totals over-advertised the pooled layer-split budget.
_pool_budget_mib pooled free and total separately, so an unknown-total GPU
(MIG/vGPU/N/A) contributed its full free with no cushion when mixed with
known-total GPUs (~(1-frac)*free over-advertise, about 500 MiB in a two-GPU
case). It now sums each GPU's own usable budget, and the layer-split fit calls
take that as an absolute budget (budget_frac=1.0, total_mib=None) so the fit and
the footprint check agree. All-known-total pools are unchanged.
LLAMA_ARG_SPLIT_MODE=tensor survived a tensor-to-layer downgrade. The downgrade
only stripped CLI extras, so the inherited env still ran the child tensor while
Studio budgeted layer split. When the final decision is layer split, a non-layer
inherited split mode (and any paired LLAMA_ARG_TENSOR_SPLIT) is now cleared from
the child env.
Inherited quantized LLAMA_ARG_CACHE_TYPE_K/_V crashed tensor mode. Tensor mode
aborts on a quantized KV cache; Studio drops a quantized cache_type_kv for the
tensor attempt but the inherited env reached the child anyway. When the final
decision is tensor split, a quantized cache-type env is now cleared so the child
uses the tensor-safe default that was budgeted.
Env-derived cache budget no longer mutates the emitted launch flags. An env-only
main KV type now informs the budget only; it is not re-emitted, so an asymmetric
K=f32,V=f16 env reaches the child as set instead of being rewritten to symmetric
--cache-type-k/-v f32.
Adds source-level regression tests for all four and confirms the documented
single-GPU/tensor/pipeline numbers are byte-identical before and after.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: tighten comments in the VRAM auto-fit code
Compress the verbose docstrings and inline comments added by this work to
succinct 2-4 line versions, drop restated/obvious ones, and cut duplicated
rationale across the two tensor-downgrade branches. Keeps the non-obvious intent
(env-inheritance precedence, the #24102 embedded-draft floor, the per-device
overhead and pool-budget rationale) while removing roughly 120 lines of comment
text from llama_cpp.py. Also trims the few longest test-comment blocks; concise
per-test scenario notes are left intact.
No logic change: verified with comment_tools.py check --strip-docstrings (code-
only signature unchanged vs the prior commit) and the full backend suite still
passes (824).
* studio: lock in env-drafter engagement for the separate-draft reserve
A review suggested an env-provided LLAMA_ARG_SPEC_DRAFT_MODEL would skip the
draft reserve and OOM. It does not: the gate's _extra_args_mtp_draft_path(extra_args)
call defaults env=None, which consults os.environ, so an env-only drafter still
sets _user_draft_via_extras and is sized via _env_draft_for_budget. Add a source
guard that the gate keeps the env-inclusive form (not extras-only env={}) and a
behavioral test mirroring the reviewed scenario, so a future cleanup can't
regress it. No production change.
* studio: carry the unsized MTP reserve and env split/offload into tensor planning
Addresses a review pass over the multi-GPU and env-inheritance paths.
Tensor planner dropped the unsized draft-KV cushion. When a separate drafter has
known weights but unreadable KV metadata, _plan_tensor_parallel receives a
non-None weights-only mtp_overhead_fn and applied the flat 2 GiB reserve only for
the no-fn case, so its binary search spent the unsized-KV cushion on context and
over-advertised. Add mtp_flat_reserve_bytes (subtracted from the pooled budget and
the even-split check), and pass it from load_model whenever _mtp_kv_unsized. The
layer path and the tensor pre-gate already kept this cushion.
Stale LLAMA_ARG_TENSOR_SPLIT survived in tensor mode. When the planner picks an
even split it emits no --tensor-split, so an inherited tensor-split env reached the
child and overrode the budgeted split. The layer downgrade branch cleared it; the
tensor branch now does too.
Env-only draft CPU offload was ignored. _extra_args_draft_offloaded_to_cpu checked
extras but not LLAMA_ARG_N_GPU_LAYERS_DRAFT, so an env-offloaded drafter was still
charged GPU budget and under-advertised context. It now consults that env (the
device flag has no env), called with env=os.environ.
Layer-split compute buffer had no fallback when GGUF dims are missing. The estimate
returns 0 then, so the layer path folded no buffer while the tensor path falls back
to the flat reserve. Use the flat reserve for the layer path too (a safe upper
bound, since the tensor buffer >= the layer one).
All four are gated on conditions the documented benchmarks don't hit; the
single-GPU/tensor/pipeline reconfirm numbers are byte-identical, and the full
backend suite passes (830) with regression tests for each fix.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: share the env-aware tensor decision across load and dedup matchers
A review pass found the inherited-LLAMA_ARG_SPLIT_MODE=tensor flip lived only
in load_model, so the two duplicate-load matchers disagreed with it.
Consolidate the decision into _effective_tensor_parallel (extras + toggle, then
flip on when extras set no split mode and the child inherits a tensor split
env). load_model, the backend matcher (_already_in_target_state) and the route
matcher (_request_matches_loaded_settings) now all call it. Before, an env-driven
tensor server compared against resolve_tensor_parallel (env-blind) in both
matchers, so a follow-up load that should dedup was seen as a mismatch and the
healthy server was needlessly killed and reloaded.
Also finish the tensor cache-type handling: when the tensor attempt drops a
quantized KV it now re-adopts a heavier inherited env cache type (f32) for the
budget, mirroring the initial adoption; and the two layer-split downgrades clear
_cache_type_from_env so the restored quantized type is actually re-emitted rather
than left to a stale inherited env.
All gated on inherited env the documented benchmarks don't set; the single-GPU,
tensor and pipeline reconfirm numbers are byte-identical, and the full backend
suite passes (832) with unit + source regression tests for the shared helper and
the route matcher.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: complete the env-aware tensor/spec handling across all paths
A second review pass found the env-aware tensor/MTP handling was applied
asymmetrically: some paths inherited LLAMA_ARG_* env, others didn't. Three real
follow-ups, plus a small consolidation so the env semantics live in one place.
1. Tensor fallback ignored the inherited tensor env. load_with_tensor_fallback
computed its retry gate with the env-blind resolve_tensor_parallel, so an
env-only tensor load (toggle off, no --split-mode extra) that crashed on a
tensor-incompatible GGUF re-raised instead of retrying layer split. It now
uses the env-aware decision; and since the inherited env would otherwise
re-engage tensor on the retry (CLI args persist, the env does too), the retry
forces --split-mode layer (CLI wins over env) so it can't re-crash.
2. Duplicate-load matchers looped reloads after a tensor->layer downgrade. Both
matchers compared the env-expanded tensor decision against the loaded server,
but load_model may downgrade tensor to layer (capacity/buffer) and scrub the
child env. The still-set parent env then made every identical request look
like a mismatch, killing and reloading a healthy layer server. Add
_tensor_parallel_matches_loaded, which only lets an inherited tensor env raise
a match against a server that actually launched tensor; a downgraded server
matches the same request (an identical load would downgrade the same way).
3. MTP binary-capability fallback leaked an inherited LLAMA_ARG_SPEC_TYPE. When
the binary lacks MTP, _emit_mtp degraded but emitted no spec flag, so an
inherited LLAMA_ARG_SPEC_TYPE=draft-mtp still reached the child and attempted
MTP the gate had budgeted off. It now emits --spec-default (CLI wins over env)
like the sibling no-head / non-MTP fallbacks.
Consolidation: moved _env_split_mode_is_tensor / _effective_tensor_parallel into
llama_server_args.py (with the new _tensor_parallel_matches_loaded) so the
lightweight tensor_fallback module can share them without importing llama_cpp;
llama_cpp re-exports them for back-compat.
All gated on inherited env the documented benchmarks don't set; the single-GPU,
tensor and pipeline reconfirm numbers are byte-identical, and the full backend
suite passes (883) with regression tests for each fix.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: budget the heavier axis of asymmetric --cache-type-k/-v extras
A review pass found the explicit-extras counterpart of the env cache-type fix.
load_model adopts the heavier inherited LLAMA_ARG_CACHE_TYPE_K/_V env for the
reserve, but the explicit-extras path used resolve_cache_type_kv, which collapses
both axes to one last-wins value. So extras such as
--cache-type-k f32 --cache-type-v f16 (lighter axis last) budgeted f16 for both
axes while the child allocates f32 on K, over-advertising context and
re-opening the OOM path this PR closes.
Add parse_cache_override_per_axis (keeps the K/V last-wins values apart) and
_extra_args_main_cache_type_for_budget (the heavier of the two by bytes/elem),
and budget from it. The user's extras are appended last and win per axis at the
child, so this only raises the reserve; the emitted command and the asymmetric
child cache are unchanged, and the common single-axis / symmetric cases resolve
to the same type as before.
Reconfirm numbers (single-GPU table, tensor, pipeline) are byte-identical, and
the full backend suite passes (892) with per-axis parser and heavier-axis budget
regression tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: fix tensor-safety masking and strip inherited HF drafter selectors
A review pass found two more env/extras edge cases on the speculative and tensor
cache paths.
Tensor-safety could miss a quantized axis. The previous change budgets the
heavier-by-bytes cache type, but that masks a quantized axis paired with a
heavier one: --cache-type-k f16 --cache-type-v q4_0 resolves to f16, so the
tensor-safety block did not fire and the q4_0 axis survived into tensor mode,
which aborts on quantized KV. Test each explicit --cache-type-k/-v axis (not just
the budget type) so any quantized axis drops the cache for the tensor attempt.
Inherited HF drafter selectors were not stripped. _extra_args_mtp_draft_path
treats --spec-draft-hf / -hfd / -hfrd / --hf-repo-draft as drafter selectors, but
_SPEC_FLAGS only stripped the local --model-draft selectors, so on an inherited-
extras Apply a stale HF drafter survived and last-wins-overrode Studio's
re-derived spec choice. Add the HF aliases to _SPEC_FLAGS. The per-drafter tuning
knobs (--spec-draft-type-*, -ngld, --spec-draft-device) are intentionally left in
place: the VRAM budget reads them via the same parsers the child honors, so they
stay consistent on inherit, and stripping them would silently move a CPU-offloaded
drafter back onto the GPU.
A third flagged item -- that the HF draft env var should be LLAMA_ARG_HFD_REPO --
was a false positive from a stale manpage; the bundled binary's common/arg.cpp
sets LLAMA_ARG_SPEC_DRAFT_HF_REPO for --spec-draft-hf, which the code already
uses, so it is left unchanged.
Reconfirm numbers (single-GPU table, tensor, pipeline) are byte-identical, and
the full backend suite passes (899) with regression tests for both fixes.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: preserve asymmetric cache on tensor downgrade and skip CPU-drafter reserve
A review pass found two more tensor-path edges, one a regression from the
per-axis cache change.
Tensor-to-layer downgrade collapsed asymmetric cache extras. The per-axis
tensor-safety check strips an asymmetric --cache-type-k/-v (tensor rejects
quantized KV), but the downgrade restored only the scalar heavier type, so a
layer fallback silently rewrote --cache-type-k q4_0 --cache-type-v f16 to
symmetric f16/f16 even though layer split supports the original. Save the
original extras before the tensor strip and restore them verbatim (minus the
user --split-mode) on both downgrade points; the budget still uses the heavier
scalar, the child gets the real asymmetric cache. Before the per-axis change this
case happened to survive (last-wins was f16, untouched), so this restores that.
Tensor mode reserved GPU VRAM for a CPU-offloaded drafter. The layer path drops
the flat MTP reserve when the only drafter is a separate CPU one with no embedded
head, but the tensor capacity gate and planner still charged it, under-advertising
context. Gate the tensor reserve on the same condition via _mtp_reserves_gpu.
Reconfirm numbers (single-GPU table, tensor, pipeline) are byte-identical (both
fixes are gated on conditions the benchmarks don't hit), and the full backend
suite passes (901) with regression tests for each.
* studio: drop now-unused llama_server_args imports from llama_cpp
The refactor re-pointed load_model and the matchers off resolve_tensor_parallel /
resolve_cache_type_kv and moved the env split-mode helper into llama_server_args,
leaving those three names imported but unused in llama_cpp. The repo's import-hoist
safety-net lint blocks that, so drop them; the env split-mode test now imports
_env_split_mode_is_tensor from its real home (llama_server_args).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix the libaray path for probe_server_capabilities()
even when running something as simple as `./llama-server --help`,
the binary still requires correct LD_LIBRARY_PATH to work - or it
returns merely an "error while loading shared libraries":
"libllama-server-impl.so: cannot open shared object file: No such file or directory"
For a local installation with no LD_LIBRARY_PATH specifically set,
the probe_server_capabilities() run of `./llana-server --help` should
share the same libaray resolution logic as start_llama_server().
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Adjust and readd comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* fix(studio): Windows GGUF cancel hang + CPU spinlock overhead (#5692)
Two fixes for Windows-native GGUF inference via llama-server:
**Issue 1 — GPU/CUDA Hang on Stream Cancellation:**
- Add `Connection: close` header to all httpx requests proxying to
llama-server, preventing Keep-Alive from masking downstream socket
closure.
- Introduce `_await_disconnect_then_close` background watcher that
polls `request.is_disconnected()` every 100ms and calls
`resp.aclose()` immediately when the client disconnects. This runs
alongside the existing cancel-POST watcher and covers client aborts
that never reach the /cancel endpoint (tab close, proxy aborts,
Colab, mobile navigation, etc.).
- Change all StreamingResponse `Connection: keep-alive` headers to
`Connection: close`.
**Issue 2 — High CPU Spinlock & KV Cache Backup Overhead:**
- Set OMP_WAIT_POLICY=PASSIVE and OMP_NUM_THREADS=2 in the
llama-server subprocess environment on Windows to prevent OpenMP
from spin-waiting on all logical cores while the GPU decodes.
- Limit `--threads` to 2 on Windows when the model is fully
GPU-offloaded (`-ngl -1`). Auto-detect otherwise.
- Pass `--cache-ram 0 --ctx-checkpoints 0 --no-cache-prompt
--checkpoint-every-n-tokens -1` on Windows to disable prompt-cache
snapshots that copy KV cache to system RAM over the WDDM/PCI-E bus.
Closes#5692.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: use local import to avoid ruff F823 (sys used before assignment)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* review: address gemini review feedback
- Simplify _fully_gpu_offloaded init: default to False, only set True
in the gpu_indices branch, drop redundant else.
- Log exceptions in _await_disconnect_then_close at debug level instead
of silent pass, per review suggestion.
* Adjust review feedback for PR #5749
- _await_disconnect_then_close: set cancel_event before resp.aclose() so
the streamer's RemoteProtocolError handler treats the watcher-driven
close as cancellation, not an upstream error. Both call sites pass
cancel_event through.
- Windows --cache-ram / --no-cache-prompt / --ctx-checkpoints block: gate
on _fully_gpu_offloaded so CPU and partial-offload Windows runs keep
prompt-cache reuse across turns.
- Windows OMP_WAIT_POLICY / OMP_NUM_THREADS env: same gate so CPU and
partial-offload Windows runs keep default OpenMP parallelism.
* Shorten code comments touched by PR #5749
* Clean up local imports and rename underscore locals in PR #5749
- Drop the function-local `import sys as _sys` introduced as an F823
workaround; remove the redundant in-function `import os`/`import sys`
block so module-level imports resolve sys/os instead. F823 no longer
triggers because no shadowing import remains inside load_model.
- Rename `_fully_gpu_offloaded` and `_t` to `fully_gpu_offloaded` and
`threads_arg`. Underscore-prefixed names usually mean private/module-
level; plain locals match Python style for in-function temporaries.
No behavior change. ruff clean, py_compile clean, 35 studio cancel-
infra tests + 13 launch-gating AST locks + 6 disconnect-watcher locks
+ 4 spoof live-import tests all pass.
* Fix Windows GGUF follow-ups for PR #5749
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Fix cache flag gating for PR #5749
* Fix Python 3.9 annotations for PR #5749
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Anmol Mishra <anmolx.work@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Daniel Han <danielhanchen@gmail.com>
Co-authored-by: wasimysaid <wasimysdev@gmail.com>
Co-authored-by: Lee Jackson <130007945+Imagineer99@users.noreply.github.com>
* Studio: surface the llama.cpp update affordance when MTP is disabled
When a model asks for MTP (auto on an MTP model, or forced mtp / mtp+ngram)
but it gets disabled, the load already degrades gracefully and serves without
speculative decoding. Until now the UI gave no hint why, or that an update
would fix it.
Record why MTP was dropped on the backend (spec_fallback_reason): the probe
found no mtp token (binary_no_mtp), the spawn aborted with an outdated-arch /
context-build error such as a prebuilt that predates the Gemma drafter
(binary_outdated), or the current build could not run it, e.g. a CUDA kernel
limit (runtime_error). Expose it in the inference status. In the chat
Speculative Decoding section, show a short note and, for the two update-fixable
reasons, an inline Update llama.cpp button that reuses the existing update flow.
A runtime_error gets the note without an update push, since a newer build may
not fix it.
Backend tests cover the reason being set / cleared. Frontend typechecks.
* Address review: tighten the update hint to genuinely outdated binaries
Reserve binary_outdated (which surfaces the Update llama.cpp affordance) for an
unknown-architecture abort, which proves the prebuilt predates the model;
classify the generic memory/context build failures as runtime_error, where an
update may not help. Frontend: only append the "Update llama.cpp to enable it"
sentence when an update is actually available, so the text never points at an
action the UI is not offering.
* Studio: enable MTP for sub-3B Gemma separate-drafter GGUFs
The sub-3B auto-drop to ngram-mod was tuned for an embedded draft head
(Qwen), whose per-token cost regresses below 3B. Gemma ships the head as a
separate root mtp-*.gguf drafter, a tiny standalone model that is cheap
enough to win below 3B: B200 Q4_K_XL bench, draft-mtp n=2 vs spec-off,
gemma-4-E2B (2B) = 1.21x (accept ~0.65) while ngram-mod is 1.00x.
Exempt a separate drafter from the sub-3B gate everywhere the threshold is
applied: the resolver (_mtp_too_small), the auto-fit VRAM reserve, the
drafter auto-download decision, and the reload-skip mirror via a
has_separate_drafter flag on _auto_mode_drops_mtp. Embedded sub-3B heads
(Qwen) still drop to ngram-mod. A drafter the binary cannot build (older
prebuilt, or a CUDA kernel limit) still aborts the spawn and the load
retries once without speculative decoding.
Adds the full Qwen3.5 + Gemma-4 (regular and QAT) auto/off/forced resolver
matrix, plus explicit sub-3B exemption tests.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Always compare the separate drafter in the reload-skip mirror
The sub-3B wrapper around the drafter compare could skip it when the drafter
was deleted out from under a running sub-3B server (detected None, stored set),
leaving a stale launch. The resolved-path compare is cheap and already handles
every case, so drop the _auto_mode_drops_mtp guard (and its now-unused imports)
and always compare when the mode can use a drafter and the user does not own
--spec-type. Addresses review feedback on #6191.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: gracefully disable MTP when the model has no head or drafter
Selecting MTP or MTP+Ngram in Speculative Decoding on a GGUF with no nextn
head and no separate drafter aborted the whole load. llama-server does not
no-op an empty draft-mtp request: it exits with 'failed to measure MTP
context memory: failed to create llama_context', surfaced to the user as a
generic 'llama-server failed to start. Check that the GGUF file is valid
and you have enough memory.'
Build-time fix in _build_speculative_flags: when a forced mtp / mtp+ngram
mode targets a model with no MTP head and no drafter (is_mtp_model is
False), default back instead of emitting draft-mtp. mtp falls back to
--spec-default; mtp+ngram keeps the ngram-mod half, which needs no head.
Real MTP models (embedded head or separate drafter), sub-3B MTP overrides,
and the auto path are unchanged.
Runtime hardening: the existing post-launch MTP retry only fired for
separate-file drafters (--model-draft in spec_flags), so an embedded-head
model that the binary cannot build still hard-failed. Gate the retry on the
spec block requesting MTP, recognise the embedded-head abort strings
('failed to measure MTP context memory', 'failed to create llama_context'),
and make the drafter name None-safe in the warning.
Tests: extend the resolver matrix (forced mtp / mtp+ngram on a non-MTP
model) and add two cases asserting the default-back emission.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: danielhanchen <michaelhan2050@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Trim and tighten code comments and docstrings across the repository. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Trim and tighten code comments and docstrings across studio/ Python. Comment-only: every changed file verified code-identical to main via AST/token comparison.
Raise ruff line-length to 100 and extend the local pre-commit format pipeline (def-signature magic-comma normalization, short multi-line assert collapse, kwarg '=' spacing, blank-line-after-short-import removal, adjacent string-literal / f-string+plain merge, redundant-pass pruning). Every transform re-checks the file AST and is dropped if it would differ; the whole-repo reformat is verified AST-identical per file and idempotent.
* studio: add --spec-draft-n-max toggle for MTP speculative decoding
Surface llama-server's --spec-draft-n-max as a first-class
LoadRequest field so users can tune the MTP draft tree size from
the chat settings panel. Default behaviour is unchanged: when the
caller omits spec_draft_n_max, the existing platform defaults still
apply (6 on GPU, 3 on CPU/Mac).
Why this matters: on context-constrained loads the draft KV cache
competes with the target model's KV cache for VRAM. Lowering
spec_draft_n_max reduces that pressure, lets a larger user context
fit, and recovers throughput; raising it pays off when draft
acceptance is high enough to amortise the extra cache.
Backend
- LoadRequest gains an optional spec_draft_n_max: int (1..16).
- LlamaCppBackend.load_model accepts and persists the override on
self._spec_draft_n_max, used in place of the hardcoded 6/3 in the
MTP emit branch.
- LoadResponse and InferenceStatusResponse echo the active value
(None when the platform default is in effect) so the UI can
hydrate the input on refresh.
- _already_in_target_state and _request_matches_loaded_settings
compare spec_draft_n_max alongside speculative_type so a value
change triggers a reload rather than no-op'ing.
- strip_shadowing_flags now strips inherited --spec-* extras when
either speculative_type or spec_draft_n_max is in fields_set, so
an inherited --spec-draft-n-max cannot last-wins-override a fresh
request's first-class field.
Frontend
- LoadModelRequest, LoadModelResponse, InferenceStatusResponse
TypeScript shapes get spec_draft_n_max.
- chat-runtime-store gains specDraftNMax / loadedSpecDraftNMax and
a setter, hydrated from /v1/status and /v1/load.
- chat-settings-sheet renders a "Draft Tokens" numeric input
directly under the Speculative Decoding switch when that switch
is on. Toggling the switch off clears the override; the Reset
button restores the loaded value.
Tests
- Four new regression tests cover _already_in_target_state with
matching / mismatching / non-MTP / unset spec_draft_n_max.
- Existing test_llama_server_args.py and test_llama_cpp_mtp_detection.py
green: 141 passed locally.
* studio: add --spec-draft-p-min and --spec-draft-p-split to spec strip set
llama.cpp server documents --spec-draft-p-min (default 0.75, min draft
acceptance probability) and --spec-draft-p-split (default 0.10). Both
are first-class spec-decoding knobs that should travel with the rest
of the --spec-* family when an Apply re-sets speculative_type, so an
inherited override doesn't leak across a fresh load.
* studio/tests: skip MTP capability-probe tests on Windows
The four probe_server_capabilities tests use a bash stub written to
tmp_path/llama-server, which Windows' subprocess can't execute
directly (no shebang resolution, .bat / .cmd would be needed). Mark
them skipif sys.platform == 'win32' so the rest of the MTP plumbing
suite stays green on Windows CI. Unix coverage is unchanged.
* studio: lower MTP GPU default --spec-draft-n-max from 6 to 2
Bench on B200 / Qwen3.6-27B-MTP-GGUF UD-Q4_K_XL across five prompt
types (essay, code, story, math, science) with greedy temp=0:
prompt OFF n=1 n=2 n=3 n=6
essay 79.1 93.4 93.8 84.7 64.6
code 79.1 104.4 116.6 113.5 103.0
story 79.1 99.2 105.7 101.8 88.9
math 79.1 100.8 110.8 111.8 98.2
science 79.1 100.1 110.8 110.8 102.9
The previous hardcoded GPU default of 6 was 17% SLOWER than spec-off
on the essay prompt (64.6 vs 79.1 t/s) and 11-50% slower than n=2 on
the rest. n=2 wins on 4/5 prompts with a 1.18x-1.47x speedup vs OFF;
n=3 wins on the math prompt by a hair. n=6 collapses once acceptance
rate drops past n=3 -- wasted draft decode dominates the per-step
budget.
Matches the dataset README ("n_max=2 is the sweet spot for 36 of 42
quants"). Keeps CPU/Mac default at 3, which empirically tracks the
narrower ngram+MTP chained budget on those platforms.
Users who want the old behaviour can pass spec_draft_n_max in
LoadRequest (the toggle this PR also adds) or --spec-draft-n-max via
llama_extra_args.
* studio: skip MTP auto-promote on sub-2B models, backfill chat usage
Two MTP-visibility fixes uncovered while bisecting llama.cpp post-#22673
on Qwen3.6-27B-MTP-GGUF UD-Q4_K_XL on B200.
Size gate. Direct llama-server bench (no Studio measurement loop) at
n_predict=192 across 9 prompts shows MTP regresses vs spec-off on
sub-2B dense models because draft cost exceeds savings:
Qwen3.5-0.8B Q4_K_XL GPU: 452.0 OFF -> 283.4 t/s n=2 (0.63x)
CPU: 84.5 OFF -> 64.9 t/s n=3 (0.77x)
Qwen3.5-4B Q4_K_XL GPU: 241.0 OFF -> 258.2 t/s n=2 (1.07x)
Qwen3.5-9B Q4_K_XL GPU: 201.6 OFF -> 228.9 t/s n=2 (1.14x)
Qwen3.5-27B Q4_K_XL GPU: 78.8 OFF -> 113.6 t/s n=2 (1.44x)
Qwen3.6-27B Q4_K_XL GPU: 78.8 OFF -> 113.6 t/s n=2 (1.44x)
Qwen3.6-35B-A3B Q4 GPU: 192.3 OFF -> 223.2 t/s n=2 (1.16x)
The 2B inflection is sharp. Skip auto-promote to draft-mtp when the
identifier reports <2.0B params; users can still force via --spec-type
or the Speculative Decoding toggle. Mirror the gate in the
reload-skip check so a sub-2B reload-with-default does not bounce a
spec-off backend.
Chat-completions usage. llama-server's final SSE chunk emits both an
OpenAI-style usage block and a custom timings block. timings.predicted_n
is always populated, but usage.completion_tokens is zero on some
server builds. The Studio chat UI computes generation t/s from
meta.usage.completion_tokens / totalStreamTime, so a zero
completion_tokens makes the UI fall back to wall-clock time
(including SSE / proxy / template overhead) which dilutes MTP gains and
makes ON look the same as OFF.
Add _backfill_usage_from_timings: if usage.completion_tokens is missing
or zero AND timings has predicted_n/prompt_n, synthesize a complete
usage dict. Apply at the streaming metadata yield in
generate_chat_completion and at the three accumulator/yield sites in
generate_chat_completion_with_tools so per-iteration counts are not
silently lost across tool calls.
Tests cover both the gate (sub-2B skips, 2B+ promotes) and the
backfill (zero usage filled, real usage preserved, empty timings
passthrough).
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: probe + emit legacy ngram-mod flags for pre-rename llama-server
llama.cpp upstream renamed the ngram-mod tuning knobs:
--draft-max -> --spec-ngram-mod-n-max (and --spec-draft-n-max)
--draft-min -> --spec-ngram-mod-n-min (and --spec-draft-n-min)
--spec-ngram-size-n -> --spec-ngram-mod-n-match
The new names are real flags on post-rename builds and stub removal
entries on the same builds (with description "argument has been
removed"). Pre-rename builds only carry the legacy names as real
flags. Studio was emitting the new names unconditionally, so a user
running a pre-rename llama-server (e.g. an older prebuilt or a
hand-installed binary) would see "unknown argument" errors when the
ngram-mod path engages, or silent drop of the ngram knobs.
Extend `probe_server_capabilities` to parse the help text into
per-flag description blocks and tell real flags apart from removal
stubs by the "argument has been removed" marker. Add three new probe
fields: `ngram_mod_flavor` ("new" / "legacy" / None),
`supports_ngram_mod`, and `spec_draft_n_max_flag` (the actual n_max
flag the binary accepts). Cached by (path, mtime) the same way as
`mtp_token`.
Add `_build_ngram_mod_flags(caps, ...)` that picks the right flag
set, returning [] when neither is usable so callers can drop ngram
chaining entirely on minimal binaries.
Wire both call sites to use the probe-driven flag set:
- CPU/Mac MTP comma-chain (--spec-type ngram-mod,draft-mtp) emits
legacy or new knobs as appropriate. If neither set is available,
degrade to MTP-only (warn but still engage spec).
- Standalone --spec-type ngram-mod branch uses the same helper.
Tests cover post-rename detection, legacy detection, removal-stub
discrimination, minimal-binary case, and all three branches of
`_build_ngram_mod_flags` plus custom n_match/n_min/n_max values.
Verified against three real binaries (Studio bundled 726704a, my
build of 45b455e HEAD, and the MTP merge baseline 2555826) all
correctly reporting ngram_mod_flavor=new.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: sub-3B MTP falls back to ngram-mod, not off
Earlier sub-2B gate disabled speculative decoding entirely for tiny
dense MTP models because the MTP draft head's per-token cost exceeds
the acceptance savings at that scale. The "fully off" fallback was
conservative -- ngram-mod has near-zero idle cost on diverse content
and consistently outperforms both off and draft-mtp at sub-3B.
Clean-methodology bench (each of 9 distinct prompts run once after
two unrelated warmup prompts so the ngram-mod hash pool is
realistically populated but never holds the exact deterministic
output we're about to measure):
Q4_K_XL on B200:
0.8B OFF=451 draft-mtp n=2=263 (0.58x) ngram-only=498 (1.10x)
2B OFF=377 draft-mtp n=2=308 (0.82x) ngram-only=369 (1.00x)
4B OFF=240 draft-mtp n=2=260 (1.08x) -- 4B+ wins with MTP
Q4_K_XL on x86 48 cores:
0.8B OFF= 80 chained n=2= 69 (0.86x) ngram-only= 95 (1.19x)
2B OFF= 62 chained n=2= 51 (0.83x) ngram-only= 63 (1.01x)
4B OFF= 31 chained n=2= 41 (1.33x)
Change:
- Raise the MTP-skip threshold from 2.0B to 3.0B (2B falls below it).
- When skipping the MTP head, fall back to --spec-type ngram-mod via
the probe-driven _build_ngram_mod_flags helper. Works on both
post-rename and pre-rename llama-server builds.
- If the binary advertises neither ngram-mod flavor, fall back to
spec-off (older binaries that don't support ngram-mod at all).
- Mirror the same fallback in _already_in_target_state so a sub-3B
reload-with-default does not bounce a ngram-mod backend.
Tests updated: monkeypatch probe_server_capabilities so the gate
behavior is deterministic regardless of which llama-server happens
to be on the host. +1 new test for the "binary has no ngram-mod
support" branch; renamed prior 2B/0.8B tests to reflect new semantics.
This generalizes the size gate to be probe-driven instead of a hard
"disable spec" branch.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: 5-mode Speculative Decoding dropdown (Auto / MTP / Ngram / MTP+Ngram / Off)
Replace the Chat Settings Speculative Decoding on/off Switch with a 5-option
Select. Auto preserves today's platform-aware resolver (MTP on MTP GGUFs,
ngram-mod fallback for sub-3B, --spec-default for non-MTP). The other 3 modes
force the user's choice on BOTH GPU and CPU: MTP emits draft-mtp only (no
ngram chain on CPU), Ngram emits ngram-mod only, MTP+Ngram emits the
ngram-mod,draft-mtp chain on both platforms. Off is the existing fully-off
state, kept so the Switch's "disable" capability isn't lost.
Backend
- New module-level _canonicalize_spec_mode(value) maps any accepted input
(canonical, legacy "default" / "draft-mtp" / "ngram-mod" / "ngram-simple",
or comma-chained "ngram-mod,draft-mtp") onto one of auto / mtp / ngram /
mtp+ngram / off / ngram-simple / None. Lets external callers and old
persisted UI state round-trip without breaking.
- LlamaCppBackend grows a _requested_spec_mode field + requested_spec_mode
property storing the canonical UI mode the user requested. Status
responses round-trip this instead of the resolved internal flag, so the
dropdown restores the picked value after reload / refresh (Auto on a 27B
MTP GGUF resolves to draft-mtp internally but the dropdown stays on
"Auto").
- The resolver block in load_model is extracted into a unit-testable
_build_speculative_flags method. Forced MTP / MTP+Ngram on a sub-3B or
non-MTP GGUF logs a warning and engages anyway (user override > the
Auto-path sub-3B fallback).
- _already_in_target_state and routes/inference._request_matches_loaded_settings
now compare canonical-requested mode, dropping the old auto-promotion
mirror. spec_draft_n_max still gates on the resolved spec so Auto + a
changed n_max still bounces a reload.
Frontend
- chat-settings-sheet.tsx: Switch swapped for Select modeled on the KV
Cache Dtype Select. Items: Auto / MTP / Ngram / MTP+Ngram / Off. Draft
Tokens input only visible when speculativeType is "mtp" or "mtp+ngram".
- chat-runtime-store.ts: initial value flips from "default" to "auto".
- use-chat-model-runtime.ts normalizeSpeculativeType mirrors the backend
canonicaliser so persisted "default" / "draft-mtp" / "ngram-mod" / chain
values hydrate to the right dropdown option.
- types/api.ts: docs the canonical wire vocabulary.
Tests
- 53 new assertions in test_llama_cpp_mtp_detection.py: full
_canonicalize_spec_mode table, a 23-row resolver matrix across
(requested mode) x (GPU/CPU) x (model size class), plus n_max override,
user-extra-args precedence, requested-mode round-trip, and graceful
degrade on an outdated llama-server without an MTP token.
- 165 existing backend tests still green. 218 total in the MTP /
server-args / reload-inheritance suite.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: reset Speculative Decoding to Auto on model switch
When the user switches from model A to a different model B, clear the
runtime store's speculativeType + specDraftNMax (and their loaded*
shadows). The new load request then carries null, the backend
canonicalises that to "auto", and its platform-aware resolver runs
fresh for the new model.
Without this, a non-MTP model loaded with "Off" carried the Off choice
into a subsequent MTP load, suppressing MTP auto-promotion (and the
sub-3B ngram-mod fallback) until the user manually opened settings and
flipped the dropdown back to Auto. The clean-sweep deep probe caught
it as anomaly A-1.
The reset only fires when currentCheckpoint != modelId, so a
same-model reapply or forceReload still honours the user's current
spec choice. End-to-end probe on Qwen3.5-4B-GGUF (non-MTP, Off) ->
Qwen3.5-0.8B-MTP confirms: dropdown shows Auto, /api/inference/status
returns speculative_type=auto, studio.log shows the Auto sub-3B
fallback emitted --spec-type ngram-mod.
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* studio: engage draft-mtp on vision MTP GGUFs
The draft-mtp auto-promotion in LlamaCppBackend.load_model was gated on
not effective_is_vision, and the spec-emit branch repeated the same
guard. Every Unsloth -MTP GGUF repo ships an mmproj projector, so
effective_is_vision was always True for those repos and the MTP speedup
silently never engaged out of the box.
llama.cpp #22673 explicitly states MTP is compatible with vision input.
The bundled b9204 server happily loads both: a manual run with
--mmproj ... --spec-type draft-mtp --spec-draft-n-max 6 logs
"loaded multimodal model" followed by
"adding speculative implementation 'draft-mtp'".
Drop the vision gate from both sites and rewrite the matching short
circuit in _already_in_target_state so reload checks reach the auto
promotion path on vision MTP loads. Add three regression tests covering
vision MTP match (auto and default), and non MTP vision repo unaffected.
Verified on a B200 with unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL:
base decode 179.7 t/s vs MTP decode 253.8 t/s, draft acceptance 0.57,
1.41x speedup on a 255 token completion. mmproj still loads and image
input remains available.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: prefer Qwen3.5 -MTP GGUF variants in default model lists
With the vision gate dropped in the previous commit, draft-mtp now
auto-engages on -MTP GGUF repos out of the box. Swap the four Qwen3.5
recommended entries in DEFAULT_MODELS_GGUF and DEFAULT_MODELS_STANDARD
to their -MTP-GGUF counterparts so new users get the speedup by default:
unsloth/Qwen3.5-4B-GGUF -> unsloth/Qwen3.5-4B-MTP-GGUF
unsloth/Qwen3.5-9B-GGUF -> unsloth/Qwen3.5-9B-MTP-GGUF
unsloth/Qwen3.5-35B-A3B-GGUF -> unsloth/Qwen3.5-35B-A3B-MTP-GGUF
unsloth/Qwen3.5-0.8B-GGUF -> unsloth/Qwen3.5-0.8B-MTP-GGUF
All four HF repos exist (HEAD 200) and ship the same UD-Q4_K_XL quant
layout as the non-MTP variants. Non-Qwen3.5 entries are untouched.
* bump version to 2026.5.4
Picks up the studio MTP vision-gate fix and the Qwen3.5 -MTP default
swap in this PR.
* studio: prefer Qwen3.6-35B-A3B-MTP-GGUF in default model lists
Same rationale as the previous Qwen3.5 swap. The Qwen3.6 MTP variant
exists at unsloth/Qwen3.6-35B-A3B-MTP-GGUF (HF HEAD 200) and now
auto-engages draft-mtp out of the box with the gate fix.
* studio: drop --spec-draft-n-max from 6 to 3 for draft-mtp
n=6 is too greedy: on Qwen3.6 the draft has to guess 6 tokens ahead
and acceptance crashes to ~0.45, leaving only ~14% throughput gain.
PR ggml-org/llama.cpp#22673's author benched n=3 at ~0.72 acceptance
and 2 to 3x speedup on the same Qwen3.6 family, and the README sample
command uses n=2 or n=3. Match that.
CPU/Mac branch already uses n=3, so this aligns both paths.
* studio: set --spec-draft-n-max back to 6 for draft-mtp on GPU
Reverts the n=3 tuning. n=6 is the original default; user-side comparisons
hold the larger draft window steady so the toggle (next commit) is the
primary on/off lever.
* studio: add Speculative Decoding toggle under Max Tokens
Adds a top-level kill switch (panel-switch under Max Tokens, mirroring
Auto-Healing Tool Calls) that forces the /load request's
speculative_type to "off" when disabled. The backend "off" branch in
LlamaCppBackend.load_model skips both the draft-mtp auto-promotion and
the spec-emit branch, so neither --spec-type draft-mtp nor
--spec-default reaches llama-server.
Wiring:
- chat-runtime-store: new speculativeDecodingEnabled bool, default
true, persisted to localStorage under unsloth_speculative_decoding,
plus a setSpeculativeDecodingEnabled setter.
- chat-settings-sheet: SpeculativeDecodingToggle rendered immediately
beneath the Max Tokens slider for non-external models.
- use-chat-model-runtime: when speculativeDecodingEnabled is false,
override speculative_type to "off" in the loadModel call so the
switch wins over any pre-existing speculativeType state (including
the existing per-model toggle in Model Settings).
Verified end to end on unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL:
toggle ON emits --spec-type draft-mtp --spec-draft-n-max 6; toggle
OFF emits zero --spec-* flags on the same MTP GGUF.
* studio: relocate Speculative Decoding toggle into Model Settings
Move the toggle out from under Max Tokens and back into the Model
Settings section, directly beneath KV Cache Dtype, where the existing
Apply/Reset workflow already drives a reload on dirty. This way flipping
the switch in the UI actually picks up: the section becomes dirty,
Apply re-runs /load with the new speculative_type.
Drop the !currentModelIsMultimodal gate so vision MTP GGUFs can also
disable speculative decoding from the UI.
Switch the toggle's off-value from null to "off" so the backend's "off"
short-circuit fires for MTP models too (null normalises to None which
re-triggers the draft-mtp auto-promotion).
Tooltip now reads "Faster generation with 0% accuracy hit".
Remove the now-redundant speculativeDecodingEnabled bool + setter from
the runtime store and the load-time override in use-chat-model-runtime;
the toggle binds directly to speculativeType.
* studio: restore OOM/TIGHT badge on recommended GGUF rows
The recommended-list row passed vramStatus=null for any GGUF repo
because the existing useRecommendedModelVram hook reads safetensors
totals from HF model info, which GGUF-only repos do not expose. As a
result, an OOM Q-quant repo would render with only a "GGUF" badge and
no visual signal that nothing in it fits.
Add useGgufRecommendedFit: per repo, fetch the variant list via the
existing /api/models/gguf-variants endpoint, take the smallest
variant's size_bytes, and classify with the same 0.7*GPU + 0.7*RAM
thresholds as GgufVariantExpander. Session-scoped cache + in-flight
dedup so a repo is requested at most once.
Wire the result into the three GGUF row sites in pickers.tsx so OOM
and TIGHT badges show on the collapsed cards.
* Revert "studio: restore OOM/TIGHT badge on recommended GGUF rows"
This reverts commit 07793b1240df72b13e51d6dc15f63c4ee8c6cba9.
The new useGgufRecommendedFit hook was treating the symptom. PR #5561
identified the real root cause: useGpuInfo was calling /api/system
with plain fetch instead of authFetch, so the session-auth check
failed silently and gpu.available stayed false everywhere. With no
GPU info, every fit check (variant expander, recommended carousel)
fell back to "no signal" and dropped the OOM/TIGHT badges.
Reverting the over-engineered hook and applying the authFetch fix
in the next commit, which restores the existing badges with one line.
* chore: replace qwen suggested with MTP variant
* fix: restore GPU info auth for GGUF fit badges
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: imagineer99 <samleejackson0@gmail.com>
* Studio: warn when llama.cpp prebuilt is too old for MTP
Layered on #5527. Adds a one-shot llama-server --help capability probe
so users get a clear signal when their prebuilt is missing MTP support,
plus a graceful fallback if they load an MTP GGUF against an outdated
binary.
What's surfaced:
1. Startup log + stderr line in main.py:lifespan() if MTP isn't
advertised:
WARNING: llama.cpp prebuilt is missing MTP support
(--spec-type mtp / draft-mtp). Run `unsloth studio update` to
refresh it. MTP GGUFs will load without speculative decoding.
2. Load-time graceful fallback in load_model's spec block: skip the
auto-emit and log a clear warning instead of letting llama-server
fail with an unknown-flag error.
3. /api/inference/status now returns llama_cpp_supports_mtp: bool so
the frontend can show a banner / popup.
Probe internals:
- Class-level cache keyed on (binary_path, mtime). One subprocess call
the first time, instant thereafter. Touching the binary (e.g. via
`unsloth studio update`) invalidates the cache automatically because
the mtime changes, so the new build is picked up without restarting
the server.
- Recognises both upstream naming forms: the original draft-mtp from
llama.cpp PR #22673 and the renamed mtp variant in later commits.
- Spec block uses whichever token the binary accepts so we emit the
right value regardless of which release the user has.
Tests:
- 6 new cases in test_llama_cpp_mtp_detection.py covering each probe
variant (draft-mtp, renamed mtp, pre-MTP build, missing binary,
mtime-based cache invalidation).
- Existing 38 MTP detection cases still pass; broader 188-test
regression suite (server args, reload inheritance, gguf metadata,
load progress, context fit, model validation) still green.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Studio: auto-enable MTP speculative decoding for MTP GGUFs
Detect Unsloth's MTP (multi-token-prediction) GGUFs and auto-emit the
right --spec-type draft-mtp flags for llama-server (llama.cpp PR
#22673), so users get the speedup without configuration.
Detection prefers the GGUF metadata field <arch>.nextn_predict_layers
(verified on Qwen3.6-27B-MTP-GGUF / qwen35 and Qwen3.6-35B-A3B-MTP-GGUF
/ qwen35moe). Falls back to a -MTP marker in the identifier / filename
so HF-mode loads can detect MTP from the repo name before the GGUF is
downloaded.
Flag presets follow the Unsloth MTP guide:
GPU: --spec-type draft-mtp --spec-draft-n-max 6
CPU/Mac: --spec-type draft-mtp --spec-draft-n-max 3 \
--spec-type ngram-mod --spec-ngram-mod-n-match 24 \
--spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 6
User overrides win: if the caller passes --spec-type / --spec-default
via unsloth run / unsloth studio run pass-through (or HTTP
llama_extra_args), the auto-emit steps aside so llama-server only sees
the user's flag. Scalar tuning knobs like --spec-draft-n-max compose
with the auto preset via llama-server's last-wins parsing.
_already_in_target_state mirrors the same promotion so a repeat /load
with unchanged settings against an MTP backend running draft-mtp
short-circuits cleanly instead of forcing a reload.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
UmranPros