ceos/unsloth
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

studio: emit one comma-chained --spec-type for CPU/Mac MTP path (#5575)

Browse source 
* studio: emit one comma-chained --spec-type for CPU/Mac MTP path

llama-server takes a single --spec-type whose value may be
comma-separated to chain implementations (e.g. ngram-mod,draft-mtp).
The CPU/Mac MTP branch in LlamaCppBackend.load_model was passing
--spec-type twice in the same invocation, which is not the documented
chaining mechanism and silently drops one of the two specs depending
on llama.cpp's argv handling.

Collapse the pair to --spec-type ngram-mod,{mtp_token} and update the
stale _extra_args_set_spec_type docstring that claimed llama-server
accumulates repeated --spec-type. Update the matching pass-through
fixture in test_llama_server_args.py.

* studio: align MTP ngram-mod knobs with llama.cpp upstream defaults

Two correctness fixes against the llama.cpp server README:

1. The CPU/Mac comma-chained branch was emitting
   --spec-ngram-mod-n-max 6 with --spec-ngram-mod-n-min 48, which is
   nonsensical (min > max). Per the upstream default the value is 64.

2. The standalone ngram-mod branch was emitting --spec-ngram-size-n,
   --draft-min, --draft-max. llama.cpp removed those arg aliases for
   ngram-mod (they live only on the ngram-simple / map families now);
   the correct knobs are --spec-ngram-mod-n-match / n-min / n-max.

Also refresh the inline comment block to point at the server README
rather than the older docs/speculative.md draft- aliases.
This commit is contained in:
Daniel Han 2026-05-19 03:16:05 -07:00 • committed by GitHub
commit 5ce4ab4d54
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 20 additions and 17 deletions
Download patch file Download diff file Expand all files Collapse all files

31
studio/backend/core/inference/llama_cpp.py
View file

@ -471,8 +471,9 @@ def _is_mtp_model_name(
def _extra_args_set_spec_type(extra_args: Optional[Iterable[str]]) -> bool:
"""User passed --spec-type / --spec-default? llama-server accumulates
repeated --spec-type, so we suppress auto-emit when this is true."""
"""User passed --spec-type / --spec-default? llama-server takes a
single --spec-type (comma-separated to chain), so suppress
auto-emit when this is true."""
if not extra_args:
return False
for raw in extra_args:
@ -2631,10 +2632,10 @@ class LlamaCppBackend:
# Qwen3-235B offloaded | 12 t/s | 21 t/s | 1.8x
# gpt-oss-120b repeat (92% accept)| 181 t/s | 814 t/s | 4.5x
#
# Params from llama.cpp docs (docs/speculative.md):
# --spec-ngram-size-n 24 (small n not recommended)
# --draft-min 48 --draft-max 64 (MoEs need long drafts;
# dense models can reduce these)
# Params from llama.cpp server README:
# --spec-ngram-mod-n-match 24 (lookup length)
# --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64
# (MoEs need long drafts; dense models can reduce these)
# ref: https://github.com/ggml-org/llama.cpp/blob/master/docs/speculative.md
# ref: https://github.com/ggml-org/llama.cpp/pull/19164
# ref: https://github.com/ggml-org/llama.cpp/pull/18471
@ -2692,20 +2693,22 @@ class LlamaCppBackend:
]
)
else:
# CPU/Mac: chain ngram-mod + MTP in one
# comma-separated --spec-type (not repeated).
# ngram-mod knobs match llama.cpp defaults
# (n-match 24, n-min 48, n-max 64).
cmd.extend(
[
"--spec-type",
mtp_token,
f"ngram-mod,{mtp_token}",
"--spec-draft-n-max",
"3",
"--spec-type",
"ngram-mod",
"--spec-ngram-mod-n-match",
"24",
"--spec-ngram-mod-n-min",
"48",
"--spec-ngram-mod-n-max",
"6",
"64",
]
)
self._speculative_type = "draft-mtp"
@ -2715,13 +2718,15 @@ class LlamaCppBackend:
elif normalized_spec in _valid_spec_types:
cmd.extend(["--spec-type", normalized_spec])
if normalized_spec == "ngram-mod":
# llama.cpp defaults; legacy --spec-ngram-size-n
# / --draft-{min,max} were removed for ngram-mod.
cmd.extend(
[
"--spec-ngram-size-n",
"--spec-ngram-mod-n-match",
"24",
"--draft-min",
"--spec-ngram-mod-n-min",
"48",
"--draft-max",
"--spec-ngram-mod-n-max",
"64",
]
)

6
studio/backend/tests/test_llama_server_args.py
View file

@ -47,17 +47,15 @@ from core.inference.llama_server_args import (
["--spec-type", "draft-mtp", "--spec-draft-n-max", "6"],
[
"--spec-type",
"draft-mtp",
"ngram-mod,draft-mtp",
"--spec-draft-n-max",
"3",
"--spec-type",
"ngram-mod",
"--spec-ngram-mod-n-match",
"24",
"--spec-ngram-mod-n-min",
"48",
"--spec-ngram-mod-n-max",
"6",
"64",
],
# Reasoning controls
["--reasoning-format", "deepseek"],