Round 4 expansion driven by direct fetches of the canonical
SamplingParams + server README pages cited in the user's request.
Adds ten more knobs the docs explicitly support but the panel doesn't
surface yet:
Knob (wire name) llama.cpp vLLM Ollama Source
----------------------------- ---------- ------ -------- ----------------
skip_special_tokens no yes no vLLM SamplingParams
spaces_between_special_tokens no yes no vLLM SamplingParams
include_stop_str_in_output no yes no vLLM SamplingParams
truncate_prompt_tokens no yes no vLLM SamplingParams
n_keep yes no no llama.cpp README
n_probs yes no no llama.cpp README
cache_prompt yes no no llama.cpp README
return_tokens yes no no llama.cpp README
timings_per_token yes no no llama.cpp README
post_sampling_probs yes no no llama.cpp README
Backend rationale:
- vLLM's documented SamplingParams class at
https://docs.vllm.ai/en/latest/api/vllm/sampling_params/ lists
skip_special_tokens (default True), spaces_between_special_tokens
(True), include_stop_str_in_output (False), truncate_prompt_tokens
(None). All four are vLLM-only; llama-server's README does not
document them and Ollama's openai/openai.go translator does not
forward them.
- llama-server's README at
https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
lists n_keep, n_probs, cache_prompt, return_tokens, timings_per_token
and post_sampling_probs as documented per-request fields. vLLM's
SamplingParams has no analog, and Ollama's OAI translator drops them.
Capability matrix:
LLAMA_CPP_CAPABILITIES: 6 llama-only true + 4 vLLM-only false.
VLLM_CAPABILITIES: 4 vLLM-only true + 6 llama-only false.
OLLAMA_CAPABILITIES: all 10 off (OAI translator drops all of them).
Every other bucket: all 10 off.
Skip-when-default rules (mirror upstream defaults):
skip_special_tokens / spaces_between_special_tokens / cache_prompt:
default true upstream — forward only when explicitly false.
include_stop_str_in_output / return_tokens / timings_per_token /
post_sampling_probs: default false — forward only when true.
truncate_prompt_tokens / n_probs: 0 / null = unset — forward when > 0.
n_keep: accepts -1 for "keep all", so the gate is value != 0.
Frontend:
- ProviderCapabilities interface +10 flags.
- InferenceParams +10 nullable fields (3 numeric + 7 boolean), all
null in DEFAULT_INFERENCE_PARAMS.
- OpenAIChatCompletionsRequest wire shape +10 optional fields.
- chat-adapter forwards each in both the external (capability-aware)
and local (capability-bypass) branches.
- chat-settings-storage adds the 3 numeric keys to the existing
nullable-number loop and 7 boolean keys to a new nullable-boolean
loop (alongside ignoreEos).
Backend:
- ChatCompletionRequest +10 Optional Fields with pydantic bounds
(truncate_prompt_tokens ge=1, n_probs ge=0; booleans unbounded;
n_keep accepts -1 so no lower bound).
- llama_cpp.py three payload builders (generate_chat_stream + the
tool-loop payload block + the final-pass stream_payload) each
accept and forward the 10 new kwargs.
- routes/inference.py _build_passthrough_payload accepts and forwards
the 10; both per-request call sites (lines ~2591, ~2790) thread
them from the request payload into the llama_cpp methods.
Test: test_local_passthrough_forwards_vllm_output_and_llama_cpp_
instrumentation round-trips all 10 fields with explicit values
matching each backend's upstream default and confirms each is absent
from the body when unset.
65/65 sampling_params_routing tests pass; frontend tsc clean.
Total local-backend knob coverage now (this PR):
Standard: temperature, top_p, top_k, min_p, repetition_penalty,
presence_penalty, frequency_penalty, seed, stop,
parallel_tool_calls (10)
llama.cpp: typical_p, top_n_sigma, repeat_last_n, dynatemp_range,
dynatemp_exponent, mirostat, mirostat_tau, mirostat_eta,
dry_multiplier, dry_base, dry_allowed_length,
dry_penalty_last_n, xtc_probability, xtc_threshold,
min_keep, ignore_eos, min_tokens, n_keep, n_probs,
cache_prompt, return_tokens, timings_per_token,
post_sampling_probs (23)
vLLM-extra: ignore_eos, min_tokens, skip_special_tokens,
spaces_between_special_tokens, include_stop_str_in_output,
truncate_prompt_tokens (6)
OpenRouter: top_a (1)
Deferred for future PRs (require array / object field shape):
- llama.cpp DRY sequence_breakers (string array)
- llama.cpp samplers ordering (string array)
- llama.cpp / vLLM logit_bias (dict)
- llama.cpp grammar (string) + json_schema (object)
- vLLM guided_json / guided_regex / guided_choice / guided_grammar
- vLLM allowed_token_ids / bad_words / stop_token_ids (int / str arrays)
- OpenAI / Ollama logprobs + top_logprobs (bool + int pairing)
- n / best_of (need SSE multi-choice handling first)
64328962d0