facdff9ad7
Cross-checked every supported sampling field against each provider's
live docs + LiteLLM's drop_params surface + the llama.cpp server
README. Pulled in the most-asked-for samplers that the PR was missing.
New ProviderCapabilities flags (default false on every SaaS provider
since none accept these):
- typicalP (already shipped one commit prior)
- topNSigma llama.cpp `top_n_sigma`
- repeatLastN llama.cpp `repeat_last_n` (paired w/ repeat_penalty)
- dynatempRange llama.cpp `dynatemp_range`
- dynatempExponent llama.cpp `dynatemp_exponent`
- mirostat llama.cpp `mirostat` mode (0/1/2)
- mirostatTau llama.cpp `mirostat_tau`
- mirostatEta llama.cpp `mirostat_eta`
- topA OpenRouter `top_a` (alternate dynamic-top-P)
Capability bucketing split: ALL_SUPPORTED retired in favor of
- LOCAL_LLAMA_CAPABILITIES -> custom / vllm / ollama / llama_cpp
(full llama.cpp sampler chain, top_a off — not a llama.cpp field)
- OPENROUTER_CAPABILITIES -> openrouter
(gateway's documented set incl. top_a, llama.cpp-only knobs off
because OpenRouter docs don't list them and they'd be silently
dropped on most underlying routes)
InferenceParams gains 8 nullable-number fields (mirroring `seed`'s
"null = unset, finite-number = forwarded" shape). DEFAULT_INFERENCE_PARAMS
defaults each to null. Persistence handler in chat-settings-storage
mirrors typicalP's nullable-float handling for all 8.
Backend:
- 8 new ChatCompletionRequest fields with appropriate `ge`/`le`
validators (mirostat 0..2, ranges 0.0..1.0 where applicable).
- llama_cpp.py: signatures + payload forwarding extended on all
three builders (chat-completion, agentic tool-loop, final-pass)
so the new fields survive the local tool-loop too. `is not None`
gating so defaults (e.g. mirostat=0) reach the wire only when the
caller explicitly opted in.
- routes/inference.py: _build_passthrough_payload extends to the
extended sampler chain; 3 call sites (generate_chat_completion,
generate_chat_completion_with_tools, _build_passthrough_payload)
forward each field from `payload.*`.
Frontend chat-adapter: external branch forwards only when capability
allows (so OpenRouter gets top_a but not mirostat, local gets mirostat
but not top_a); local branch forwards unconditionally when the value
is meaningful (e.g. mirostat != 0, dynatemp_range > 0).
Test pinning the new field round-trip through _build_passthrough_payload
added; full PR-touched suite now 163 passing (was 161).
References:
- llama.cpp server params: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
- OpenRouter params: https://openrouter.ai/docs/api/reference/parameters
- LiteLLM provider params: https://docs.litellm.ai/docs/completion/input
|
||
|---|---|---|
| .. | ||
| __init__.py | ||
| _html_to_md.py | ||
| anthropic_compat.py | ||
| audio_codecs.py | ||
| chat_template_helpers.py | ||
| defaults.py | ||
| external_provider.py | ||
| inference.py | ||
| key_exchange.py | ||
| llama_cpp.py | ||
| llama_server_args.py | ||
| mlx_inference.py | ||
| orchestrator.py | ||
| pricing.py | ||
| providers.py | ||
| safetensors_agentic.py | ||
| tool_call_parser.py | ||
| tools.py | ||
| worker.py | ||