70021b1f6a
The per-family inference defaults registry (inference_defaults.json)
exposes scalar sampling fields only (temperature, top_p, top_k, min_p,
presence_penalty). Modern reasoning models also want template-level
kwargs: gpt-oss takes reasoning_effort (low|medium|high); Nemotron-3
takes enable_thinking. Today those reach llama-server only when an
OpenAI SDK client sets extra_body.chat_template_kwargs explicitly, so
external clients hitting /v1/chat/completions never benefit from the
recommended defaults.
This patch:
* Adds an optional ``chat_template_kwargs`` block per family in
inference_defaults.json. Bootstrap rows:
gpt-oss -> reasoning_effort: medium
nemotron -> enable_thinking: true
* Extends load_inference_config to surface chat_template_kwargs in
the returned dict (None when unset).
* Extends _build_openai_passthrough_body to merge family defaults
(lookup via load_inference_config(model_identifier)) with per-request
overrides. Priority order, base to top:
family_defaults -> payload.enable_thinking -> payload.reasoning_effort
-> extra_body.chat_template_kwargs
* Bumps two family temperatures to match upstream HF model cards:
devstral 0.7 -> 0.15 (Devstral-Small-2-Instruct card)
ministral 0.15 -> 0.05 (Ministral-3 card: "below 0.1 for production")
|
||
|---|---|---|
| .. | ||
| model_defaults | ||
| __init__.py | ||
| full_finetune.yaml | ||
| inference_defaults.json | ||
| lora_text.yaml | ||
| vision_lora.yaml | ||