887e7a31c4
Remove the hard max_tokens=2048 default and le=4096 cap for GGUF chat completions. When max_tokens is not set (None), the field is omitted from the llama-server payload entirely, letting the model generate until it produces an EOS token or hits the context limit. This is critical for thinking/reasoning models (Qwen3.5, DeepSeek-R1, etc.) where the thinking phase alone can consume 1000+ tokens before the actual answer. With the previous 2048 default, simple questions like "What is 2+2?" used all tokens on thinking and produced empty visible responses. Changes: - llama_cpp.py: max_tokens default None, only include in payload when explicitly set - models/inference.py: default None, remove le=4096 cap - routes/inference.py: pass max_tokens directly, no "or 2048" fallback llama-server handles omitted max_tokens gracefully (generates until EOS or context limit). The context size (-c flag, default 4096) acts as the hard upper bound. |
||
|---|---|---|
| .. | ||
| data_recipe | ||
| .gitkeep | ||
| __init__.py | ||
| auth.py | ||
| datasets.py | ||
| export.py | ||
| inference.py | ||
| models.py | ||
| training.py | ||