If CUDA_VISIBLE_DEVICES is already set in the environment (e.g.,
by the user or a wrapper script), only consider those GPUs when
selecting devices for llama-server. nvidia-smi reports all physical
GPUs regardless of CUDA_VISIBLE_DEVICES, so we filter its output
to match the allowed set.
Without this, the GPU selector could pick a GPU outside the user's
allowed set, overriding their restriction.
Automatically select the best GPU(s) for a GGUF model based on
file size and available VRAM, instead of relying on hardcoded
-ngl -1 or letting llama-server guess.
Logic:
1. Measure total GGUF file size (including split shards)
2. Query free memory per GPU via nvidia-smi
3. If the model fits in 70% of the most-free GPU's memory,
pin to that single GPU (CUDA_VISIBLE_DEVICES=X, no --fit)
4. If it needs multiple GPUs, pick the N most-free GPUs
(CUDA_VISIBLE_DEVICES=X,Y, no --fit)
5. If it's too large for all GPUs combined, omit
CUDA_VISIBLE_DEVICES and use --fit on to let llama-server
handle partial offloading
The 70% threshold accounts for KV cache and compute buffers
that sit on top of the model weights.
Removed the -ngl parameter (was hardcoded to -1). llama-server's
default of "auto" handles layer offloading correctly, especially
with --fit on for oversized models.
Tested on 8x B200:
- 1B model (0.75 GB): picks 1 GPU, no --fit
- 27B model (17 GB): picks 1 GPU, no --fit
- 405B model (230 GB): picks 2 GPUs, no --fit
- 2TB model: all GPUs, --fit on
Refactor command building (deduplicate HF/local paths) and add
flags for better performance:
- --parallel 1: studio is single-user, so only 1 inference slot
is needed. The previous auto-detect picked 4 slots, wasting
VRAM on 3 unused KV caches.
- --flash-attn on: force flash attention for faster inference.
Default is "auto" which may not always enable it.
- --fit on: auto-adjust parameters to fit in available device
memory. Already the default but now explicit.
Also cleaned up the duplicated command building for HF vs local
mode into a single block.
Remove the hard max_tokens=2048 default and le=4096 cap for GGUF
chat completions. When max_tokens is not set (None), the field is
omitted from the llama-server payload entirely, letting the model
generate until it produces an EOS token or hits the context limit.
This is critical for thinking/reasoning models (Qwen3.5, DeepSeek-R1,
etc.) where the thinking phase alone can consume 1000+ tokens before
the actual answer. With the previous 2048 default, simple questions
like "What is 2+2?" used all tokens on thinking and produced empty
visible responses.
Changes:
- llama_cpp.py: max_tokens default None, only include in payload
when explicitly set
- models/inference.py: default None, remove le=4096 cap
- routes/inference.py: pass max_tokens directly, no "or 2048" fallback
llama-server handles omitted max_tokens gracefully (generates until
EOS or context limit). The context size (-c flag, default 4096) acts
as the hard upper bound.
llama-server sends thinking/reasoning tokens as "reasoning_content"
in the SSE delta (separate from "content"). The studio was only
reading delta.content, so all reasoning tokens from models like
Qwen3.5, Qwen3-Thinking, DeepSeek-R1, etc. were silently dropped.
This caused "replies with nothing" for thinking models: the model
would spend its entire token budget on reasoning, produce zero
content tokens, and the user would see an empty response.
Fix: read reasoning_content from the delta and wrap it in
<think>...</think> tags. The frontend already has full support
for these tags (parse-assistant-content.ts splits them into
reasoning parts, reasoning.tsx renders a collapsible "Thinking..."
indicator).
Verified with Qwen3.5-27B-GGUF (UD-Q4_K_XL):
- Before: "What is 2+2?" -> empty response (all tokens in reasoning)
- After: shows collapsible thinking + answer "4"
startswith(prefix) could match unrelated split variants whose names
extend the selected file's prefix (e.g. model-Q8_0-v2-00001-of-...).
Now builds an exact regex from the chosen file's base prefix and shard
total so only true siblings are downloaded.
Substring matching (e.g. "Q8_0" in filename) could match superset
variants like "IQ8_0", causing wrong quantizations to be downloaded.
Now uses word-boundary regex for variant matching and discovers split
shards by shared filename prefix rather than treating all variant
matches as shards.
LlamaCppBackend.load_model() only downloaded the first matching GGUF
file. For split models (e.g. 7B Q8_0 with 3 shards), llama-server
needs all shards present. Now collects and downloads all matching files.
Fixes two bugs:
1. Chat template tags (<|im_start|>, <|im_end|>) leaking into output
because /v1/completions treated them as literal text
2. Image hallucination because image_b64 was never passed to llama-server
Now llama-server handles chat templates natively and receives images
as OpenAI-format multimodal content parts for vision models.
Replace Python-side GGUF download with llama-server's native -hf flag for
HuggingFace repos. Add frontend variant picker so users can choose
quantization (Q4_K_M, Q8_0, BF16, etc.) with file sizes. Fix vision
detection via mmproj files instead of hardcoding is_vision=False.
