* studio: extract param count from model name as fallback
When HuggingFace API doesn't return totalParams for a model,
extract the param count from the model name (e.g. "Qwen3-0.6B"
-> "0.6B", "Llama-3.2-1B-Instruct" -> "1B"). Applied to both
the recommended list and HF search results.
* studio: read GGUF context_length via fast header parser, set max tokens
- Fast GGUF metadata reader (~30-55ms) parses only KV header, skips
tensor data and large arrays (tokenizer vocab etc)
- Extracts context_length and chat_template from GGUF metadata
- Returns context_length in LoadResponse for frontend to use
- Frontend sets maxTokens to actual context_length for GGUFs (e.g.
262144 for Qwen3.5-9B, 131072 for Qwen2.5-7B)
- Max Tokens slider shows "Max" and is locked for GGUFs
- Auto-load path also uses actual context_length from load response
- Toast auto-dismiss (5s) and close button for auto-load toast
* studio: GGUF TTS audio support (from PR #4318)
Add GGUF TTS audio generation via llama-server. When a GGUF model
loads, the backend probes its vocabulary to detect audio codecs
(SNAC/BiCodec/DAC/CSM/Whisper). If detected, the codec is pre-loaded
and the model is reported as audio to the frontend.
During chat, TTS models route to the audio generation path which sends
a per-codec prompt to llama-server's /completion endpoint, extracts
generated tokens/text, and decodes to WAV using AudioCodecManager.
Also strips base64 audio data from prior assistant messages to prevent
context overflow.
Co-authored-by: Manan Shah <mananshah511@gmail.com>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Remove package-lock.json from tracking
* studio: per-model inference defaults, GGUF max tokens fix, reasoning toggle
- Add inference_defaults.json with per-model-family sampling parameters
for ~50 families (Qwen3.5, Qwen3, Gemma-3, Llama-3, DeepSeek, etc.).
Values sourced from unslothai/docs and Ollama params blobs.
- Family-based lookup in inference_config.py: extracts model family from
identifier, matches against patterns (longest match first), merges with
priority: model-specific YAML > family JSON > default.yaml.
- Fix GGUF Max Tokens slider locked at "Max": store ggufContextLength
separately from maxTokens so the slider is adjustable (step=64).
- Fix Ministral YAML: top_p was literal string "default", now 0.95.
- Add reasoning toggle for thinking models (Qwen3.5, Qwen3, DeepSeek-R1,
DeepSeek-V3.1, etc.): detect enable_thinking support from GGUF chat
template metadata, pass --jinja to llama-server, send
chat_template_kwargs per-request. Frontend shows "Reasoning is ON/OFF"
pill button next to attachment button in composer.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: remove default system prompt injection
Backend was injecting "You are a helpful AI assistant." when no system
prompt was provided. Neither unslothai/docs nor Ollama specify a default
system prompt for most models. Now defaults to empty string, letting the
model's own chat template handle system behavior.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: use lightbulb icons and "Think" label for reasoning toggle
Lightbulb on when thinking enabled, lightbulb-off when disabled.
Label is just "Think" in both states; grayed out styling when off.
* studio: fix HTML file upload breaking chat
Replace SimpleTextAttachmentAdapter with custom TextAttachmentAdapter
(excludes text/html) and HtmlAttachmentAdapter that strips tags via
DOMParser, removing scripts/styles and extracting readable text content
instead of dumping raw HTML markup into the conversation.
* studio: show chat template in Configuration panel
Display the model's Jinja2 chat template in a new "Chat Template"
section under Settings (now open by default). For GGUFs, reads from
GGUF metadata; for safetensors, reads from tokenizer.chat_template.
Template is editable with a "Restore default chat template" button
that appears when modified. Section only shows when a model with a
chat template is loaded.
* studio: editable chat template with Apply & Reload
Chat template section now functional:
- Editing the template shows "Apply & Reload" (reloads model with
custom template) and "Revert changes" buttons
- For GGUFs: writes template to temp .jinja file, passes
--chat-template-file to llama-server on reload
- For non-GGUF: passes chat_template_override in load request
- Settings section now open by default
- selectModel supports forceReload to reload same model
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* studio: fix DeepSeek reasoning detection and auto-load metadata
- Set _model_identifier before _read_gguf_metadata so DeepSeek
"thinking" template detection works (was always None before)
- Populate ggufContextLength, supportsReasoning, reasoningEnabled,
defaultChatTemplate in autoLoadSmallestModel GGUF path
* studio: add spacing before BETA badge in navbar
Add gap-1.5 on the logo Link container to space the BETA label
from the wordmark.
Co-authored-by: Imagineer99 <Imagineer99@users.noreply.github.com>
* studio: vertically center BETA badge with logo
---------
Co-authored-by: Manan Shah <mananshah511@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Imagineer99 <Imagineer99@users.noreply.github.com>
* Strip <think> blocks from LLM assist model output
* Add debug logging for raw LLM assist output
* Quiet llama-server logs, use structlog in llm_assist
* Fix think-tag stripping when response is inside tags
* Remove debug logging of raw model output
* Clarify GGUF download logs: show cache hit vs actual download
* Clarify heuristic-detected mapping in UI text
* Default helper model to Qwen3-4B-Instruct-2507 UD-Q4_K_XL
* Remove package-lock.json from tracking, add to .gitignore
* Auto-open mapping dialog on Start Training for custom_heuristic format
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Use last think block when extracting inner content (review feedback)
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
- Use -c 0 for llama-server (model's native context size, no 4096 cap)
- Run non-GGUF backend.load_model in asyncio.to_thread for progress polling
- Auto-load smallest downloaded model when user chats without selecting one
- Wait for in-progress model load before inference (no "No model loaded" error)
- Add modelLoading flag to zustand store for cross-component coordination
- Dynamic top models: send 8 GGUFs + 8 hub models, frontend caps 4+4 after dedup
- Case-insensitive dedup: downloaded models correctly hide from recommended list
- Prevent duplicate toasts: guard against double selectModel calls
- Model selector waits for cached data before rendering (no empty flash)
- Toast close button positioned at top-right with proper spacing
- Sampling section expanded by default in chat settings
- Global toast close button styling fix
Change all repetition_penalty defaults from 1.1 (or 1.05/1.2 in
presets) to 1.0 across the entire backend and frontend. Most models
handle repetition well on their own and a non-1.0 penalty can degrade
output quality, especially for code, structured output, and creative
tasks.
Files changed:
- Backend: inference.py, llama_cpp.py, orchestrator.py, worker.py,
models/inference.py (Field defaults)
- Frontend: chat-settings-sheet.tsx (Creative/Precise presets),
runtime-provider.tsx (auto-title generation)
GGUF repos with mmproj files (e.g. Qwen3.5-0.8B-GGUF) are already
detected as vision-capable by list_gguf_variants(), and is_vision is
set correctly in ModelConfig. However, the HF download path only
downloaded the main GGUF file without the mmproj projection file,
so llama-server started without --mmproj and rejected image uploads
with "text-only model" errors.
Add _download_mmproj() to LlamaCppBackend that:
- Lists repo files for mmproj*.gguf matches
- Prefers mmproj-F16.gguf (best quality), falls back to any mmproj
- Downloads via hf_hub_download (uses the same HF cache)
In load_model(), when is_vision=True and no explicit mmproj_path was
provided (HF mode), auto-download the mmproj after the main GGUF.
The downloaded path is passed to llama-server via --mmproj.
- Run GGUF load_model in asyncio.to_thread so the event loop stays free
for progress polling during download (was blocking all requests).
- Extract download phase out of the lock in LlamaCppBackend.load_model
so unload_model/cancel can take effect immediately during download.
- Fix "downloaded" badge for split GGUFs: check total cached bytes
across all shards vs expected size, not just first shard existence.
- Respect CUDA_VISIBLE_DEVICES in /api/system GPU reporting so the
frontend GGUF fit estimation uses actual available VRAM.
- Sort tight variants (need CPU offload) smallest-first instead of
largest-first -- closer to GPU budget = faster inference.
- Fix cancel: use refs instead of React state for abort controller and
toast ID so both cancel buttons (text + toast) work reliably. Make
cancel synchronous (fire-and-forget unload) for instant UI response.
Check abortCtrl.signal.aborted after loadModel returns to prevent
ghost model state. Skip rollback and suppress errors on cancel.
- Dynamic top 4 GGUF models fetched from HF API sorted by downloads,
prepended to the default recommended list.
- Remove turnAnchor="top" for auto-scroll to bottom during generation.
- Set default toast duration to 10s (was infinite for loading toasts).
- Deduplicate cached GGUF repos using scan_cache_dir API (fixes
Qwen/X-GGUF vs qwen/x-gguf duplicates from lowercased HF cache).
- Pre-compile repo_id validation regex to silence CodeQL ReDoS warning.
- Change welcome text and default suggestion text.
_get_gpu_free_memory was filtering by CUDA_VISIBLE_DEVICES, so with
CUDA_VISIBLE_DEVICES='0' set by the training env, llama-server only
saw 1 GPU and used --fit for CPU offloading instead of spreading
across all 8 GPUs.
Since llama-server manages its own GPU allocation (the _select_gpus
method picks GPUs and sets CUDA_VISIBLE_DEVICES for the subprocess),
the query must see ALL physical GPUs to make the right decision.
1. n_gpu_layers kwarg: accept (and ignore) in load_model signature
so callers like llm_assist.py don't get TypeError
2. mmproj exclusion: filter out mmproj files in _find_smallest_fitting_variant
so fallback doesn't pick a tiny vision projection as the "model"
3. Shard preservation after fallback: re-discover shards for the
fallback variant instead of resetting to empty list, so split
GGUFs download all shards
4. Orphan cleanup safety: only kill llama-server processes whose
cmdline contains ".unsloth/", avoiding termination of unrelated
llama-server instances on the same machine
5. Path expression sanitization: validate repo_id format before using
it in cache directory lookups
1. Interruptible downloads: load_model now checks a cancel event
between shard downloads. unload_model sets the event so cancel
stops the download at the next shard boundary.
2. /api/models/cached-gguf endpoint: scans the HF cache for
already-downloaded GGUF repos with their total size and cache path.
3. "Downloaded" section in Hub model picker: shows cached GGUF repos
at the top (before Recommended) so users can quickly re-load
previously downloaded models without re-downloading.
The unload endpoint checked is_loaded (requires healthy=True), but
during initial loading the server is not yet healthy. Cancel had no
effect because the unload route fell through to the Unsloth backend.
Fix: add is_active property (process exists, loading or loaded) and
check it in the unload route so cancel kills llama-server even during
the download/loading phase.
Also: toast cancel button now properly triggers the backend unload.
When the studio process is killed (SIGTERM/SIGKILL), atexit handlers
may not run in the subprocess orchestrator, leaving llama-server
processes orphaned and holding GPU memory. This caused OOM errors when
trying to load a new model after a studio restart.
On init, LlamaCppBackend now runs pgrep to find and SIGKILL any stale
llama-server processes before starting fresh.
The smallest-fitting-variant fallback now groups split GGUF shards
by their variant prefix and sums all shard sizes per variant.
For example, DeepSeek-V3.2 UD-Q4_K_XL has 9 shards totaling
379.8 GB. The previous code treated each shard as a separate
"variant" and would have incorrectly selected a single 50 GB shard
as fitting, ignoring the other 8 shards needed.
Tested with unsloth/DeepSeek-V3.2-GGUF (237 GGUF files, 27
variants from 150 GB to 1.25 TB). Correctly groups and sorts
all variants by total size.
Two changes for GGUF variant selection:
1. Default variant preference now starts with UD-Q4_K_XL (Unsloth
Dynamic quantization) which provides better quality per bit than
standard Q4_K_M. Also added UD-Q2_K_XL, UD-IQ2_M, UD-IQ1_M,
UD-IQ1_S as small fallback options.
2. If the selected variant doesn't fit on disk, automatically fall
back to the smallest GGUF variant in the repo that does fit.
Queries all GGUF file sizes via get_paths_info() and picks the
smallest one under the free disk space limit. If nothing fits,
raises a clear error.
This means users with limited disk space won't get a download
error -- they'll get a smaller quantization instead.
Query file sizes from HuggingFace via get_paths_info() before
downloading, and compare against free disk space on the cache
partition. Raises a clear error if there is not enough space,
instead of failing mid-download.
Uses get_paths_info() instead of repo_info() because xet-stored
repos return size=None from repo_info().siblings, but
get_paths_info() returns the actual file sizes.
If the size check fails for any reason (network error, API change),
it logs a warning and continues with the download anyway.
If CUDA_VISIBLE_DEVICES is already set in the environment (e.g.,
by the user or a wrapper script), only consider those GPUs when
selecting devices for llama-server. nvidia-smi reports all physical
GPUs regardless of CUDA_VISIBLE_DEVICES, so we filter its output
to match the allowed set.
Without this, the GPU selector could pick a GPU outside the user's
allowed set, overriding their restriction.
Automatically select the best GPU(s) for a GGUF model based on
file size and available VRAM, instead of relying on hardcoded
-ngl -1 or letting llama-server guess.
Logic:
1. Measure total GGUF file size (including split shards)
2. Query free memory per GPU via nvidia-smi
3. If the model fits in 70% of the most-free GPU's memory,
pin to that single GPU (CUDA_VISIBLE_DEVICES=X, no --fit)
4. If it needs multiple GPUs, pick the N most-free GPUs
(CUDA_VISIBLE_DEVICES=X,Y, no --fit)
5. If it's too large for all GPUs combined, omit
CUDA_VISIBLE_DEVICES and use --fit on to let llama-server
handle partial offloading
The 70% threshold accounts for KV cache and compute buffers
that sit on top of the model weights.
Removed the -ngl parameter (was hardcoded to -1). llama-server's
default of "auto" handles layer offloading correctly, especially
with --fit on for oversized models.
Tested on 8x B200:
- 1B model (0.75 GB): picks 1 GPU, no --fit
- 27B model (17 GB): picks 1 GPU, no --fit
- 405B model (230 GB): picks 2 GPUs, no --fit
- 2TB model: all GPUs, --fit on
Refactor command building (deduplicate HF/local paths) and add
flags for better performance:
- --parallel 1: studio is single-user, so only 1 inference slot
is needed. The previous auto-detect picked 4 slots, wasting
VRAM on 3 unused KV caches.
- --flash-attn on: force flash attention for faster inference.
Default is "auto" which may not always enable it.
- --fit on: auto-adjust parameters to fit in available device
memory. Already the default but now explicit.
Also cleaned up the duplicated command building for HF vs local
mode into a single block.
Remove the hard max_tokens=2048 default and le=4096 cap for GGUF
chat completions. When max_tokens is not set (None), the field is
omitted from the llama-server payload entirely, letting the model
generate until it produces an EOS token or hits the context limit.
This is critical for thinking/reasoning models (Qwen3.5, DeepSeek-R1,
etc.) where the thinking phase alone can consume 1000+ tokens before
the actual answer. With the previous 2048 default, simple questions
like "What is 2+2?" used all tokens on thinking and produced empty
visible responses.
Changes:
- llama_cpp.py: max_tokens default None, only include in payload
when explicitly set
- models/inference.py: default None, remove le=4096 cap
- routes/inference.py: pass max_tokens directly, no "or 2048" fallback
llama-server handles omitted max_tokens gracefully (generates until
EOS or context limit). The context size (-c flag, default 4096) acts
as the hard upper bound.
llama-server sends thinking/reasoning tokens as "reasoning_content"
in the SSE delta (separate from "content"). The studio was only
reading delta.content, so all reasoning tokens from models like
Qwen3.5, Qwen3-Thinking, DeepSeek-R1, etc. were silently dropped.
This caused "replies with nothing" for thinking models: the model
would spend its entire token budget on reasoning, produce zero
content tokens, and the user would see an empty response.
Fix: read reasoning_content from the delta and wrap it in
<think>...</think> tags. The frontend already has full support
for these tags (parse-assistant-content.ts splits them into
reasoning parts, reasoning.tsx renders a collapsible "Thinking..."
indicator).
Verified with Qwen3.5-27B-GGUF (UD-Q4_K_XL):
- Before: "What is 2+2?" -> empty response (all tokens in reasoning)
- After: shows collapsible thinking + answer "4"
startswith(prefix) could match unrelated split variants whose names
extend the selected file's prefix (e.g. model-Q8_0-v2-00001-of-...).
Now builds an exact regex from the chosen file's base prefix and shard
total so only true siblings are downloaded.
Substring matching (e.g. "Q8_0" in filename) could match superset
variants like "IQ8_0", causing wrong quantizations to be downloaded.
Now uses word-boundary regex for variant matching and discovers split
shards by shared filename prefix rather than treating all variant
matches as shards.
LlamaCppBackend.load_model() only downloaded the first matching GGUF
file. For split models (e.g. 7B Q8_0 with 3 shards), llama-server
needs all shards present. Now collects and downloads all matching files.
Fixes two bugs:
1. Chat template tags (<|im_start|>, <|im_end|>) leaking into output
because /v1/completions treated them as literal text
2. Image hallucination because image_b64 was never passed to llama-server
Now llama-server handles chat templates natively and receives images
as OpenAI-format multimodal content parts for vision models.
Replace Python-side GGUF download with llama-server's native -hf flag for
HuggingFace repos. Add frontend variant picker so users can choose
quantization (Q4_K_M, Q8_0, BF16, etc.) with file sizes. Fix vision
detection via mmproj files instead of hardcoding is_vision=False.
