The smallest-fitting-variant fallback now groups split GGUF shards
by their variant prefix and sums all shard sizes per variant.
For example, DeepSeek-V3.2 UD-Q4_K_XL has 9 shards totaling
379.8 GB. The previous code treated each shard as a separate
"variant" and would have incorrectly selected a single 50 GB shard
as fitting, ignoring the other 8 shards needed.
Tested with unsloth/DeepSeek-V3.2-GGUF (237 GGUF files, 27
variants from 150 GB to 1.25 TB). Correctly groups and sorts
all variants by total size.
Two changes for GGUF variant selection:
1. Default variant preference now starts with UD-Q4_K_XL (Unsloth
Dynamic quantization) which provides better quality per bit than
standard Q4_K_M. Also added UD-Q2_K_XL, UD-IQ2_M, UD-IQ1_M,
UD-IQ1_S as small fallback options.
2. If the selected variant doesn't fit on disk, automatically fall
back to the smallest GGUF variant in the repo that does fit.
Queries all GGUF file sizes via get_paths_info() and picks the
smallest one under the free disk space limit. If nothing fits,
raises a clear error.
This means users with limited disk space won't get a download
error -- they'll get a smaller quantization instead.
Query file sizes from HuggingFace via get_paths_info() before
downloading, and compare against free disk space on the cache
partition. Raises a clear error if there is not enough space,
instead of failing mid-download.
Uses get_paths_info() instead of repo_info() because xet-stored
repos return size=None from repo_info().siblings, but
get_paths_info() returns the actual file sizes.
If the size check fails for any reason (network error, API change),
it logs a warning and continues with the download anyway.
Set HF_HOME, HF_HUB_CACHE, HF_XET_CACHE, UV_CACHE_DIR, and
VLLM_CACHE_ROOT to a unified location under ~/.unsloth/studio/cache/
on startup. This keeps all model downloads, datasets, and caches
in one place instead of scattered across ~/.cache/huggingface,
~/.cache/uv, etc.
Layout:
~/.unsloth/studio/cache/
huggingface/ (HF_HOME)
hub/ (HF_HUB_CACHE -- model/dataset downloads)
xet/ (HF_XET_CACHE -- xet blob store)
uv/ (UV_CACHE_DIR -- uv package cache)
vllm/ (VLLM_CACHE_ROOT -- vllm compiled kernels)
Only sets variables that are not already in the environment, so
user overrides (e.g. HF_HOME=/data/models) are respected.
Cross-platform: uses Path.home() which resolves correctly on
Linux (~), macOS (~), and Windows (C:\Users\<user>).
If CUDA_VISIBLE_DEVICES is already set in the environment (e.g.,
by the user or a wrapper script), only consider those GPUs when
selecting devices for llama-server. nvidia-smi reports all physical
GPUs regardless of CUDA_VISIBLE_DEVICES, so we filter its output
to match the allowed set.
Without this, the GPU selector could pick a GPU outside the user's
allowed set, overriding their restriction.
Automatically select the best GPU(s) for a GGUF model based on
file size and available VRAM, instead of relying on hardcoded
-ngl -1 or letting llama-server guess.
Logic:
1. Measure total GGUF file size (including split shards)
2. Query free memory per GPU via nvidia-smi
3. If the model fits in 70% of the most-free GPU's memory,
pin to that single GPU (CUDA_VISIBLE_DEVICES=X, no --fit)
4. If it needs multiple GPUs, pick the N most-free GPUs
(CUDA_VISIBLE_DEVICES=X,Y, no --fit)
5. If it's too large for all GPUs combined, omit
CUDA_VISIBLE_DEVICES and use --fit on to let llama-server
handle partial offloading
The 70% threshold accounts for KV cache and compute buffers
that sit on top of the model weights.
Removed the -ngl parameter (was hardcoded to -1). llama-server's
default of "auto" handles layer offloading correctly, especially
with --fit on for oversized models.
Tested on 8x B200:
- 1B model (0.75 GB): picks 1 GPU, no --fit
- 27B model (17 GB): picks 1 GPU, no --fit
- 405B model (230 GB): picks 2 GPUs, no --fit
- 2TB model: all GPUs, --fit on
Refactor command building (deduplicate HF/local paths) and add
flags for better performance:
- --parallel 1: studio is single-user, so only 1 inference slot
is needed. The previous auto-detect picked 4 slots, wasting
VRAM on 3 unused KV caches.
- --flash-attn on: force flash attention for faster inference.
Default is "auto" which may not always enable it.
- --fit on: auto-adjust parameters to fit in available device
memory. Already the default but now explicit.
Also cleaned up the duplicated command building for HF vs local
mode into a single block.
Remove the hard max_tokens=2048 default and le=4096 cap for GGUF
chat completions. When max_tokens is not set (None), the field is
omitted from the llama-server payload entirely, letting the model
generate until it produces an EOS token or hits the context limit.
This is critical for thinking/reasoning models (Qwen3.5, DeepSeek-R1,
etc.) where the thinking phase alone can consume 1000+ tokens before
the actual answer. With the previous 2048 default, simple questions
like "What is 2+2?" used all tokens on thinking and produced empty
visible responses.
Changes:
- llama_cpp.py: max_tokens default None, only include in payload
when explicitly set
- models/inference.py: default None, remove le=4096 cap
- routes/inference.py: pass max_tokens directly, no "or 2048" fallback
llama-server handles omitted max_tokens gracefully (generates until
EOS or context limit). The context size (-c flag, default 4096) acts
as the hard upper bound.
llama-server sends thinking/reasoning tokens as "reasoning_content"
in the SSE delta (separate from "content"). The studio was only
reading delta.content, so all reasoning tokens from models like
Qwen3.5, Qwen3-Thinking, DeepSeek-R1, etc. were silently dropped.
This caused "replies with nothing" for thinking models: the model
would spend its entire token budget on reasoning, produce zero
content tokens, and the user would see an empty response.
Fix: read reasoning_content from the delta and wrap it in
<think>...</think> tags. The frontend already has full support
for these tags (parse-assistant-content.ts splits them into
reasoning parts, reasoning.tsx renders a collapsible "Thinking..."
indicator).
Verified with Qwen3.5-27B-GGUF (UD-Q4_K_XL):
- Before: "What is 2+2?" -> empty response (all tokens in reasoning)
- After: shows collapsible thinking + answer "4"
* fix: resolve compare mode deadlock, cancel_event poisoning, and add dispatcher-based IPC optimization
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* revert to 2048 tokens
* refactor: extract dispatcher timeout values into named constants
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: guard dispatcher shutdown against active compare mailboxes
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* miscallenous studio
* chore: upload dataset misc
* chore: redudancy studio cleanup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: adress the pr comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* fix: adress comments about recipes
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix: quotation marks
* diceware passphrase generation
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Roland Tannous <rolandtannous@gravityq.ai>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix(seed): disable remote code execution for seed inspect loads
* fix(test): use __file__-relative path in seed test
The test used a CWD-relative path (`studio/backend/routes/...`) which
only resolved when pytest was invoked from the repo root. Use
`Path(__file__).resolve()` so the test passes regardless of CWD.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Test <test@test.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix: disable remote code loading for ai-assist model hint lookup
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Roland Tannous <115670425+rolandtannous@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
The studio was disabling flex attention entirely on Blackwell+ GPUs
(sm_120 and above) by setting UNSLOTH_ENABLE_FLEX_ATTENTION=0 at
startup. This was a workaround for the flex_attention backward kernel
exceeding shared memory limits on these GPUs.
The root cause is now fixed in unsloth-zoo (PR #542) which patches the
backward kernel config selection to generate safe fallback configs that
fit within the GPU's shared memory limit. With that fix, flex attention
works correctly on Blackwell GPUs and provides a ~1.3x speedup over
the SDPA fallback.
* Update CODEOWNERS for studio and cli
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
- Workers now compute backend_path and venv_t5 locally via Path(__file__)
- Moved .venv_t5 to ~/.unsloth/studio/.venv_t5
- Added ensure_studio_directories() call on server startup
- Expanded CLI studio command into sub-app with setup subcommand
Tier 1 check-format was picking images.zip over testmini.parquet,
causing wrong columns (image/label) and broken VLM mapping.
Also log first VLM conversion failure instead of swallowing silently.
